==> Building on aurorus ==> Checking for remote environment... ==> Syncing package to remote host... sending incremental file list created directory packages/seaweedfs ./ .SRCINFO 430 100% 0.00kB/s 0:00:00 430 100% 0.00kB/s 0:00:00 (xfr#1, to-chk=7/9) .nvchecker.toml 91 100% 88.87kB/s 0:00:00 91 100% 88.87kB/s 0:00:00 (xfr#2, to-chk=6/9) LICENSE 646 100% 630.86kB/s 0:00:00 646 100% 630.86kB/s 0:00:00 (xfr#3, to-chk=5/9) PKGBUILD 1,486 100% 1.42MB/s 0:00:00 1,486 100% 1.42MB/s 0:00:00 (xfr#4, to-chk=4/9) REUSE.toml 375 100% 366.21kB/s 0:00:00 375 100% 366.21kB/s 0:00:00 (xfr#5, to-chk=3/9) seaweedfs-4.44-1.log 731 100% 713.87kB/s 0:00:00 731 100% 713.87kB/s 0:00:00 (xfr#6, to-chk=2/9) LICENSES/ LICENSES/0BSD.txt -> ../LICENSE sent 2,504 bytes received 185 bytes 1,075.60 bytes/sec total size is 3,232 speedup is 1.20 ==> Applying RISC-V patches... sending incremental file list ./ riscv64.patch 248 100% 0.00kB/s 0:00:00 248 100% 0.00kB/s 0:00:00 (xfr#1, to-chk=0/2) sent 299 bytes received 38 bytes 134.80 bytes/sec total size is 248 speedup is 0.74 patching file PKGBUILD ==> Patching arch to riscv64... ==> Running pkgctl build --arch riscv64 on remote host... ==> WARNING: invalid architecture: riscv64 ==> Updating pacman database cache [?25l:: Synchronizing package databases... core downloading... extra downloading... multilib downloading... [?25h==> Building seaweedfs -> repo: extra -> arch: riscv64 -> worker: felix-2 ==> Building seaweedfs for [extra] (riscv64) Note: in a future version of systemd-nspawn the default set of permitted socket address families will be restricted to AF_INET, AF_INET6 and AF_UNIX. Use --restrict-address-families= to configure the set of permitted socket address families, or set RestrictAddressFamilies= in a .nspawn file. [?25l:: Synchronizing package databases... core downloading... extra downloading... :: Starting full system upgrade... there is nothing to do [?25h==> Building in chroot for [extra] (riscv64)... ==> Synchronizing chroot copy [/var/lib/archbuild/extra-riscv64/root] -> [felix-2]...done ==> Making package: seaweedfs 4.44-1 (Mon Aug 24 21:37:51 2026) ==> Retrieving sources...  -> Downloading seaweedfs-4.44.tar.gz... % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 00:01 0 0 0 0 0 0 0 0 0 00:01 0 0 0 0 0 0 0 0 0 00:01 0 0 98.73M 0 794.4k 0 0 329.0k 0 05:07 00:02 05:05 793.5k 5 98.73M 5 5.00M 0 0 1.43M 0 01:08 00:03 01:05 2.42M 9 98.73M 9 9.38M 0 0 2.09M 0 00:47 00:04 00:43 3.06M 15 98.73M 15 15.29M 0 0 2.78M 0 00:35 00:05 00:30 3.74M 21 98.73M 21 21.33M 0 0 3.28M 0 00:30 00:06 00:24 4.19M 28 98.73M 28 28.14M 0 0 3.72M 0 00:26 00:07 00:19 5.32M 34 98.73M 34 34.54M 0 0 4.03M 0 00:24 00:08 00:16 5.81M 40 98.73M 40 39.57M 0 0 4.14M 0 00:23 00:09 00:14 5.94M 46 98.73M 46 45.79M 0 0 4.33M 0 00:22 00:10 00:12 6.02M 53 98.73M 53 52.66M 0 0 4.55M 0 00:21 00:11 00:10 6.18M 60 98.73M 60 59.89M 0 0 4.75M 0 00:20 00:12 00:08 6.27M 68 98.73M 68 67.47M 0 0 4.95M 0 00:19 00:13 00:06 6.51M 75 98.73M 75 74.35M 0 0 5.08M 0 00:19 00:14 00:05 6.86M 80 98.73M 80 79.37M 0 0 5.08M 0 00:19 00:15 00:04 6.62M 85 98.73M 85 84.62M 0 0 5.08M 0 00:19 00:16 00:03 6.29M 92 98.73M 92 91.56M 0 0 5.17M 0 00:19 00:17 00:02 6.21M 99 98.73M 99 98.55M 0 0 5.26M 0 00:18 00:18 6.08M 100 98.73M 100 98.73M 0 0 5.24M 0 00:18 00:18 6.08M 100 98.73M 100 98.73M 0 0 5.24M 0 00:18 00:18 6.08M 100 98.73M 100 98.73M 0 0 5.24M 0 00:18 00:18 6.08M ==> Validating source files with sha256sums... seaweedfs-4.44.tar.gz ... Passed Note: in a future version of systemd-nspawn the default set of permitted socket address families will be restricted to AF_INET, AF_INET6 and AF_UNIX. Use --restrict-address-families= to configure the set of permitted socket address families, or set RestrictAddressFamilies= in a .nspawn file. ==> Making package: seaweedfs 4.44-1 (Mon Aug 24 21:38:47 2026) ==> Checking runtime dependencies... ==> Installing missing dependencies... [?25lresolving dependencies... looking for conflicting packages... Package (1) New Version Net Change Download Size extra/mailcap 2.1.54-3 0.11 MiB 0.03 MiB Total Download Size: 0.03 MiB Total Installed Size: 0.11 MiB :: Proceed with installation? [Y/n] :: Retrieving packages... mailcap-2.1.54-3-any downloading... checking keyring... checking package integrity... loading package files... checking for file conflicts... :: Processing package changes... installing mailcap... :: Running post-transaction hooks... (1/1) Arming ConditionNeedsUpdate... [?25h==> Checking buildtime dependencies... ==> Installing missing dependencies... [?25lresolving dependencies... looking for conflicting packages... Package (1) New Version Net Change extra/go 2:1.27.0-1 216.77 MiB Total Installed Size: 216.77 MiB :: Proceed with installation? [Y/n] checking keyring... checking package integrity... loading package files... checking for file conflicts... :: Processing package changes... installing go... :: Running post-transaction hooks... (1/1) Arming ConditionNeedsUpdate... [?25h==> Retrieving sources...  -> Found seaweedfs-4.44.tar.gz ==> WARNING: Skipping all source file integrity checks. ==> Extracting sources...  -> Extracting seaweedfs-4.44.tar.gz with bsdtar ==> Starting prepare()... ==> Starting build()... ==> Starting check()... ? github.com/seaweedfs/seaweedfs/weed [no test files] === RUN TestStaticGzMirror --- PASS: TestStaticGzMirror (0.18s) === RUN TestStaticHandlerGzip --- PASS: TestStaticHandlerGzip (0.16s) === RUN TestStaticHandlerIdentity --- PASS: TestStaticHandlerIdentity (0.01s) === RUN TestStaticHandlerNotModified --- PASS: TestStaticHandlerNotModified (0.00s) === RUN TestStaticHandlerVariantETags --- PASS: TestStaticHandlerVariantETags (0.00s) === RUN TestStaticHandlerNotFound --- PASS: TestStaticHandlerNotFound (0.00s) === RUN TestAcceptsGzip --- PASS: TestAcceptsGzip (0.00s) PASS ok github.com/seaweedfs/seaweedfs/weed/admin 0.797s === RUN TestApplyDefaults_WithEmbeddedStruct --- PASS: TestApplyDefaults_WithEmbeddedStruct (0.00s) === RUN TestApplyDefaults_PartiallySet --- PASS: TestApplyDefaults_PartiallySet (0.00s) === RUN TestApplyDefaults_NonPointer --- PASS: TestApplyDefaults_NonPointer (0.00s) === RUN TestApplyDefaults_NonStruct --- PASS: TestApplyDefaults_NonStruct (0.00s) === RUN TestApplyDefaults_EmptySchema --- PASS: TestApplyDefaults_EmptySchema (0.00s) === RUN TestApplyDefaults_MissingSchemaField --- PASS: TestApplyDefaults_MissingSchemaField (0.00s) PASS ok github.com/seaweedfs/seaweedfs/weed/admin/config 0.251s === RUN TestAdminLockManagerRefCountsOverlappingHolds === PAUSE TestAdminLockManagerRefCountsOverlappingHolds === RUN TestAdminLockManagerYieldsBeforeReacquire === PAUSE TestAdminLockManagerYieldsBeforeReacquire === RUN TestAdminLockManagerFairnessWindowForcesFullRelease === PAUSE TestAdminLockManagerFairnessWindowForcesFullRelease === RUN TestAdminLockManagerAcquireWaitsForReleaseInFlight === PAUSE TestAdminLockManagerAcquireWaitsForReleaseInFlight === RUN TestListClusterNodesRequestIncludesFilerGroup --- PASS: TestListClusterNodesRequestIncludesFilerGroup (0.00s) === RUN TestListClusterNodesRequestKeepsDefaultFilerGroupEmpty --- PASS: TestListClusterNodesRequestKeepsDefaultFilerGroupEmpty (0.00s) === RUN TestCleanMaintenanceScript === RUN TestCleanMaintenanceScript/empty === RUN TestCleanMaintenanceScript/only_lock_unlock === RUN TestCleanMaintenanceScript/strips_lock_and_unlock === RUN TestCleanMaintenanceScript/case_insensitive_lock === RUN TestCleanMaintenanceScript/preserves_comments_removal === RUN TestCleanMaintenanceScript/no_lock_unlock_present === RUN TestCleanMaintenanceScript/windows_line_endings === RUN TestCleanMaintenanceScript/lock_with_inline_comment === RUN TestCleanMaintenanceScript/command_with_inline_comment_preserved === RUN TestCleanMaintenanceScript/only_inline_comment_after_stripping === RUN TestCleanMaintenanceScript/typical_master_default --- PASS: TestCleanMaintenanceScript (0.00s) --- PASS: TestCleanMaintenanceScript/empty (0.00s) --- PASS: TestCleanMaintenanceScript/only_lock_unlock (0.00s) --- PASS: TestCleanMaintenanceScript/strips_lock_and_unlock (0.00s) --- PASS: TestCleanMaintenanceScript/case_insensitive_lock (0.00s) --- PASS: TestCleanMaintenanceScript/preserves_comments_removal (0.00s) --- PASS: TestCleanMaintenanceScript/no_lock_unlock_present (0.00s) --- PASS: TestCleanMaintenanceScript/windows_line_endings (0.00s) --- PASS: TestCleanMaintenanceScript/lock_with_inline_comment (0.00s) --- PASS: TestCleanMaintenanceScript/command_with_inline_comment_preserved (0.00s) --- PASS: TestCleanMaintenanceScript/only_inline_comment_after_stripping (0.00s) --- PASS: TestCleanMaintenanceScript/typical_master_default (0.00s) === RUN TestValidateBucketLifecycleRules_Valid --- PASS: TestValidateBucketLifecycleRules_Valid (0.01s) === RUN TestValidateBucketLifecycleRules_Empty --- PASS: TestValidateBucketLifecycleRules_Empty (0.00s) === RUN TestValidateBucketLifecycleRules_TooManyRules --- PASS: TestValidateBucketLifecycleRules_TooManyRules (0.01s) === RUN TestValidateBucketLifecycleRules_DuplicateIDs --- PASS: TestValidateBucketLifecycleRules_DuplicateIDs (0.00s) === RUN TestValidateBucketLifecycleRules_IDTooLong --- PASS: TestValidateBucketLifecycleRules_IDTooLong (0.00s) === RUN TestValidateBucketLifecycleRules_InvalidStatus --- PASS: TestValidateBucketLifecycleRules_InvalidStatus (0.00s) === RUN TestValidateBucketLifecycleRules_ExpirationDaysAndDateMutuallyExclusive --- PASS: TestValidateBucketLifecycleRules_ExpirationDaysAndDateMutuallyExclusive (0.00s) === RUN TestValidateBucketLifecycleRules_DeleteMarkerWithExpirationDaysRejected --- PASS: TestValidateBucketLifecycleRules_DeleteMarkerWithExpirationDaysRejected (0.00s) === RUN TestValidateBucketLifecycleRules_DeleteMarkerWithExpirationDateRejected --- PASS: TestValidateBucketLifecycleRules_DeleteMarkerWithExpirationDateRejected (0.00s) === RUN TestValidateBucketLifecycleRules_DeleteMarkerAlone --- PASS: TestValidateBucketLifecycleRules_DeleteMarkerAlone (0.00s) === RUN TestValidateBucketLifecycleRules_InvalidExpirationDate --- PASS: TestValidateBucketLifecycleRules_InvalidExpirationDate (0.00s) === RUN TestValidateBucketLifecycleRules_NegativeExpirationDays --- PASS: TestValidateBucketLifecycleRules_NegativeExpirationDays (0.00s) === RUN TestValidateBucketLifecycleRules_StandaloneNewerNoncurrentVersions --- PASS: TestValidateBucketLifecycleRules_StandaloneNewerNoncurrentVersions (0.00s) === RUN TestValidateBucketLifecycleRules_NegativeAbortMultipartDays --- PASS: TestValidateBucketLifecycleRules_NegativeAbortMultipartDays (0.00s) === RUN TestValidateBucketLifecycleRules_SizeGreaterThanNotLessThanSizeLessThan --- PASS: TestValidateBucketLifecycleRules_SizeGreaterThanNotLessThanSizeLessThan (0.00s) === RUN TestValidateBucketLifecycleRules_NegativeSizeBounds --- PASS: TestValidateBucketLifecycleRules_NegativeSizeBounds (0.00s) === RUN TestValidateBucketLifecycleRules_EmptyTagKey --- PASS: TestValidateBucketLifecycleRules_EmptyTagKey (0.00s) === RUN TestValidateBucketLifecycleRules_EmptyTagValue --- PASS: TestValidateBucketLifecycleRules_EmptyTagValue (0.00s) === RUN TestValidateBucketLifecycleRules_NoAction --- PASS: TestValidateBucketLifecycleRules_NoAction (0.00s) === RUN TestFromBucketLifecycleRule_MapsAllFields --- PASS: TestFromBucketLifecycleRule_MapsAllFields (0.01s) === RUN TestFromBucketLifecycleRule_EmptyExpirationDate --- PASS: TestFromBucketLifecycleRule_EmptyExpirationDate (0.00s) === RUN TestFromBucketLifecycleRule_InvalidExpirationDate --- PASS: TestFromBucketLifecycleRule_InvalidExpirationDate (0.01s) === RUN TestBucketLifecycleMutation_SetsXML --- PASS: TestBucketLifecycleMutation_SetsXML (0.00s) === RUN TestBucketLifecycleMutation_ClearsBothKeys --- PASS: TestBucketLifecycleMutation_ClearsBothKeys (0.00s) === RUN TestBucketLifecycleMutation_TouchesOnlyLifecycleKeys --- PASS: TestBucketLifecycleMutation_TouchesOnlyLifecycleKeys (0.00s) === RUN TestSetBucketLifecycle_RejectsOversizedConfiguration --- PASS: TestSetBucketLifecycle_RejectsOversizedConfiguration (0.15s) === RUN TestBucketLifecycleErrorStatus --- PASS: TestBucketLifecycleErrorStatus (0.00s) === RUN TestCollectCollectionStatsECUnevenShards --- PASS: TestCollectCollectionStatsECUnevenShards (0.00s) === RUN TestCollectCollectionStatsECEmptyCollection --- PASS: TestCollectCollectionStatsECEmptyCollection (0.00s) === RUN TestCollectCollectionStatsECFileAndDeleteCountAggregation --- PASS: TestCollectCollectionStatsECFileAndDeleteCountAggregation (0.00s) === RUN TestCollectCollectionStatsECFileCountMaxDedupe --- PASS: TestCollectCollectionStatsECFileCountMaxDedupe (0.00s) === RUN TestTotalCollectionFileCount --- PASS: TestTotalCollectionFileCount (0.00s) === RUN TestTotalCollectionFileCountUnderReplicated --- PASS: TestTotalCollectionFileCountUnderReplicated (0.00s) === RUN TestApplyMaintenanceConfigFromToml I0824 22:14:43.321819 config_toml.go:98 Applied [maintenance.vacuum] settings from admin.toml I0824 22:14:43.330684 config_toml.go:110 Applied [maintenance.erasure_coding] settings from admin.toml --- PASS: TestApplyMaintenanceConfigFromToml (0.06s) === RUN TestApplyMaintenanceConfigFromTomlNoKeys --- PASS: TestApplyMaintenanceConfigFromTomlNoKeys (0.00s) === RUN TestApplyMaintenanceConfigFromTomlRequiresDataDir --- PASS: TestApplyMaintenanceConfigFromTomlRequiresDataDir (0.05s) === RUN TestApplyPluginConfigFromTomlUpdatesExistingConfig I0824 22:14:43.472525 config_toml.go:155 Applied [maintenance.vacuum] settings from admin.toml to plugin config --- PASS: TestApplyPluginConfigFromTomlUpdatesExistingConfig (0.08s) === RUN TestApplyPluginConfigFromTomlClampsRetryValues I0824 22:14:43.476590 config_toml.go:155 Applied [maintenance.vacuum] settings from admin.toml to plugin config --- PASS: TestApplyPluginConfigFromTomlClampsRetryValues (0.00s) === RUN TestApplyPluginConfigFromTomlLeavesMissingConfigToBootstrap --- PASS: TestApplyPluginConfigFromTomlLeavesMissingConfigToBootstrap (0.00s) === RUN TestApplyPluginTomlDefaultsOverlaysBootstrapConfig I0824 22:14:43.478932 config_toml.go:166 Applied [maintenance.vacuum] settings from admin.toml to plugin defaults --- PASS: TestApplyPluginTomlDefaultsOverlaysBootstrapConfig (0.00s) === RUN TestApplyPluginConfigFromTomlErasureCodingKeyPlacement I0824 22:14:43.481414 config_toml.go:155 Applied [maintenance.erasure_coding] settings from admin.toml to plugin config --- PASS: TestApplyPluginConfigFromTomlErasureCodingKeyPlacement (0.00s) === RUN TestGenerateBreadcrumbs === RUN TestGenerateBreadcrumbs/root_path === RUN TestGenerateBreadcrumbs/simple_path === RUN TestGenerateBreadcrumbs/nested_path === RUN TestGenerateBreadcrumbs/bucket_path === RUN TestGenerateBreadcrumbs/bucket_nested_path === RUN TestGenerateBreadcrumbs/path_with_trailing_slash --- PASS: TestGenerateBreadcrumbs (0.00s) --- PASS: TestGenerateBreadcrumbs/root_path (0.00s) --- PASS: TestGenerateBreadcrumbs/simple_path (0.00s) --- PASS: TestGenerateBreadcrumbs/nested_path (0.00s) --- PASS: TestGenerateBreadcrumbs/bucket_path (0.00s) --- PASS: TestGenerateBreadcrumbs/bucket_nested_path (0.00s) --- PASS: TestGenerateBreadcrumbs/path_with_trailing_slash (0.00s) === RUN TestPathHandlingWithForwardSlashes === RUN TestPathHandlingWithForwardSlashes/root === RUN TestPathHandlingWithForwardSlashes/single_level === RUN TestPathHandlingWithForwardSlashes/multiple_levels === RUN TestPathHandlingWithForwardSlashes/bucket_path --- PASS: TestPathHandlingWithForwardSlashes (0.00s) --- PASS: TestPathHandlingWithForwardSlashes/root (0.00s) --- PASS: TestPathHandlingWithForwardSlashes/single_level (0.00s) --- PASS: TestPathHandlingWithForwardSlashes/multiple_levels (0.00s) --- PASS: TestPathHandlingWithForwardSlashes/bucket_path (0.00s) === RUN TestParentPathCalculationLogic === RUN TestParentPathCalculationLogic/root_path === RUN TestParentPathCalculationLogic/single_level === RUN TestParentPathCalculationLogic/two_levels === RUN TestParentPathCalculationLogic/deep_nesting === RUN TestParentPathCalculationLogic/bucket_root === RUN TestParentPathCalculationLogic/bucket_directory --- PASS: TestParentPathCalculationLogic (0.00s) --- PASS: TestParentPathCalculationLogic/root_path (0.00s) --- PASS: TestParentPathCalculationLogic/single_level (0.00s) --- PASS: TestParentPathCalculationLogic/two_levels (0.00s) --- PASS: TestParentPathCalculationLogic/deep_nesting (0.00s) --- PASS: TestParentPathCalculationLogic/bucket_root (0.00s) --- PASS: TestParentPathCalculationLogic/bucket_directory (0.00s) === RUN TestFileExtensionHandlingLogic === RUN TestFileExtensionHandlingLogic/file.txt === RUN TestFileExtensionHandlingLogic/file.log === RUN TestFileExtensionHandlingLogic/archive.tar.gz === RUN TestFileExtensionHandlingLogic/image.jpg === RUN TestFileExtensionHandlingLogic/document.pdf === RUN TestFileExtensionHandlingLogic/data.json === RUN TestFileExtensionHandlingLogic/noextension === RUN TestFileExtensionHandlingLogic/.hidden === RUN TestFileExtensionHandlingLogic/file.TXT === RUN TestFileExtensionHandlingLogic/file.JPG --- PASS: TestFileExtensionHandlingLogic (0.00s) --- PASS: TestFileExtensionHandlingLogic/file.txt (0.00s) --- PASS: TestFileExtensionHandlingLogic/file.log (0.00s) --- PASS: TestFileExtensionHandlingLogic/archive.tar.gz (0.00s) --- PASS: TestFileExtensionHandlingLogic/image.jpg (0.00s) --- PASS: TestFileExtensionHandlingLogic/document.pdf (0.00s) --- PASS: TestFileExtensionHandlingLogic/data.json (0.00s) --- PASS: TestFileExtensionHandlingLogic/noextension (0.00s) --- PASS: TestFileExtensionHandlingLogic/.hidden (0.00s) --- PASS: TestFileExtensionHandlingLogic/file.TXT (0.00s) --- PASS: TestFileExtensionHandlingLogic/file.JPG (0.00s) === RUN TestBucketPathDetectionLogic === RUN TestBucketPathDetectionLogic/root_is_not_a_bucket_path === RUN TestBucketPathDetectionLogic/buckets_root === RUN TestBucketPathDetectionLogic/single_bucket === RUN TestBucketPathDetectionLogic/bucket_with_nested_path === RUN TestBucketPathDetectionLogic/non-bucket_path --- PASS: TestBucketPathDetectionLogic (0.00s) --- PASS: TestBucketPathDetectionLogic/root_is_not_a_bucket_path (0.00s) --- PASS: TestBucketPathDetectionLogic/buckets_root (0.00s) --- PASS: TestBucketPathDetectionLogic/single_bucket (0.00s) --- PASS: TestBucketPathDetectionLogic/bucket_with_nested_path (0.00s) --- PASS: TestBucketPathDetectionLogic/non-bucket_path (0.00s) === RUN TestPathJoinHandlesEdgeCases === RUN TestPathJoinHandlesEdgeCases/root_directory === RUN TestPathJoinHandlesEdgeCases/simple_directory === RUN TestPathJoinHandlesEdgeCases/nested_directory === RUN TestPathJoinHandlesEdgeCases/handles_trailing_slash === RUN TestPathJoinHandlesEdgeCases/handles_empty_name --- PASS: TestPathJoinHandlesEdgeCases (0.00s) --- PASS: TestPathJoinHandlesEdgeCases/root_directory (0.00s) --- PASS: TestPathJoinHandlesEdgeCases/simple_directory (0.00s) --- PASS: TestPathJoinHandlesEdgeCases/nested_directory (0.00s) --- PASS: TestPathJoinHandlesEdgeCases/handles_trailing_slash (0.00s) --- PASS: TestPathJoinHandlesEdgeCases/handles_empty_name (0.00s) === RUN TestWindowsPathNormalizationBehavior === RUN TestWindowsPathNormalizationBehavior/backslash_separator === RUN TestWindowsPathNormalizationBehavior/mixed_separators === RUN TestWindowsPathNormalizationBehavior/already_normalized === RUN TestWindowsPathNormalizationBehavior/simple_backslash_path === RUN TestWindowsPathNormalizationBehavior/deep_nested_path --- PASS: TestWindowsPathNormalizationBehavior (0.00s) --- PASS: TestWindowsPathNormalizationBehavior/backslash_separator (0.00s) --- PASS: TestWindowsPathNormalizationBehavior/mixed_separators (0.00s) --- PASS: TestWindowsPathNormalizationBehavior/already_normalized (0.00s) --- PASS: TestWindowsPathNormalizationBehavior/simple_backslash_path (0.00s) --- PASS: TestWindowsPathNormalizationBehavior/deep_nested_path (0.00s) === RUN TestBreadcrumbPathFormatting === RUN TestBreadcrumbPathFormatting/breadcrumbs_for_/ === RUN TestBreadcrumbPathFormatting/breadcrumbs_for_/folder === RUN TestBreadcrumbPathFormatting/breadcrumbs_for_/folder/subfolder === RUN TestBreadcrumbPathFormatting/breadcrumbs_for_/buckets/mybucket === RUN TestBreadcrumbPathFormatting/breadcrumbs_for_/buckets/mybucket/data --- PASS: TestBreadcrumbPathFormatting (0.00s) --- PASS: TestBreadcrumbPathFormatting/breadcrumbs_for_/ (0.00s) --- PASS: TestBreadcrumbPathFormatting/breadcrumbs_for_/folder (0.00s) --- PASS: TestBreadcrumbPathFormatting/breadcrumbs_for_/folder/subfolder (0.00s) --- PASS: TestBreadcrumbPathFormatting/breadcrumbs_for_/buckets/mybucket (0.00s) --- PASS: TestBreadcrumbPathFormatting/breadcrumbs_for_/buckets/mybucket/data (0.00s) === RUN TestDirectoryNavigation === RUN TestDirectoryNavigation/navigate_from_root === RUN TestDirectoryNavigation/navigate_from_single_folder === RUN TestDirectoryNavigation/navigate_from_nested_folder === RUN TestDirectoryNavigation/navigate_bucket_contents --- PASS: TestDirectoryNavigation (0.00s) --- PASS: TestDirectoryNavigation/navigate_from_root (0.00s) --- PASS: TestDirectoryNavigation/navigate_from_single_folder (0.00s) --- PASS: TestDirectoryNavigation/navigate_from_nested_folder (0.00s) --- PASS: TestDirectoryNavigation/navigate_bucket_contents (0.00s) === RUN TestResolveEntryMimePrefersStoredMime --- PASS: TestResolveEntryMimePrefersStoredMime (0.00s) === RUN TestResolveEntryMimePrefersStoredMimeMalformedParameter --- PASS: TestResolveEntryMimePrefersStoredMimeMalformedParameter (0.00s) === RUN TestResolveEntryMimeFallsBackToFilename --- PASS: TestResolveEntryMimeFallsBackToFilename (0.01s) === RUN TestResolveEntryMimeReturnsDirectoryMime --- PASS: TestResolveEntryMimeReturnsDirectoryMime (0.00s) === RUN TestResolveEntryMimeWhitespaceMimeFallsBackToFilename --- PASS: TestResolveEntryMimeWhitespaceMimeFallsBackToFilename (0.00s) === RUN TestIcebergLocationToFilerPath --- PASS: TestIcebergLocationToFilerPath (0.00s) === RUN TestFormatParquetRow --- PASS: TestFormatParquetRow (0.15s) === RUN TestTruncateCell --- PASS: TestTruncateCell (0.00s) === RUN TestExpirePluginJobAPI === RUN TestExpirePluginJobAPI/empty_job_id === RUN TestExpirePluginJobAPI/invalid_json === RUN TestExpirePluginJobAPI/job_not_found === RUN TestExpirePluginJobAPI/successful_expire === RUN TestExpirePluginJobAPI/non-active_job --- PASS: TestExpirePluginJobAPI (0.03s) --- PASS: TestExpirePluginJobAPI/empty_job_id (0.02s) --- PASS: TestExpirePluginJobAPI/invalid_json (0.00s) --- PASS: TestExpirePluginJobAPI/job_not_found (0.00s) --- PASS: TestExpirePluginJobAPI/successful_expire (0.00s) --- PASS: TestExpirePluginJobAPI/non-active_job (0.00s) === RUN TestApplyDescriptorDefaultsToPersistedConfigBackfillsAdminDefaults === PAUSE TestApplyDescriptorDefaultsToPersistedConfigBackfillsAdminDefaults === RUN TestApplyDescriptorDefaultsToPersistedConfigReplacesBlankAdminScript === PAUSE TestApplyDescriptorDefaultsToPersistedConfigReplacesBlankAdminScript === RUN TestFilterTrackedJobsByLane === PAUSE TestFilterTrackedJobsByLane === RUN TestFilterActivitiesByLane === PAUSE TestFilterActivitiesByLane === RUN TestCollectTierStats --- PASS: TestCollectTierStats (0.00s) === RUN TestCollectTierStatsEmpty --- PASS: TestCollectTierStatsEmpty (0.02s) === RUN TestGetObjectStoreUserDetailsIncludesInlinePolicies I0824 22:14:43.706386 config_loader.go:73 Using explicit credential store: memory --- PASS: TestGetObjectStoreUserDetailsIncludesInlinePolicies (0.00s) === RUN TestFilerAddressFunctionInterface user_management_test.go:46: FilerEtcStore correctly implements SetFilerAddressFunc interface --- PASS: TestFilerAddressFunctionInterface (0.00s) === RUN TestGenerateAccessKey --- PASS: TestGenerateAccessKey (0.00s) === RUN TestGenerateSecretKey --- PASS: TestGenerateSecretKey (0.00s) === RUN TestGenerateAccountId --- PASS: TestGenerateAccountId (0.00s) === RUN TestVacuumMonitorStateTransitions === PAUSE TestVacuumMonitorStateTransitions === RUN TestSyncVacuumState === PAUSE TestSyncVacuumState === RUN TestBuildExportVolumeDerivedFields --- PASS: TestBuildExportVolumeDerivedFields (0.00s) === RUN TestBuildExportDiskFilterAndTotals --- PASS: TestBuildExportDiskFilterAndTotals (0.00s) === RUN TestBuildExportDiskPrunesWhenNoMatch --- PASS: TestBuildExportDiskPrunesWhenNoMatch (0.00s) === RUN TestFindDuplicateVolumeIdsForExport --- PASS: TestFindDuplicateVolumeIdsForExport (0.00s) === RUN TestSendToWorkerDropsOnCancelledContext --- PASS: TestSendToWorkerDropsOnCancelledContext (0.00s) === RUN TestHeartbeatDuringUnregisterDoesNotPanic --- PASS: TestHeartbeatDuringUnregisterDoesNotPanic (0.06s) === CONT TestAdminLockManagerRefCountsOverlappingHolds === CONT TestApplyDescriptorDefaultsToPersistedConfigReplacesBlankAdminScript --- PASS: TestAdminLockManagerRefCountsOverlappingHolds (0.01s) === CONT TestApplyDescriptorDefaultsToPersistedConfigBackfillsAdminDefaults --- PASS: TestApplyDescriptorDefaultsToPersistedConfigBackfillsAdminDefaults (0.00s) --- PASS: TestApplyDescriptorDefaultsToPersistedConfigReplacesBlankAdminScript (0.00s) === CONT TestAdminLockManagerAcquireWaitsForReleaseInFlight === CONT TestAdminLockManagerFairnessWindowForcesFullRelease === CONT TestAdminLockManagerYieldsBeforeReacquire === CONT TestVacuumMonitorStateTransitions === CONT TestSyncVacuumState === RUN TestSyncVacuumState/no_change_when_state_matches === RUN TestSyncVacuumState/worker_connects_triggers_disable I0824 22:14:43.792258 admin_server.go:370 Vacuum plugin worker connected, disabling master automatic vacuum === RUN TestSyncVacuumState/worker_disconnects_triggers_enable I0824 22:14:43.793350 admin_server.go:379 Vacuum plugin worker disconnected, re-enabling master automatic vacuum --- PASS: TestSyncVacuumState (0.00s) --- PASS: TestSyncVacuumState/no_change_when_state_matches (0.00s) --- PASS: TestSyncVacuumState/worker_connects_triggers_disable (0.00s) --- PASS: TestSyncVacuumState/worker_disconnects_triggers_enable (0.00s) === CONT TestFilterActivitiesByLane --- PASS: TestFilterActivitiesByLane (0.00s) === CONT TestFilterTrackedJobsByLane --- PASS: TestFilterTrackedJobsByLane (0.00s) --- PASS: TestVacuumMonitorStateTransitions (0.02s) --- PASS: TestAdminLockManagerFairnessWindowForcesFullRelease (0.09s) --- PASS: TestAdminLockManagerYieldsBeforeReacquire (0.11s) --- PASS: TestAdminLockManagerAcquireWaitsForReleaseInFlight (0.11s) PASS ok github.com/seaweedfs/seaweedfs/weed/admin/dash 9.521s === RUN TestSetupRoutes_RegistersPluginSchedulerStatesAPI_NoAuth --- PASS: TestSetupRoutes_RegistersPluginSchedulerStatesAPI_NoAuth (0.08s) === RUN TestSetupRoutes_RegistersPluginSchedulerStatesAPI_WithAuth --- PASS: TestSetupRoutes_RegistersPluginSchedulerStatesAPI_WithAuth (0.05s) === RUN TestSetupRoutes_RegistersBucketLifecycleAPI_NoAuth --- PASS: TestSetupRoutes_RegistersBucketLifecycleAPI_NoAuth (0.04s) === RUN TestSetupRoutes_RegistersBucketLifecycleAPI_WithAuth --- PASS: TestSetupRoutes_RegistersBucketLifecycleAPI_WithAuth (0.05s) === RUN TestSetupRoutes_RegistersPluginPages_NoAuth --- PASS: TestSetupRoutes_RegistersPluginPages_NoAuth (0.05s) === RUN TestSetupRoutes_RegistersPluginPages_WithAuth --- PASS: TestSetupRoutes_RegistersPluginPages_WithAuth (0.10s) === RUN TestValidateAndCleanFilePath_AllowsControlChars --- PASS: TestValidateAndCleanFilePath_AllowsControlChars (0.00s) === RUN TestValidateAndCleanFilePath_RejectsEmpty --- PASS: TestValidateAndCleanFilePath_RejectsEmpty (0.00s) PASS ok github.com/seaweedfs/seaweedfs/weed/admin/handlers 12.116s ? github.com/seaweedfs/seaweedfs/weed/admin/internal/httputil [no test files] === RUN TestMaintenanceManager_ErrorHandling E0824 22:15:21.833024 maintenance_manager.go:393 Maintenance scan failed: dial tcp [::1]:19333: connect: connection refused (will retry with backoff) E0824 22:15:21.852295 maintenance_manager.go:395 Maintenance scan still failing after 2 attempts: dial tcp [::1]:19333: connect: connection refused (backoff: 2s) E0824 22:15:21.863292 maintenance_manager.go:395 Maintenance scan still failing after 3 attempts: dial tcp [::1]:19333: connect: connection refused (backoff: 4s) E0824 22:15:21.863516 maintenance_manager.go:395 Maintenance scan still failing after 6 attempts: dial tcp [::1]:19333: connect: connection refused (backoff: 32s) E0824 22:15:21.863603 maintenance_manager.go:395 Maintenance scan still failing after 9 attempts: dial tcp [::1]:19333: connect: connection refused (backoff: 4m16s) E0824 22:15:21.863644 maintenance_manager.go:395 Maintenance scan still failing after 10 attempts: dial tcp [::1]:19333: connect: connection refused (backoff: 5m0s) --- PASS: TestMaintenanceManager_ErrorHandling (0.06s) === RUN TestIsConnectionError --- PASS: TestIsConnectionError (0.00s) === RUN TestMaintenanceManager_GetErrorState E0824 22:15:21.865122 maintenance_manager.go:399 Maintenance scan failed: test error E0824 22:15:21.865303 maintenance_manager.go:399 Maintenance scan failed: test error --- PASS: TestMaintenanceManager_GetErrorState (0.00s) === RUN TestMaintenanceManager_LogThrottling E0824 22:15:21.865801 maintenance_manager.go:399 Maintenance scan failed: test error E0824 22:15:21.866011 maintenance_manager.go:399 Maintenance scan failed: test error E0824 22:15:21.866091 maintenance_manager.go:399 Maintenance scan failed: test error E0824 22:15:21.866158 maintenance_manager.go:399 Maintenance scan failed: test error E0824 22:15:21.866234 maintenance_manager.go:399 Maintenance scan failed: test error E0824 22:15:21.866388 maintenance_manager.go:399 Maintenance scan failed: test error E0824 22:15:21.866451 maintenance_manager.go:399 Maintenance scan failed: test error --- PASS: TestMaintenanceManager_LogThrottling (0.00s) === RUN TestCanScheduleTaskNow_FallbackLogic --- PASS: TestCanScheduleTaskNow_FallbackLogic (0.00s) === RUN TestCanScheduleTaskNow_FallbackWithRunningTasks --- PASS: TestCanScheduleTaskNow_FallbackWithRunningTasks (0.00s) === RUN TestCanScheduleTaskNow_DifferentTaskTypes --- PASS: TestCanScheduleTaskNow_DifferentTaskTypes (0.00s) === RUN TestCanScheduleTaskNow_WithIntegration --- PASS: TestCanScheduleTaskNow_WithIntegration (0.02s) === RUN TestGetRunningTaskCount --- PASS: TestGetRunningTaskCount (0.00s) === RUN TestCanExecuteTaskType --- PASS: TestCanExecuteTaskType (0.00s) === RUN TestGetMaxConcurrentForTaskType_DefaultBehavior --- PASS: TestGetMaxConcurrentForTaskType_DefaultBehavior (0.00s) === RUN TestCanScheduleTaskNow_NilTask --- PASS: TestCanScheduleTaskNow_NilTask (0.03s) === RUN TestCanScheduleTaskNow_EmptyTaskType --- PASS: TestCanScheduleTaskNow_EmptyTaskType (0.00s) === RUN TestCanScheduleTaskNow_WithPolicy --- PASS: TestCanScheduleTaskNow_WithPolicy (0.03s) === RUN TestMaintenanceQueue_TaskIDPreservation I0824 22:15:21.955739 maintenance_queue.go:205 Task queued: ec_task_123 (erasure_coding) volume 100 on server1, priority 1, reason: I0824 22:15:21.956108 maintenance_queue.go:205 Task queued: manual_id_456 (vacuum) volume 0 on , priority 0, reason: --- PASS: TestMaintenanceQueue_TaskIDPreservation (0.00s) === RUN TestMaintenanceQueue_ActiveTopologySync I0824 22:15:21.957226 maintenance_queue.go:205 Task queued: sync_test_123 (balance) volume 100 on server1, priority 0, reason: I0824 22:15:21.973990 maintenance_queue.go:503 Task assigned: sync_test_123 (balance) → worker worker1 (volume 100, server server1) I0824 22:15:21.974678 maintenance_queue.go:610 Task completed: sync_test_123 (balance) worker worker1, duration 16.938334ms, volume 100 --- PASS: TestMaintenanceQueue_ActiveTopologySync (0.02s) === RUN TestMaintenanceQueue_StaleWorkerCapacityRelease I0824 22:15:21.975889 maintenance_queue.go:205 Task queued: stale_test_123 (balance) volume 100 on server1, priority 0, reason: I0824 22:15:21.976306 maintenance_queue.go:503 Task assigned: stale_test_123 (balance) → worker worker1 (volume 100, server server1) W0824 22:15:21.976494 maintenance_queue.go:984 Removed stale maintenance worker worker1 --- PASS: TestMaintenanceQueue_StaleWorkerCapacityRelease (0.00s) === RUN TestMaintenanceManager_CancelTaskCapacityRelease I0824 22:15:22.004536 maintenance_queue.go:205 Task queued: cancel_test_123 (balance) volume 100 on server1, priority 0, reason: --- PASS: TestMaintenanceManager_CancelTaskCapacityRelease (0.03s) === RUN TestMaintenanceQueue_LoadRequeuesInFlightTasks I0824 22:15:22.005799 maintenance_queue.go:96 Task queue initialized: re-queued 2 in-flight task(s) from persistence; scanner will re-detect the rest --- PASS: TestMaintenanceQueue_LoadRequeuesInFlightTasks (0.00s) === RUN TestMaintenanceQueue_RetryCapacitySync I0824 22:15:22.034609 maintenance_queue.go:205 Task queued: retry_test_123 (balance) volume 100 on server1, priority 0, reason: I0824 22:15:22.035137 maintenance_queue.go:503 Task assigned: retry_test_123 (balance) → worker worker1 (volume 100, server server1) W0824 22:15:22.035369 maintenance_queue.go:583 Task failed, scheduling retry: retry_test_123 (balance) attempt 1/3, worker worker1, duration 373.179µs, error: simulated failure --- PASS: TestMaintenanceQueue_RetryCapacitySync (0.03s) === RUN TestMaintenanceQueue_AssignTaskRollback I0824 22:15:22.036596 maintenance_queue.go:205 Task queued: rollback_test_123 (balance) volume 100 on server1, priority 0, reason: W0824 22:15:22.036865 maintenance_queue.go:470 Failed to update ActiveTopology for task assignment rollback_test_123: pending task rollback_test_123 not found. Rolling back assignment. --- PASS: TestMaintenanceQueue_AssignTaskRollback (0.00s) === RUN TestGetNextTask_SkipsVolumeConflictsAcrossTypes I0824 22:15:22.037751 maintenance_queue.go:205 Task queued: t1 (balance) volume 100 on server1, priority 2, reason: I0824 22:15:22.037956 maintenance_queue.go:205 Task queued: t3 (vacuum) volume 200 on server1, priority 1, reason: I0824 22:15:22.038127 maintenance_queue.go:503 Task assigned: t1 (balance) → worker worker1 (volume 100, server server1) I0824 22:15:22.038518 maintenance_queue.go:503 Task assigned: t3 (vacuum) → worker worker2 (volume 200, server server1) --- PASS: TestGetNextTask_SkipsVolumeConflictsAcrossTypes (0.00s) === RUN TestAddTask_OnePendingTaskPerVolume I0824 22:15:22.038855 maintenance_queue.go:205 Task queued: t1 (balance) volume 100 on server1, priority 0, reason: --- PASS: TestAddTask_OnePendingTaskPerVolume (0.00s) === RUN TestAddTask_RejectsWhenVolumeHasRunningTask I0824 22:15:22.039237 maintenance_queue.go:205 Task queued: t1 (balance) volume 100 on server1, priority 0, reason: --- PASS: TestAddTask_RejectsWhenVolumeHasRunningTask (0.00s) === RUN TestPendingOperations_ConflictDetection --- PASS: TestPendingOperations_ConflictDetection (0.00s) === RUN TestPendingOperations_CapacityProjection --- PASS: TestPendingOperations_CapacityProjection (0.00s) === RUN TestPendingOperations_VolumeFiltering --- PASS: TestPendingOperations_VolumeFiltering (0.00s) === RUN TestPendingOperations_OperationLifecycle --- PASS: TestPendingOperations_OperationLifecycle (0.00s) === RUN TestPendingOperations_StaleCleanup W0824 22:15:22.041870 pending_operations.go:265 Removed stale pending operation: volume 301, task task-stale, age 24h0m0.00004314s --- PASS: TestPendingOperations_StaleCleanup (0.00s) PASS ok github.com/seaweedfs/seaweedfs/weed/admin/maintenance 9.319s === RUN TestDecorateClusterContext_NonS3LifecycleIsPassThrough --- PASS: TestDecorateClusterContext_NonS3LifecycleIsPassThrough (0.04s) === RUN TestDecorateClusterContext_RpsZeroSkipsAllocation --- PASS: TestDecorateClusterContext_RpsZeroSkipsAllocation (0.00s) === RUN TestDecorateClusterContext_NoExecutorsSkipsAllocation --- PASS: TestDecorateClusterContext_NoExecutorsSkipsAllocation (0.00s) === RUN TestDecorateClusterContext_SingletonJobGetsFullBudget --- PASS: TestDecorateClusterContext_SingletonJobGetsFullBudget (0.07s) === RUN TestDecorateClusterContext_SharedEvenlyWhenParallelLimited --- PASS: TestDecorateClusterContext_SharedEvenlyWhenParallelLimited (0.00s) === RUN TestDecorateClusterContext_MaxJobsExceedsExecutors --- PASS: TestDecorateClusterContext_MaxJobsExceedsExecutors (0.00s) === RUN TestDecorateClusterContext_BurstSharedWhenParallel --- PASS: TestDecorateClusterContext_BurstSharedWhenParallel (0.00s) === RUN TestDecorateClusterContext_BurstZeroOmitsKey --- PASS: TestDecorateClusterContext_BurstZeroOmitsKey (0.00s) === RUN TestDecorateClusterContext_BurstFloorIsOneWhenDividesBelowOne --- PASS: TestDecorateClusterContext_BurstFloorIsOneWhenDividesBelowOne (0.00s) === RUN TestDecorateClusterContext_DoesNotMutateInput --- PASS: TestDecorateClusterContext_DoesNotMutateInput (0.00s) === RUN TestDecorateClusterContext_NilInputPassesThrough --- PASS: TestDecorateClusterContext_NilInputPassesThrough (0.00s) === RUN TestConfigStoreDescriptorRoundTrip === PAUSE TestConfigStoreDescriptorRoundTrip === RUN TestConfigStoreRunHistoryRetention === PAUSE TestConfigStoreRunHistoryRetention === RUN TestConfigStoreListJobTypes === PAUSE TestConfigStoreListJobTypes === RUN TestConfigStoreMonitorStateRoundTrip === PAUSE TestConfigStoreMonitorStateRoundTrip === RUN TestConfigStoreSaveJobTypeConfigIfNotExists === PAUSE TestConfigStoreSaveJobTypeConfigIfNotExists === RUN TestConfigStoreJobDetailRoundTrip === PAUSE TestConfigStoreJobDetailRoundTrip === RUN TestConfigStoreDeleteJobDetail === PAUSE TestConfigStoreDeleteJobDetail === RUN TestObservationAttributesAreFlattened --- PASS: TestObservationAttributesAreFlattened (0.04s) === RUN TestObservationIsReplacedByALaterOne --- PASS: TestObservationIsReplacedByALaterOne (0.00s) === RUN TestObservationAttributeStringHandlesAbsence --- PASS: TestObservationAttributeStringHandlesAbsence (0.00s) === RUN TestObservationLookupIsScopedToFormat --- PASS: TestObservationLookupIsScopedToFormat (0.00s) === RUN TestRunDetectionSendsCancelOnContextDone === PAUSE TestRunDetectionSendsCancelOnContextDone === RUN TestExecuteJobSendsCancelOnContextDone === PAUSE TestExecuteJobSendsCancelOnContextDone === RUN TestAdminScriptExecutionBlocksOtherDetection --- PASS: TestAdminScriptExecutionBlocksOtherDetection (0.12s) === RUN TestAdminScriptExecutionBlocksOtherExecution --- PASS: TestAdminScriptExecutionBlocksOtherExecution (0.12s) === RUN TestEnsureJobTypeConfigFromDescriptorBootstrapsDefaults === PAUSE TestEnsureJobTypeConfigFromDescriptorBootstrapsDefaults === RUN TestEnsureJobTypeConfigFromDescriptorDoesNotOverwriteExisting === PAUSE TestEnsureJobTypeConfigFromDescriptorDoesNotOverwriteExisting === RUN TestRunDetectionIncludesLatestSuccessfulRun --- PASS: TestRunDetectionIncludesLatestSuccessfulRun (0.00s) === RUN TestRunDetectionOmitsLastSuccessfulRunWhenNoSuccessHistory --- PASS: TestRunDetectionOmitsLastSuccessfulRunWhenNoSuccessHistory (0.00s) === RUN TestRunDetectionWithReportCapturesDetectionActivities --- PASS: TestRunDetectionWithReportCapturesDetectionActivities (0.00s) === RUN TestRunDetectionAdminScriptUsesLastCompletedRun --- PASS: TestRunDetectionAdminScriptUsesLastCompletedRun (0.00s) === RUN TestPluginLoadsPersistedMonitorStateOnStart === PAUSE TestPluginLoadsPersistedMonitorStateOnStart === RUN TestPluginPersistsMonitorStateAfterJobUpdates === PAUSE TestPluginPersistsMonitorStateAfterJobUpdates === RUN TestTrackExecutionQueuedMarksPendingState === PAUSE TestTrackExecutionQueuedMarksPendingState === RUN TestHandleJobProgressUpdateCarriesWorkerIDInActivities === PAUSE TestHandleJobProgressUpdateCarriesWorkerIDInActivities === RUN TestHandleJobProgressUpdateWithoutJobIDTracksDetectionActivities === PAUSE TestHandleJobProgressUpdateWithoutJobIDTracksDetectionActivities === RUN TestHandleJobCompletedCarriesWorkerIDInActivitiesAndRunHistory === PAUSE TestHandleJobCompletedCarriesWorkerIDInActivitiesAndRunHistory === RUN TestTrackExecutionStartStoresJobPayloadDetails === PAUSE TestTrackExecutionStartStoresJobPayloadDetails === RUN TestTrackExecutionStartStoresErasureCodingExecutionPlan === PAUSE TestTrackExecutionStartStoresErasureCodingExecutionPlan === RUN TestBuildJobDetailIncludesActivitiesAndRunRecord === PAUSE TestBuildJobDetailIncludesActivitiesAndRunRecord === RUN TestBuildJobDetailLoadsFromDiskWhenMemoryCleared === PAUSE TestBuildJobDetailLoadsFromDiskWhenMemoryCleared === RUN TestLoadSchedulerPolicyUsesAdminConfig === PAUSE TestLoadSchedulerPolicyUsesAdminConfig === RUN TestLoadSchedulerPolicyUsesDescriptorDefaultsWhenConfigMissing === PAUSE TestLoadSchedulerPolicyUsesDescriptorDefaultsWhenConfigMissing === RUN TestReserveScheduledExecutorRespectsPerWorkerLimit === PAUSE TestReserveScheduledExecutorRespectsPerWorkerLimit === RUN TestFilterScheduledProposalsDedupe === PAUSE TestFilterScheduledProposalsDedupe === RUN TestBuildScheduledJobSpecDoesNotReuseProposalID === PAUSE TestBuildScheduledJobSpecDoesNotReuseProposalID === RUN TestFilterProposalsWithActiveJobs === PAUSE TestFilterProposalsWithActiveJobs === RUN TestReserveScheduledExecutorTimesOutWhenNoExecutor === PAUSE TestReserveScheduledExecutorTimesOutWhenNoExecutor === RUN TestReserveScheduledExecutorWaitsForWorkerCapacity === PAUSE TestReserveScheduledExecutorWaitsForWorkerCapacity === RUN TestShouldSkipDetectionForWaitingJobs === PAUSE TestShouldSkipDetectionForWaitingJobs === RUN TestWaitingBacklogThresholdHonorsMaxResultsCap === PAUSE TestWaitingBacklogThresholdHonorsMaxResultsCap === RUN TestListSchedulerStatesIncludesPolicyAndState === PAUSE TestListSchedulerStatesIncludesPolicyAndState === RUN TestListSchedulerStatesShowsDisabledWhenNoPolicy === PAUSE TestListSchedulerStatesShowsDisabledWhenNoPolicy === RUN TestPickDetectorPrefersLeasedWorker === PAUSE TestPickDetectorPrefersLeasedWorker === RUN TestPickDetectorReassignsWhenLeaseIsStale === PAUSE TestPickDetectorReassignsWhenLeaseIsStale === RUN TestRunLaneSchedulerIterationLockBehavior === PAUSE TestRunLaneSchedulerIterationLockBehavior === RUN TestDispatchScheduledProposalsLocksPerJob === PAUSE TestDispatchScheduledProposalsLocksPerJob === RUN TestScheduledAttemptContextDrainGrace === PAUSE TestScheduledAttemptContextDrainGrace === RUN TestScheduledEstimatedRuntime === PAUSE TestScheduledEstimatedRuntime === RUN TestDispatchScheduledProposalsDrainsStartedJobOnWindowClose === PAUSE TestDispatchScheduledProposalsDrainsStartedJobOnWindowClose === RUN TestPruneSchedulerState_DefaultLaneKeepsForeignLanesAndPrunesOwnStale --- PASS: TestPruneSchedulerState_DefaultLaneKeepsForeignLanesAndPrunesOwnStale (0.01s) === RUN TestPruneSchedulerState_LifecycleLaneLeavesDefaultLane --- PASS: TestPruneSchedulerState_LifecycleLaneLeavesDefaultLane (0.00s) === RUN TestPruneDetectorLeases_IsLaneScoped --- PASS: TestPruneDetectorLeases_IsLaneScoped (0.00s) === RUN TestLaneStatus_LifecycleNextDetectionSurvivesDefaultLanePrune --- PASS: TestLaneStatus_LifecycleNextDetectionSurvivesDefaultLanePrune (0.00s) === RUN TestRequestObjectPreviewReturnsTheWorkersRows === PAUSE TestRequestObjectPreviewReturnsTheWorkersRows === RUN TestRequestObjectPreviewSurfacesTheWorkersError === PAUSE TestRequestObjectPreviewSurfacesTheWorkersError === RUN TestRequestObjectPreviewNeedsAnObservation === PAUSE TestRequestObjectPreviewNeedsAnObservation === RUN TestRequestObjectPreviewNeedsAConnectedWorker === PAUSE TestRequestObjectPreviewNeedsAConnectedWorker === RUN TestObjectPreviewResponseRacesShutdown === PAUSE TestObjectPreviewResponseRacesShutdown === RUN TestRegistryPickDetectorPrefersMoreFreeSlots === PAUSE TestRegistryPickDetectorPrefersMoreFreeSlots === RUN TestRegistryPickExecutorAllowsSameWorker === PAUSE TestRegistryPickExecutorAllowsSameWorker === RUN TestRegistryDetectableJobTypes === PAUSE TestRegistryDetectableJobTypes === RUN TestRegistryJobTypes === PAUSE TestRegistryJobTypes === RUN TestRegistryListExecutorsSortedBySlots === PAUSE TestRegistryListExecutorsSortedBySlots === RUN TestRegistryPickExecutorRoundRobinForTopTie === PAUSE TestRegistryPickExecutorRoundRobinForTopTie === RUN TestRegistryListExecutorsRoundRobinForTopTie === PAUSE TestRegistryListExecutorsRoundRobinForTopTie === RUN TestRegistrySkipsStaleWorkersForSelectionAndListing === PAUSE TestRegistrySkipsStaleWorkersForSelectionAndListing === RUN TestRegistryReturnsNoDetectorWhenAllWorkersStale === PAUSE TestRegistryReturnsNoDetectorWhenAllWorkersStale === RUN TestRegistryHasCapableWorkerWithDetectCapability === PAUSE TestRegistryHasCapableWorkerWithDetectCapability === RUN TestRegistryHasCapableWorkerWithExecuteCapability === PAUSE TestRegistryHasCapableWorkerWithExecuteCapability === RUN TestRegistryHasCapableWorkerReturnsFalseWhenNoWorkers === PAUSE TestRegistryHasCapableWorkerReturnsFalseWhenNoWorkers === RUN TestRegistryHasCapableWorkerSkipsStaleWorkers === PAUSE TestRegistryHasCapableWorkerSkipsStaleWorkers === RUN TestRegistryHasCapableWorkerIgnoresNilCapability === PAUSE TestRegistryHasCapableWorkerIgnoresNilCapability === RUN TestJobTypeLaneMapCoversKnownTypes --- PASS: TestJobTypeLaneMapCoversKnownTypes (0.00s) === RUN TestJobTypeLaneFallsBackToDefault --- PASS: TestJobTypeLaneFallsBackToDefault (0.00s) === RUN TestAllLanesHaveIdleSleep --- PASS: TestAllLanesHaveIdleSleep (0.00s) === RUN TestLaneRequiresLock === RUN TestLaneRequiresLock/Default === RUN TestLaneRequiresLock/Iceberg === RUN TestLaneRequiresLock/Lifecycle === RUN TestLaneRequiresLock/Unknown --- PASS: TestLaneRequiresLock (0.00s) --- PASS: TestLaneRequiresLock/Default (0.00s) --- PASS: TestLaneRequiresLock/Iceberg (0.00s) --- PASS: TestLaneRequiresLock/Lifecycle (0.00s) --- PASS: TestLaneRequiresLock/Unknown (0.00s) === RUN TestKnownJobTypesInMap --- PASS: TestKnownJobTypesInMap (0.00s) === RUN TestGetSchedulerStatusIncludesInProcessJobs --- PASS: TestGetSchedulerStatusIncludesInProcessJobs (0.01s) === RUN TestGetSchedulerStatusIncludesLastDetectionCount --- PASS: TestGetSchedulerStatusIncludesLastDetectionCount (0.00s) === RUN TestGetLaneSchedulerStatusShowsActiveConcurrentLaneWork --- PASS: TestGetLaneSchedulerStatusShowsActiveConcurrentLaneWork (0.02s) === CONT TestConfigStoreDescriptorRoundTrip === CONT TestRegistryHasCapableWorkerWithDetectCapability === CONT TestReserveScheduledExecutorTimesOutWhenNoExecutor --- PASS: TestRegistryHasCapableWorkerWithDetectCapability (0.00s) === CONT TestHandleJobProgressUpdateCarriesWorkerIDInActivities === CONT TestRegistryHasCapableWorkerWithExecuteCapability --- PASS: TestRegistryHasCapableWorkerWithExecuteCapability (0.00s) === CONT TestLoadSchedulerPolicyUsesAdminConfig --- PASS: TestReserveScheduledExecutorTimesOutWhenNoExecutor (0.00s) === CONT TestRegistryReturnsNoDetectorWhenAllWorkersStale --- PASS: TestLoadSchedulerPolicyUsesAdminConfig (0.00s) === CONT TestRegistrySkipsStaleWorkersForSelectionAndListing --- PASS: TestRegistryReturnsNoDetectorWhenAllWorkersStale (0.00s) === CONT TestFilterProposalsWithActiveJobs --- PASS: TestRegistrySkipsStaleWorkersForSelectionAndListing (0.00s) === CONT TestRegistryListExecutorsRoundRobinForTopTie --- PASS: TestRegistryListExecutorsRoundRobinForTopTie (0.00s) === CONT TestRegistryPickExecutorRoundRobinForTopTie === CONT TestBuildJobDetailLoadsFromDiskWhenMemoryCleared --- PASS: TestHandleJobProgressUpdateCarriesWorkerIDInActivities (0.00s) --- PASS: TestRegistryPickExecutorRoundRobinForTopTie (0.00s) === CONT TestRegistryListExecutorsSortedBySlots --- PASS: TestRegistryListExecutorsSortedBySlots (0.00s) === CONT TestRegistryJobTypes --- PASS: TestRegistryJobTypes (0.00s) === CONT TestRegistryDetectableJobTypes --- PASS: TestRegistryDetectableJobTypes (0.00s) === CONT TestReserveScheduledExecutorRespectsPerWorkerLimit --- PASS: TestFilterProposalsWithActiveJobs (0.00s) === CONT TestRegistryPickExecutorAllowsSameWorker --- PASS: TestRegistryPickExecutorAllowsSameWorker (0.00s) === CONT TestRegistryPickDetectorPrefersMoreFreeSlots --- PASS: TestReserveScheduledExecutorRespectsPerWorkerLimit (0.00s) === CONT TestLoadSchedulerPolicyUsesDescriptorDefaultsWhenConfigMissing --- PASS: TestRegistryPickDetectorPrefersMoreFreeSlots (0.00s) === CONT TestObjectPreviewResponseRacesShutdown === CONT TestPickDetectorReassignsWhenLeaseIsStale === CONT TestShouldSkipDetectionForWaitingJobs === CONT TestTrackExecutionQueuedMarksPendingState --- PASS: TestPickDetectorReassignsWhenLeaseIsStale (0.00s) === CONT TestRequestObjectPreviewNeedsAConnectedWorker --- PASS: TestTrackExecutionQueuedMarksPendingState (0.00s) === CONT TestRequestObjectPreviewNeedsAnObservation --- PASS: TestRequestObjectPreviewNeedsAConnectedWorker (0.00s) === CONT TestPluginPersistsMonitorStateAfterJobUpdates --- PASS: TestRequestObjectPreviewNeedsAnObservation (0.00s) === CONT TestRequestObjectPreviewSurfacesTheWorkersError === CONT TestRequestObjectPreviewReturnsTheWorkersRows --- PASS: TestShouldSkipDetectionForWaitingJobs (0.01s) --- PASS: TestRequestObjectPreviewSurfacesTheWorkersError (0.00s) === CONT TestPluginLoadsPersistedMonitorStateOnStart --- PASS: TestRequestObjectPreviewReturnsTheWorkersRows (0.00s) === CONT TestWaitingBacklogThresholdHonorsMaxResultsCap --- PASS: TestWaitingBacklogThresholdHonorsMaxResultsCap (0.00s) === CONT TestDispatchScheduledProposalsDrainsStartedJobOnWindowClose === CONT TestReserveScheduledExecutorWaitsForWorkerCapacity === CONT TestBuildJobDetailIncludesActivitiesAndRunRecord === CONT TestEnsureJobTypeConfigFromDescriptorBootstrapsDefaults --- PASS: TestEnsureJobTypeConfigFromDescriptorBootstrapsDefaults (0.00s) === CONT TestExecuteJobSendsCancelOnContextDone === CONT TestTrackExecutionStartStoresErasureCodingExecutionPlan === CONT TestRunDetectionSendsCancelOnContextDone === CONT TestListSchedulerStatesIncludesPolicyAndState --- PASS: TestExecuteJobSendsCancelOnContextDone (0.00s) --- PASS: TestRunDetectionSendsCancelOnContextDone (0.00s) === CONT TestConfigStoreListJobTypes --- PASS: TestConfigStoreListJobTypes (0.00s) === CONT TestPickDetectorPrefersLeasedWorker --- PASS: TestLoadSchedulerPolicyUsesDescriptorDefaultsWhenConfigMissing (0.02s) === CONT TestRegistryHasCapableWorkerSkipsStaleWorkers === CONT TestEnsureJobTypeConfigFromDescriptorDoesNotOverwriteExisting === CONT TestBuildScheduledJobSpecDoesNotReuseProposalID === CONT TestConfigStoreDeleteJobDetail === CONT TestConfigStoreMonitorStateRoundTrip === CONT TestScheduledEstimatedRuntime === CONT TestScheduledAttemptContextDrainGrace === CONT TestDispatchScheduledProposalsLocksPerJob === RUN TestDispatchScheduledProposalsLocksPerJob/DefaultLaneLocksEachJob === PAUSE TestDispatchScheduledProposalsLocksPerJob/DefaultLaneLocksEachJob === RUN TestDispatchScheduledProposalsLocksPerJob/LifecycleLaneNeedsNoLock === PAUSE TestDispatchScheduledProposalsLocksPerJob/LifecycleLaneNeedsNoLock === CONT TestDispatchScheduledProposalsLocksPerJob/DefaultLaneLocksEachJob === CONT TestRunLaneSchedulerIterationLockBehavior === RUN TestRunLaneSchedulerIterationLockBehavior/Default === CONT TestDispatchScheduledProposalsLocksPerJob/LifecycleLaneNeedsNoLock === CONT TestRegistryHasCapableWorkerReturnsFalseWhenNoWorkers === CONT TestConfigStoreJobDetailRoundTrip === PAUSE TestRunLaneSchedulerIterationLockBehavior/Default === RUN TestRunLaneSchedulerIterationLockBehavior/Iceberg === PAUSE TestRunLaneSchedulerIterationLockBehavior/Iceberg === RUN TestRunLaneSchedulerIterationLockBehavior/Lifecycle === PAUSE TestRunLaneSchedulerIterationLockBehavior/Lifecycle === CONT TestRunLaneSchedulerIterationLockBehavior/Default === CONT TestConfigStoreRunHistoryRetention === CONT TestRunLaneSchedulerIterationLockBehavior/Iceberg === CONT TestRunLaneSchedulerIterationLockBehavior/Lifecycle === CONT TestConfigStoreSaveJobTypeConfigIfNotExists === RUN TestConfigStoreSaveJobTypeConfigIfNotExists/in-memory === PAUSE TestConfigStoreSaveJobTypeConfigIfNotExists/in-memory === CONT TestHandleJobProgressUpdateWithoutJobIDTracksDetectionActivities === CONT TestHandleJobCompletedCarriesWorkerIDInActivitiesAndRunHistory === CONT TestFilterScheduledProposalsDedupe === CONT TestRegistryHasCapableWorkerIgnoresNilCapability === CONT TestListSchedulerStatesShowsDisabledWhenNoPolicy === CONT TestTrackExecutionStartStoresJobPayloadDetails === RUN TestConfigStoreSaveJobTypeConfigIfNotExists/on-disk === PAUSE TestConfigStoreSaveJobTypeConfigIfNotExists/on-disk --- PASS: TestPickDetectorPrefersLeasedWorker (0.00s) --- PASS: TestReserveScheduledExecutorWaitsForWorkerCapacity (0.06s) --- PASS: TestRegistryHasCapableWorkerSkipsStaleWorkers (0.00s) === CONT TestConfigStoreSaveJobTypeConfigIfNotExists/in-memory === CONT TestConfigStoreSaveJobTypeConfigIfNotExists/on-disk --- PASS: TestDispatchScheduledProposalsDrainsStartedJobOnWindowClose (0.11s) --- PASS: TestListSchedulerStatesIncludesPolicyAndState (0.10s) --- PASS: TestBuildScheduledJobSpecDoesNotReuseProposalID (0.00s) --- PASS: TestEnsureJobTypeConfigFromDescriptorDoesNotOverwriteExisting (0.02s) --- PASS: TestConfigStoreDeleteJobDetail (0.04s) --- PASS: TestBuildJobDetailLoadsFromDiskWhenMemoryCleared (0.19s) --- PASS: TestPluginLoadsPersistedMonitorStateOnStart (0.19s) --- PASS: TestBuildJobDetailIncludesActivitiesAndRunRecord (0.18s) --- PASS: TestConfigStoreMonitorStateRoundTrip (0.01s) --- PASS: TestScheduledEstimatedRuntime (0.00s) --- PASS: TestScheduledAttemptContextDrainGrace (0.00s) --- PASS: TestConfigStoreDescriptorRoundTrip (0.24s) --- PASS: TestRegistryHasCapableWorkerReturnsFalseWhenNoWorkers (0.00s) --- PASS: TestPluginPersistsMonitorStateAfterJobUpdates (0.25s) --- PASS: TestDispatchScheduledProposalsLocksPerJob (0.00s) --- PASS: TestDispatchScheduledProposalsLocksPerJob/DefaultLaneLocksEachJob (0.00s) --- PASS: TestDispatchScheduledProposalsLocksPerJob/LifecycleLaneNeedsNoLock (0.01s) --- PASS: TestConfigStoreJobDetailRoundTrip (0.01s) --- PASS: TestRunLaneSchedulerIterationLockBehavior (0.01s) --- PASS: TestRunLaneSchedulerIterationLockBehavior/Default (0.00s) --- PASS: TestRunLaneSchedulerIterationLockBehavior/Iceberg (0.00s) --- PASS: TestRunLaneSchedulerIterationLockBehavior/Lifecycle (0.00s) --- PASS: TestFilterScheduledProposalsDedupe (0.00s) --- PASS: TestHandleJobProgressUpdateWithoutJobIDTracksDetectionActivities (0.00s) --- PASS: TestHandleJobCompletedCarriesWorkerIDInActivitiesAndRunHistory (0.00s) --- PASS: TestRegistryHasCapableWorkerIgnoresNilCapability (0.00s) --- PASS: TestListSchedulerStatesShowsDisabledWhenNoPolicy (0.00s) --- PASS: TestTrackExecutionStartStoresErasureCodingExecutionPlan (0.24s) --- PASS: TestTrackExecutionStartStoresJobPayloadDetails (0.02s) --- PASS: TestObjectPreviewResponseRacesShutdown (0.30s) --- PASS: TestConfigStoreSaveJobTypeConfigIfNotExists (0.04s) --- PASS: TestConfigStoreSaveJobTypeConfigIfNotExists/in-memory (0.00s) --- PASS: TestConfigStoreSaveJobTypeConfigIfNotExists/on-disk (0.02s) --- PASS: TestConfigStoreRunHistoryRetention (0.24s) PASS ok github.com/seaweedfs/seaweedfs/weed/admin/plugin 6.602s === RUN TestActiveTopologyBasicOperations --- PASS: TestActiveTopologyBasicOperations (0.13s) === RUN TestActiveTopologyUpdate --- PASS: TestActiveTopologyUpdate (0.00s) === RUN TestTaskLifecycle --- PASS: TestTaskLifecycle (0.00s) === RUN TestTaskDetectionScenarios === RUN TestTaskDetectionScenarios/Empty_cluster_-_no_tasks_needed === RUN TestTaskDetectionScenarios/Unbalanced_cluster_-_balance_task_needed === RUN TestTaskDetectionScenarios/High_garbage_ratio_-_vacuum_task_needed === RUN TestTaskDetectionScenarios/Large_volumes_-_EC_task_needed === RUN TestTaskDetectionScenarios/Recent_tasks_-_no_immediate_re-detection --- PASS: TestTaskDetectionScenarios (0.00s) --- PASS: TestTaskDetectionScenarios/Empty_cluster_-_no_tasks_needed (0.00s) --- PASS: TestTaskDetectionScenarios/Unbalanced_cluster_-_balance_task_needed (0.00s) --- PASS: TestTaskDetectionScenarios/High_garbage_ratio_-_vacuum_task_needed (0.00s) --- PASS: TestTaskDetectionScenarios/Large_volumes_-_EC_task_needed (0.00s) --- PASS: TestTaskDetectionScenarios/Recent_tasks_-_no_immediate_re-detection (0.00s) === RUN TestTargetSelectionScenarios === RUN TestTargetSelectionScenarios/Balance_task_-_find_least_loaded_disk === RUN TestTargetSelectionScenarios/EC_task_-_find_multiple_available_disks === RUN TestTargetSelectionScenarios/Vacuum_task_-_cross-type_tasks_do_not_block_per_disk --- PASS: TestTargetSelectionScenarios (0.03s) --- PASS: TestTargetSelectionScenarios/Balance_task_-_find_least_loaded_disk (0.00s) --- PASS: TestTargetSelectionScenarios/EC_task_-_find_multiple_available_disks (0.00s) --- PASS: TestTargetSelectionScenarios/Vacuum_task_-_cross-type_tasks_do_not_block_per_disk (0.00s) === RUN TestDiskLoadCalculation --- PASS: TestDiskLoadCalculation (0.00s) === RUN TestCrossTypeTasksDoNotBlockPerDisk --- PASS: TestCrossTypeTasksDoNotBlockPerDisk (0.00s) === RUN TestECPlanningNotBlockedByUnrelatedBalance --- PASS: TestECPlanningNotBlockedByUnrelatedBalance (0.00s) === RUN TestECPlannerSeesEachPhysicalDisk --- PASS: TestECPlannerSeesEachPhysicalDisk (0.00s) === RUN TestVolumeIndexResolvesPhysicalDiskZero --- PASS: TestVolumeIndexResolvesPhysicalDiskZero (0.01s) === RUN TestECShardIndexResolvesPhysicalDiskZero --- PASS: TestECShardIndexResolvesPhysicalDiskZero (0.00s) === RUN TestPublicInterfaces --- PASS: TestPublicInterfaces (0.00s) === RUN TestDestinationPlanning === RUN TestDestinationPlanning/GetAvailableDisks_functionality === RUN TestDestinationPlanning/Topology_provides_planning_information --- PASS: TestDestinationPlanning (0.03s) --- PASS: TestDestinationPlanning/GetAvailableDisks_functionality (0.00s) --- PASS: TestDestinationPlanning/Topology_provides_planning_information (0.00s) === RUN TestGetDisksWithEffectiveCapacityNotCappedAtTenByLoad === PAUSE TestGetDisksWithEffectiveCapacityNotCappedAtTenByLoad === RUN TestCountTopologyResources_multiDiskPerNode --- PASS: TestCountTopologyResources_multiDiskPerNode (0.00s) === RUN TestStorageSlotChangeArithmetic --- PASS: TestStorageSlotChangeArithmetic (0.00s) === RUN TestStorageSlotChange --- PASS: TestStorageSlotChange (0.03s) === RUN TestStorageSlotChangeCapacityCalculation --- PASS: TestStorageSlotChangeCapacityCalculation (0.00s) === RUN TestGetDisksWithEffectiveCapacity_UnknownEmptyDiskFallback --- PASS: TestGetDisksWithEffectiveCapacity_UnknownEmptyDiskFallback (0.00s) === RUN TestECMultipleTargets storage_slot_test.go:512: EC operation distributed 14 shards across 3 targets storage_slot_test.go:513: Capacity impacts: EC source reserves with zero impact, Targets minimal (shards < 10) --- PASS: TestECMultipleTargets (0.00s) === RUN TestCapacityReservationCycle storage_slot_test.go:611: Capacity lifecycle with StorageSlotChange: Pending -> Assigned -> Released -> Applied storage_slot_test.go:612: Source: 10 -> 11 -> 11 -> 10 -> 11 (freed by pending balance, then applied) storage_slot_test.go:613: Target: 10 -> 9 -> 9 -> 10 -> 9 (reserved by pending, then applied) --- PASS: TestCapacityReservationCycle (0.00s) === RUN TestReplicatedVolumeECOperations storage_slot_test.go:766: Replicated volume EC operation: 3 source replicas, 14 EC shards distributed across 3 destinations storage_slot_test.go:768: Each source replica reserves with zero capacity impact, destinations receive EC shards --- PASS: TestReplicatedVolumeECOperations (0.00s) === RUN TestECWithOldShardCleanup storage_slot_test.go:919: EC operation with cleanup: 2 volume replicas + 2 old EC shard locations → 14 new EC shards storage_slot_test.go:921: Volume sources have zero impact, old EC shard sources free capacity, new destinations consume shard slots --- PASS: TestECWithOldShardCleanup (0.00s) === RUN TestDetailedCapacityCalculations storage_slot_test.go:997: Detailed capacity calculation: VolumeSlots=80, ShardSlots=-5 storage_slot_test.go:999: Capacity impact: VolumeSlots=0, ShardSlots=5 storage_slot_test.go:1001: Simple capacity (backward compatible): 80 --- PASS: TestDetailedCapacityCalculations (0.01s) === RUN TestStorageSlotChangeConversions storage_slot_test.go:1052: Conversion tests passed: 10 shards = 1 volume slot storage_slot_test.go:1053: Mixed capacity (2 volumes + 15 shards) = 3 equivalent volume slots storage_slot_test.go:1055: Available capacity (10 volumes) = 100 total shard slots storage_slot_test.go:1057: NOTE: This test adapts automatically to erasure_coding.DataShardsCount = 10 --- PASS: TestStorageSlotChangeConversions (0.00s) === CONT TestGetDisksWithEffectiveCapacityNotCappedAtTenByLoad --- PASS: TestGetDisksWithEffectiveCapacityNotCappedAtTenByLoad (0.00s) PASS ok github.com/seaweedfs/seaweedfs/weed/admin/topology 4.787s === RUN TestFileBrowserPathURLEscapesQueryPath --- PASS: TestFileBrowserPathURLEscapesQueryPath (0.00s) === RUN TestFileBrowserPageURLEscapesCursor --- PASS: TestFileBrowserPageURLEscapesCursor (0.00s) PASS ok github.com/seaweedfs/seaweedfs/weed/admin/view/app 13.233s ? github.com/seaweedfs/seaweedfs/weed/admin/view/components [no test files] ? github.com/seaweedfs/seaweedfs/weed/admin/view/layout [no test files] === RUN TestConcurrentAddRemoveNodes --- PASS: TestConcurrentAddRemoveNodes (0.02s) === RUN TestListClusterNodeUpdates --- PASS: TestListClusterNodeUpdates (0.00s) === RUN TestIsKnownNode --- PASS: TestIsKnownNode (0.00s) === RUN TestLockClientHostMatchesFilerRing --- PASS: TestLockClientHostMatchesFilerRing (0.05s) === RUN TestLockClientHostFallsBackToSeed --- PASS: TestLockClientHostFallsBackToSeed (0.00s) === RUN TestLockClientSetRingVersionGuard --- PASS: TestLockClientSetRingVersionGuard (0.00s) === RUN TestLockClientPrimaryForKey --- PASS: TestLockClientPrimaryForKey (0.00s) === RUN TestLockClientPriorOwnerForKey --- PASS: TestLockClientPriorOwnerForKey (0.00s) === RUN TestLockClientPriorOwnerForKeyExpires --- PASS: TestLockClientPriorOwnerForKeyExpires (0.05s) === RUN TestLockRingManager_BatchesRapidChanges I0824 22:15:17.424539 lock_ring_manager.go:139 LockRing: broadcasting ring update for group "default" version 1787609717423673012: [filer2:8888 filer3:8888 filer1:8888] --- PASS: TestLockRingManager_BatchesRapidChanges (0.21s) === RUN TestLockRingManager_DropAndJoinBatched I0824 22:15:17.536674 lock_ring_manager.go:139 LockRing: broadcasting ring update for group "default" version 1787609717536384333: [filer1:8888 filer2:8888 filer3:8888] I0824 22:15:17.644822 lock_ring_manager.go:139 LockRing: broadcasting ring update for group "default" version 1787609717644417135: [filer1:8888 filer2:8888 filer4:8888] --- PASS: TestLockRingManager_DropAndJoinBatched (0.21s) === RUN TestLockRingManager_VersionIncrements I0824 22:15:17.805202 lock_ring_manager.go:139 LockRing: broadcasting ring update for group "default" version 1787609717804684079: [filer1:8888] I0824 22:15:17.914411 lock_ring_manager.go:139 LockRing: broadcasting ring update for group "default" version 1787609717914003861: [filer1:8888 filer2:8888] --- PASS: TestLockRingManager_VersionIncrements (0.22s) === RUN TestLockRingManager_FlushPending I0824 22:15:17.967100 lock_ring_manager.go:139 LockRing: broadcasting ring update for group "default" version 1787609717966776522: [filer1:8888 filer2:8888] --- PASS: TestLockRingManager_FlushPending (0.00s) === RUN TestLockRingManager_MultipleGroups I0824 22:15:18.024407 lock_ring_manager.go:139 LockRing: broadcasting ring update for group "group2" version 1787609718023982922: [filer2:8888] I0824 22:15:18.024819 lock_ring_manager.go:139 LockRing: broadcasting ring update for group "group1" version 1787609718024256002: [filer1:8888] --- PASS: TestLockRingManager_MultipleGroups (0.11s) === RUN TestLockRingManager_GetServers --- PASS: TestLockRingManager_GetServers (0.00s) === RUN TestLockRingManager_NoBroadcastWithoutFn I0824 22:15:18.095260 lock_ring_manager.go:139 LockRing: broadcasting ring update for group "default" version 1787609718094508117: [filer1:8888] --- PASS: TestLockRingManager_NoBroadcastWithoutFn (0.06s) === RUN TestLockRingManager_GetLastUpdateReturnsBroadcastState I0824 22:15:18.136807 lock_ring_manager.go:139 LockRing: broadcasting ring update for group "default" version 1787609718136579483: [filer1:8888 filer2:8888] --- PASS: TestLockRingManager_GetLastUpdateReturnsBroadcastState (0.00s) PASS ok github.com/seaweedfs/seaweedfs/weed/cluster 5.410s === RUN TestDLM_PrimaryCrash_BackupPromotes --- PASS: TestDLM_PrimaryCrash_BackupPromotes (0.09s) === RUN TestDLM_BackupCrash_PrimaryContinues --- PASS: TestDLM_BackupCrash_PrimaryContinues (0.04s) === RUN TestDLM_BothPrimaryAndBackupCrash --- PASS: TestDLM_BothPrimaryAndBackupCrash (0.05s) === RUN TestDLM_RollingRestart distributed_lock_manager_test.go:337: Locks survived rolling restart: 2 / 5 --- PASS: TestDLM_RollingRestart (0.27s) === RUN TestDLM_GenerationIncrementsOnNewAcquisition --- PASS: TestDLM_GenerationIncrementsOnNewAcquisition (3.11s) === RUN TestDLM_ReplicationFailure_PrimaryStillWorks --- PASS: TestDLM_ReplicationFailure_PrimaryStillWorks (0.05s) === RUN TestDLM_UnlockReplicatesToBackup --- PASS: TestDLM_UnlockReplicatesToBackup (0.05s) === RUN TestDLM_TopologyChange_LockSurvivesServerAddition --- PASS: TestDLM_TopologyChange_LockSurvivesServerAddition (0.08s) === RUN TestDLM_ConsistentHashing_MinimalDisruption distributed_lock_manager_test.go:477: Lock distribution before: map[filer1:8888:7 filer2:8888:22 filer3:8888:21] --- PASS: TestDLM_ConsistentHashing_MinimalDisruption (0.92s) === RUN TestDLM_NodeDropAndJoin_OwnershipDisruption distributed_lock_manager_test.go:593: Locks disrupted from departed filer3: 44 / 100 distributed_lock_manager_test.go:594: Locks disrupted from surviving filer1/filer2: 14 / 100 distributed_lock_manager_test.go:595: Locks now on new filer4: 24 / 100 --- PASS: TestDLM_NodeDropAndJoin_OwnershipDisruption (2.16s) === RUN TestDLM_RenewalDuringTransferWindow distributed_lock_manager_test.go:671: Key transfer-window-lock-0: primary changed from filer3:8888 to filer4:8888, but renewal on old primary succeeded --- PASS: TestDLM_RenewalDuringTransferWindow (0.07s) === RUN TestDLM_StaleReplicationRejected --- PASS: TestDLM_StaleReplicationRejected (0.00s) === RUN TestHashRing_BasicOperations --- PASS: TestHashRing_BasicOperations (0.00s) === RUN TestHashRing_DuplicateAddRemove --- PASS: TestHashRing_DuplicateAddRemove (0.00s) === RUN TestHashRing_SetServers --- PASS: TestHashRing_SetServers (0.00s) === RUN TestHashRing_ConsistencyOnRemoval hash_ring_test.go:101: Keys that moved: 380 / 1000 --- PASS: TestHashRing_ConsistencyOnRemoval (0.11s) === RUN TestHashRing_BackupIsSuccessor --- PASS: TestHashRing_BackupIsSuccessor (0.49s) === RUN TestHashRing_Distribution hash_ring_test.go:150: Server filer2:8888: 1073 keys (7.3% deviation) hash_ring_test.go:150: Server filer1:8888: 677 keys (32.3% deviation) hash_ring_test.go:150: Server filer3:8888: 1250 keys (25.0% deviation) --- PASS: TestHashRing_Distribution (0.01s) === RUN TestHashRing_GetPrimary --- PASS: TestHashRing_GetPrimary (0.00s) === RUN TestHashRing_OrderIndependent --- PASS: TestHashRing_OrderIndependent (0.04s) === RUN TestLockManager_GetLockOwnerIgnoresExpiredLock --- PASS: TestLockManager_GetLockOwnerIgnoresExpiredLock (0.00s) === RUN TestLockManager_InsertLockRejectsStaleTransfer --- PASS: TestLockManager_InsertLockRejectsStaleTransfer (0.00s) === RUN TestLockManager_InsertLockAdvancesGenerationCounter --- PASS: TestLockManager_InsertLockAdvancesGenerationCounter (0.00s) === RUN TestLockManager_InsertBackupLockRejectsOlderGeneration --- PASS: TestLockManager_InsertBackupLockRejectsOlderGeneration (0.00s) === RUN TestLockManager_InsertBackupLockKeepsPrimaryRole --- PASS: TestLockManager_InsertBackupLockKeepsPrimaryRole (0.00s) === RUN TestLockManager_RemoveBackupLockRejectsOlderGeneration --- PASS: TestLockManager_RemoveBackupLockRejectsOlderGeneration (0.00s) === RUN TestLockRing_PriorOwner --- PASS: TestLockRing_PriorOwner (5.03s) === RUN TestLockRing_PriorOwnerExpires --- PASS: TestLockRing_PriorOwnerExpires (0.05s) === RUN TestLockRing_SetSnapshot --- PASS: TestLockRing_SetSnapshot (0.24s) === RUN TestLockRing_SnapshotCompaction --- PASS: TestLockRing_SnapshotCompaction (0.24s) === RUN TestLockRing_VersionRejectsStale I0824 22:15:32.211005 lock_ring.go:53 LockRing: rejecting stale update v2 (current v3) --- PASS: TestLockRing_VersionRejectsStale (0.02s) PASS ok github.com/seaweedfs/seaweedfs/weed/cluster/lock_manager 19.436s ? github.com/seaweedfs/seaweedfs/weed/cluster/maintenance [no test files] === RUN TestIsIgnorable404_WrappedErrNotFound --- PASS: TestIsIgnorable404_WrappedErrNotFound (0.08s) === RUN TestIsIgnorable404_BrokenUnwrapChain --- PASS: TestIsIgnorable404_BrokenUnwrapChain (0.01s) === RUN TestIsIgnorable404_NonIgnorableError --- PASS: TestIsIgnorable404_NonIgnorableError (0.01s) === RUN TestIsIgnorable404_TransientLookupNotSwallowed === RUN TestIsIgnorable404_TransientLookupNotSwallowed/lookup_file_id_race === RUN TestIsIgnorable404_TransientLookupNotSwallowed/volume_id_not_found_race --- PASS: TestIsIgnorable404_TransientLookupNotSwallowed (0.00s) --- PASS: TestIsIgnorable404_TransientLookupNotSwallowed/lookup_file_id_race (0.00s) --- PASS: TestIsIgnorable404_TransientLookupNotSwallowed/volume_id_not_found_race (0.00s) === RUN TestErrorClassifiersNilSafe --- PASS: TestErrorClassifiersNilSafe (0.00s) === RUN TestIsSourceLookupError_NonLookupErrors === RUN TestIsSourceLookupError_NonLookupErrors/genuine_S3_404 === RUN TestIsSourceLookupError_NonLookupErrors/network_error === RUN TestIsSourceLookupError_NonLookupErrors/plain_not_found_without_volume_id --- PASS: TestIsSourceLookupError_NonLookupErrors (0.00s) --- PASS: TestIsSourceLookupError_NonLookupErrors/genuine_S3_404 (0.00s) --- PASS: TestIsSourceLookupError_NonLookupErrors/network_error (0.00s) --- PASS: TestIsSourceLookupError_NonLookupErrors/plain_not_found_without_volume_id (0.00s) === RUN TestEventSupersessionProbe_PathDerivation === RUN TestEventSupersessionProbe_PathDerivation/legacy_event_without_NewParentPath === RUN TestEventSupersessionProbe_PathDerivation/rename_event_with_NewParentPath --- PASS: TestEventSupersessionProbe_PathDerivation (0.00s) --- PASS: TestEventSupersessionProbe_PathDerivation/legacy_event_without_NewParentPath (0.00s) --- PASS: TestEventSupersessionProbe_PathDerivation/rename_event_with_NewParentPath (0.00s) === RUN TestEventSourceSuperseded_Guards --- PASS: TestEventSourceSuperseded_Guards (0.00s) === RUN TestInitialSnapshotTargetKey --- PASS: TestInitialSnapshotTargetKey (0.03s) === RUN TestLogicalPathResolvesVersionedLayout === RUN TestLogicalPathResolvesVersionedLayout/version_file_reports_the_object_key === RUN TestLogicalPathResolvesVersionedLayout/versions_directory_reports_the_object_key === RUN TestLogicalPathResolvesVersionedLayout/unversioned_object_is_unchanged === RUN TestLogicalPathResolvesVersionedLayout/v__prefix_outside_a_versions_directory_is_a_normal_name --- PASS: TestLogicalPathResolvesVersionedLayout (0.00s) --- PASS: TestLogicalPathResolvesVersionedLayout/version_file_reports_the_object_key (0.00s) --- PASS: TestLogicalPathResolvesVersionedLayout/versions_directory_reports_the_object_key (0.00s) --- PASS: TestLogicalPathResolvesVersionedLayout/unversioned_object_is_unchanged (0.00s) --- PASS: TestLogicalPathResolvesVersionedLayout/v__prefix_outside_a_versions_directory_is_a_normal_name (0.00s) === RUN TestLogicalPathRawKeepsStoredPath --- PASS: TestLogicalPathRawKeepsStoredPath (0.00s) === RUN TestToScanEventClassifiesOperations === RUN TestToScanEventClassifiesOperations/create === RUN TestToScanEventClassifiesOperations/delete === RUN TestToScanEventClassifiesOperations/update_in_place === RUN TestToScanEventClassifiesOperations/rename --- PASS: TestToScanEventClassifiesOperations (0.00s) --- PASS: TestToScanEventClassifiesOperations/create (0.00s) --- PASS: TestToScanEventClassifiesOperations/delete (0.00s) --- PASS: TestToScanEventClassifiesOperations/update_in_place (0.00s) --- PASS: TestToScanEventClassifiesOperations/rename (0.00s) === RUN TestDescribeEntryMarksDeleteMarkers --- PASS: TestDescribeEntryMarksDeleteMarkers (0.00s) === RUN TestParseScanTimeHonoursTimezone --- PASS: TestParseScanTimeHonoursTimezone (0.01s) === RUN TestVersionedFilePathRewrittenForRemote --- PASS: TestVersionedFilePathRewrittenForRemote (0.00s) === RUN TestVersionsDirectoryFilteredByHasData --- PASS: TestVersionsDirectoryFilteredByHasData (0.00s) === RUN TestIsVersionedPath === RUN TestIsVersionedPath/version_file_in_.versions_dir === RUN TestIsVersionedPath/regular_file === RUN TestIsVersionedPath/.versions_directory_entry === RUN TestIsVersionedPath/non-version_file_in_.versions_dir === RUN TestIsVersionedPath/file_in_user_dir_ending_with_.versions_but_no_v__prefix === RUN TestIsVersionedPath/regular_directory === RUN TestIsVersionedPath/file_named_like_.versions_dir === RUN TestIsVersionedPath/.versions_dir_outside_/buckets/ === RUN TestIsVersionedPath/v__file_outside_/buckets/ --- PASS: TestIsVersionedPath (0.00s) --- PASS: TestIsVersionedPath/version_file_in_.versions_dir (0.00s) --- PASS: TestIsVersionedPath/regular_file (0.00s) --- PASS: TestIsVersionedPath/.versions_directory_entry (0.00s) --- PASS: TestIsVersionedPath/non-version_file_in_.versions_dir (0.00s) --- PASS: TestIsVersionedPath/file_in_user_dir_ending_with_.versions_but_no_v__prefix (0.00s) --- PASS: TestIsVersionedPath/regular_directory (0.00s) --- PASS: TestIsVersionedPath/file_named_like_.versions_dir (0.00s) --- PASS: TestIsVersionedPath/.versions_dir_outside_/buckets/ (0.00s) --- PASS: TestIsVersionedPath/v__file_outside_/buckets/ (0.00s) === RUN TestDeleteMarkerDetectedBeforeHasDataFilter --- PASS: TestDeleteMarkerDetectedBeforeHasDataFilter (0.01s) === RUN TestRewriteVersionedSourcePath === RUN TestRewriteVersionedSourcePath/version_file_in_.versions_dir === RUN TestRewriteVersionedSourcePath/regular_file === RUN TestRewriteVersionedSourcePath/version_file_at_bucket_root === RUN TestRewriteVersionedSourcePath/non-bucket_path_not_rewritten === RUN TestRewriteVersionedSourcePath/non-bucket_mount_not_rewritten === RUN TestRewriteVersionedSourcePath/non-version_file_in_.versions_dir === RUN TestRewriteVersionedSourcePath/dir_not_ending_in_.versions --- PASS: TestRewriteVersionedSourcePath (0.01s) --- PASS: TestRewriteVersionedSourcePath/version_file_in_.versions_dir (0.00s) --- PASS: TestRewriteVersionedSourcePath/regular_file (0.00s) --- PASS: TestRewriteVersionedSourcePath/version_file_at_bucket_root (0.00s) --- PASS: TestRewriteVersionedSourcePath/non-bucket_path_not_rewritten (0.00s) --- PASS: TestRewriteVersionedSourcePath/non-bucket_mount_not_rewritten (0.00s) --- PASS: TestRewriteVersionedSourcePath/non-version_file_in_.versions_dir (0.00s) --- PASS: TestRewriteVersionedSourcePath/dir_not_ending_in_.versions (0.00s) === RUN TestPathAncestors --- PASS: TestPathAncestors (0.00s) === RUN TestFileVsFileConflict --- PASS: TestFileVsFileConflict (0.00s) === RUN TestFileUnderActiveDirConflict --- PASS: TestFileUnderActiveDirConflict (0.00s) === RUN TestDirWithActiveFileUnder --- PASS: TestDirWithActiveFileUnder (0.00s) === RUN TestDirVsDirConflict --- PASS: TestDirVsDirConflict (0.00s) === RUN TestRenameConflict --- PASS: TestRenameConflict (0.00s) === RUN TestActiveRenameConflict --- PASS: TestActiveRenameConflict (0.00s) === RUN TestRootDirConflict --- PASS: TestRootDirConflict (0.00s) === RUN TestIndexCleanup --- PASS: TestIndexCleanup (0.00s) === RUN TestWatermarkWithHeap --- PASS: TestWatermarkWithHeap (0.00s) === RUN TestNonBarrierDirUpdateDoesNotBlockDescendants === RUN TestNonBarrierDirUpdateDoesNotBlockDescendants/attribute_update_on_/dir1_does_not_block_file_under_it === RUN TestNonBarrierDirUpdateDoesNotBlockDescendants/barrier_dir_create_at_the_same_path_still_blocks_descendants === RUN TestNonBarrierDirUpdateDoesNotBlockDescendants/barrier_dir_delete_still_waits_for_in-flight_descendants === RUN TestNonBarrierDirUpdateDoesNotBlockDescendants/non-barrier_dir_update_still_keeps_ancestor_barrier_waiting --- PASS: TestNonBarrierDirUpdateDoesNotBlockDescendants (0.00s) --- PASS: TestNonBarrierDirUpdateDoesNotBlockDescendants/attribute_update_on_/dir1_does_not_block_file_under_it (0.00s) --- PASS: TestNonBarrierDirUpdateDoesNotBlockDescendants/barrier_dir_create_at_the_same_path_still_blocks_descendants (0.00s) --- PASS: TestNonBarrierDirUpdateDoesNotBlockDescendants/barrier_dir_delete_still_waits_for_in-flight_descendants (0.00s) --- PASS: TestNonBarrierDirUpdateDoesNotBlockDescendants/non-barrier_dir_update_still_keeps_ancestor_barrier_waiting (0.00s) === RUN TestSamePathBarrierSerialization === RUN TestSamePathBarrierSerialization/barrier_dir_at_p_blocks_same-path_file === RUN TestSamePathBarrierSerialization/barrier_dir_at_p_blocks_another_same-path_barrier_dir === RUN TestSamePathBarrierSerialization/barrier_dir_at_p_blocks_non-barrier_update_at_same_path === RUN TestSamePathBarrierSerialization/file_at_p_blocks_same-path_barrier_dir === RUN TestSamePathBarrierSerialization/non-barrier_update_at_p_blocks_incoming_barrier_dir_at_same_path === RUN TestSamePathBarrierSerialization/non-barrier_update_at_p_does_NOT_block_same-path_non-barrier_update --- PASS: TestSamePathBarrierSerialization (0.02s) --- PASS: TestSamePathBarrierSerialization/barrier_dir_at_p_blocks_same-path_file (0.01s) --- PASS: TestSamePathBarrierSerialization/barrier_dir_at_p_blocks_another_same-path_barrier_dir (0.00s) --- PASS: TestSamePathBarrierSerialization/barrier_dir_at_p_blocks_non-barrier_update_at_same_path (0.00s) --- PASS: TestSamePathBarrierSerialization/file_at_p_blocks_same-path_barrier_dir (0.00s) --- PASS: TestSamePathBarrierSerialization/non-barrier_update_at_p_blocks_incoming_barrier_dir_at_same_path (0.00s) --- PASS: TestSamePathBarrierSerialization/non-barrier_update_at_p_does_NOT_block_same-path_non-barrier_update (0.00s) === RUN TestMetadataProcessorEmptyMarkerKeepsWatermarkStale filer_sync_jobs_test.go:612: marker carried fresh ts 3601000000000 but watermark stayed stale at 1000000000 --- PASS: TestMetadataProcessorEmptyMarkerKeepsWatermarkStale (0.01s) === RUN TestFailedJobHoldsWatermark E0824 22:18:42.604380 filer_sync_jobs.go:274 process directory:"/dir" event_notification:{new_entry:{name:"b.txt"}} ts_ns:200: AccessDenied: Access Denied; holding sync offset at 1970-01-01 00:00:00.0000002 +0000 UTC so this event is replayed on restart --- PASS: TestFailedJobHoldsWatermark (0.03s) === RUN TestFailedJobHoldsWatermarkAtOldestFailure E0824 22:18:42.644307 filer_sync_jobs.go:274 process directory:"/dir" event_notification:{new_entry:{name:"f200.txt"}} ts_ns:200: AccessDenied: Access Denied; holding sync offset at 1970-01-01 00:00:00.0000002 +0000 UTC so this event is replayed on restart E0824 22:18:42.654634 filer_sync_jobs.go:276 process directory:"/dir" event_notification:{new_entry:{name:"f400.txt"}} ts_ns:400: AccessDenied: Access Denied --- PASS: TestFailedJobHoldsWatermarkAtOldestFailure (0.10s) === RUN TestDestKeyPreservesColonForNonLocalSink === RUN TestDestKeyPreservesColonForNonLocalSink/non-incremental === RUN TestDestKeyPreservesColonForNonLocalSink/incremental --- PASS: TestDestKeyPreservesColonForNonLocalSink (0.00s) --- PASS: TestDestKeyPreservesColonForNonLocalSink/non-incremental (0.00s) --- PASS: TestDestKeyPreservesColonForNonLocalSink/incremental (0.00s) === RUN TestPathIsEqualOrUnderUsesDirectoryBoundaries === RUN TestPathIsEqualOrUnderUsesDirectoryBoundaries/equal === RUN TestPathIsEqualOrUnderUsesDirectoryBoundaries/descendant === RUN TestPathIsEqualOrUnderUsesDirectoryBoundaries/sibling_prefix === RUN TestPathIsEqualOrUnderUsesDirectoryBoundaries/root === RUN TestPathIsEqualOrUnderUsesDirectoryBoundaries/empty --- PASS: TestPathIsEqualOrUnderUsesDirectoryBoundaries (0.00s) --- PASS: TestPathIsEqualOrUnderUsesDirectoryBoundaries/equal (0.00s) --- PASS: TestPathIsEqualOrUnderUsesDirectoryBoundaries/descendant (0.00s) --- PASS: TestPathIsEqualOrUnderUsesDirectoryBoundaries/sibling_prefix (0.00s) --- PASS: TestPathIsEqualOrUnderUsesDirectoryBoundaries/root (0.00s) --- PASS: TestPathIsEqualOrUnderUsesDirectoryBoundaries/empty (0.00s) === RUN TestMatchesExcludePathUsesDirectoryBoundaries --- PASS: TestMatchesExcludePathUsesDirectoryBoundaries (0.00s) === RUN TestGenProcessFunctionRenameCreatesThenDeletes --- PASS: TestGenProcessFunctionRenameCreatesThenDeletes (0.02s) === RUN TestGenProcessFunctionRenameUsesMoveEntryWhenSupported --- PASS: TestGenProcessFunctionRenameUsesMoveEntryWhenSupported (0.00s) === RUN TestGenProcessFunctionRenameMoverKeepsOldWhenDeletesDisabled --- PASS: TestGenProcessFunctionRenameMoverKeepsOldWhenDeletesDisabled (0.01s) === RUN TestGenProcessFunctionInPlaceUpdateUsesUpdateEntry --- PASS: TestGenProcessFunctionInPlaceUpdateUsesUpdateEntry (0.00s) === RUN TestGenProcessFunctionInPlaceUpdateFallbackDeletesBeforeCreate --- PASS: TestGenProcessFunctionInPlaceUpdateFallbackDeletesBeforeCreate (0.00s) === RUN TestGenProcessFunctionRenameCreateOnlyWhenDeletesDisabled --- PASS: TestGenProcessFunctionRenameCreateOnlyWhenDeletesDisabled (0.00s) === RUN TestGenProcessFunctionRenameToSiblingPrefixBecomesDelete --- PASS: TestGenProcessFunctionRenameToSiblingPrefixBecomesDelete (0.00s) === RUN TestGenProcessFunctionRenameFromExcludedDirBecomesCreate --- PASS: TestGenProcessFunctionRenameFromExcludedDirBecomesCreate (0.00s) === RUN TestStreamDeliveryBottleneck filer_sync_subscription_test.go:131: 1 stream: 2000 events 3.851s 519 events/sec filer_sync_subscription_test.go:136: 10 streams: 2000 events 162ms 12378 events/sec filer_sync_subscription_test.go:140: Speedup: 23.8x (10 parallel streams vs 1 stream) --- PASS: TestStreamDeliveryBottleneck (4.05s) === RUN TestConcurrencyIneffectiveOnStreamBottleneck filer_sync_subscription_test.go:174: concurrency= 32: 1000 events 1.091s 916 events/sec filer_sync_subscription_test.go:174: concurrency=128: 1000 events 1.144s 874 events/sec filer_sync_subscription_test.go:174: concurrency=512: 1000 events 2.393s 418 events/sec filer_sync_subscription_test.go:180: concurrency 512 vs 32: 0.46x (expected ~1.0x when stream-limited) filer_sync_subscription_test.go:183: unexpected ratio 0.46: concurrency should not affect stream-limited throughput --- FAIL: TestConcurrencyIneffectiveOnStreamBottleneck (4.64s) === RUN TestLogBufferSubscriptionThroughput === RUN TestLogBufferSubscriptionThroughput/single_subscriber_root filer_sync_subscription_test.go:259: 1 subscriber (/): 2000 events 17.467s 115 events/sec === RUN TestLogBufferSubscriptionThroughput/parallel_subscribers_per_dir filer_sync_subscription_test.go:315: 10 subscribers: 2000 events 498ms 4016 events/sec === NAME TestLogBufferSubscriptionThroughput filer_sync_subscription_test.go:325: LogBuffer speedup: 35.1x (10 parallel subscribers vs 1) --- PASS: TestLogBufferSubscriptionThroughput (18.30s) --- PASS: TestLogBufferSubscriptionThroughput/single_subscriber_root (17.47s) --- PASS: TestLogBufferSubscriptionThroughput/parallel_subscribers_per_dir (0.50s) === RUN TestVerifySyncMissingFile [MISSING] /root/file.txt (size=100, etag=d41d8cd98f00b204e9800998ecf8427e-0) --- PASS: TestVerifySyncMissingFile (0.01s) === RUN TestVerifySyncOnlyInB === RUN TestVerifySyncOnlyInB/bidirectional [ONLY_IN_B] /root/extra.txt === RUN TestVerifySyncOnlyInB/active-passive_ignores_onlyInB --- PASS: TestVerifySyncOnlyInB (0.02s) --- PASS: TestVerifySyncOnlyInB/bidirectional (0.00s) --- PASS: TestVerifySyncOnlyInB/active-passive_ignores_onlyInB (0.01s) === RUN TestVerifySyncSizeMismatch [SIZE_MISMATCH] /root/data.bin (a=1024, b=512, mtime equal [chunk-level issue]) --- PASS: TestVerifySyncSizeMismatch (0.00s) === RUN TestVerifySyncConcurrencyBound --- PASS: TestVerifySyncConcurrencyBound (0.07s) === RUN TestVerifySyncETagMismatch [ETAG_MISMATCH] /root/data.bin (a=112233, b=445566, mtime equal [chunk-level issue]) --- PASS: TestVerifySyncETagMismatch (0.00s) === RUN TestVerifySyncChunkReorder --- PASS: TestVerifySyncChunkReorder (0.01s) === RUN TestVerifySyncGenuineChunkDivergence [ETAG_MISMATCH] /root/model.bin (a=bc86f2b2489529a978fb72f5c7924a06-3, b=835086ae0fedc4b8173fed690601b0d0-3, mtime equal [chunk-level issue]) --- PASS: TestVerifySyncGenuineChunkDivergence (0.00s) === RUN TestVerifySyncChunkCountDiffStaysEtagMismatch [ETAG_MISMATCH] /root/data.bin (a=4054c6f7787ecb7b323323155b0c0583-3, b=be6996cb706f34d65b9fee8745325abf-2, mtime equal [chunk-level issue]) --- PASS: TestVerifySyncChunkCountDiffStaysEtagMismatch (0.00s) === RUN TestVerifySyncDuplicateOffsetStaysEtagMismatch [ETAG_MISMATCH] /root/data.bin (a=cb662da895d59679c42914df7a97a8f8-2, b=5ea5b42ba5a8f329a95a02fbb277546d-2, mtime equal [chunk-level issue]) --- PASS: TestVerifySyncDuplicateOffsetStaysEtagMismatch (0.00s) === RUN TestVerifySyncManifestChunkStaysEtagMismatch [ETAG_MISMATCH] /root/data.bin (a=0a5ffa6ef1e4da9211ba94feda315660-2, b=278c4fdf68ec7ad5e134331f06ea61f4-2, mtime equal [chunk-level issue]) --- PASS: TestVerifySyncManifestChunkStaysEtagMismatch (0.00s) === RUN TestVerifySyncEmptyChunkETagStaysEtagMismatch [ETAG_MISMATCH] /root/data.bin (a=2abb7e6461a0bdb2a4a2a2d16bbd4ae2-3, b=a356dbadc2bbb35f128fb27178f8a89b-3, mtime equal [chunk-level issue]) --- PASS: TestVerifySyncEmptyChunkETagStaysEtagMismatch (0.00s) === RUN TestVerifySyncChunkReorderJSONEmittedByDefault --- PASS: TestVerifySyncChunkReorderJSONEmittedByDefault (0.01s) === RUN TestVerifySyncCutoffTime === RUN TestVerifySyncCutoffTime/A-only_recent_file_is_skipped,_not_reported_missing === RUN TestVerifySyncCutoffTime/A-only_old_file_is_reported_missing [MISSING] /old.txt (size=10, etag=d41d8cd98f00b204e9800998ecf8427e-0) === RUN TestVerifySyncCutoffTime/B-only_recent_file_is_skipped,_not_reported_as_ONLY_IN_B === RUN TestVerifySyncCutoffTime/B-only_old_file_is_reported_as_ONLY_IN_B [ONLY_IN_B] /old.txt --- PASS: TestVerifySyncCutoffTime (0.00s) --- PASS: TestVerifySyncCutoffTime/A-only_recent_file_is_skipped,_not_reported_missing (0.00s) --- PASS: TestVerifySyncCutoffTime/A-only_old_file_is_reported_missing (0.00s) --- PASS: TestVerifySyncCutoffTime/B-only_recent_file_is_skipped,_not_reported_as_ONLY_IN_B (0.00s) --- PASS: TestVerifySyncCutoffTime/B-only_old_file_is_reported_as_ONLY_IN_B (0.00s) === RUN TestVerifySyncCutoffMatchedFileBSideRecent --- PASS: TestVerifySyncCutoffMatchedFileBSideRecent (0.00s) === RUN TestVerifySyncMissingDirRecursesEvenWithRecentMtime [MISSING] /subdir/ (directory) [MISSING] /subdir/old.txt (size=10, etag=d41d8cd98f00b204e9800998ecf8427e-0) --- PASS: TestVerifySyncMissingDirRecursesEvenWithRecentMtime (0.01s) === RUN TestVerifySyncRootPath --- PASS: TestVerifySyncRootPath (0.00s) === RUN TestVerifySyncNoDeadlockDeepTree --- PASS: TestVerifySyncNoDeadlockDeepTree (0.18s) === RUN TestVerifySyncByteOrderSkew --- PASS: TestVerifySyncByteOrderSkew (0.01s) === RUN TestVerifySyncSourcesListConcurrently --- PASS: TestVerifySyncSourcesListConcurrently (0.00s) === RUN TestVerifySyncListErrorNoBogusDiffs --- PASS: TestVerifySyncListErrorNoBogusDiffs (0.00s) === RUN TestFixEcxFromShards I0824 22:19:10.162855 ec_encoder.go:138 encodeDatFile /tmp/TestFixEcxFromShards1178091437/001/7.dat size:3000 with EC context 10+4 (total: 14) I0824 22:19:10.486771 fix.go:407 volume 7: no .dat size in .vif; reconstructing padded .dat (10485760 bytes) from 10 data shards I0824 22:19:10.542092 fix.go:426 volume 7: wrote /tmp/TestFixEcxFromShards1178091437/001/7.ecx from 10 data shards I0824 22:19:10.547875 fix.go:447 volume 7: wrote /tmp/TestFixEcxFromShards1178091437/001/7.vif (version 3, datFileSize 3000, ec 10+4) --- PASS: TestFixEcxFromShards (0.46s) === RUN TestFixEcxFromShardsWithVif I0824 22:19:10.577183 ec_encoder.go:138 encodeDatFile /tmp/TestFixEcxFromShardsWithVif2364536150/001/9.dat size:3000 with EC context 10+4 (total: 14) I0824 22:19:10.709509 fix.go:426 volume 9: wrote /tmp/TestFixEcxFromShardsWithVif2364536150/001/9.ecx from 10 data shards --- PASS: TestFixEcxFromShardsWithVif (0.15s) === RUN TestFixEcxFromShardsMissingShards I0824 22:19:10.742228 ec_encoder.go:138 encodeDatFile /tmp/TestFixEcxFromShardsMissingShards1325948251/001/11.dat size:3000 with EC context 10+4 (total: 14) I0824 22:19:10.882952 fix.go:384 volume 11: 11/14 shards present; reconstructing missing shards (10+4 (total: 14)) before index rebuild I0824 22:19:10.888726 ec_encoder.go:269 rebuilding /tmp/TestFixEcxFromShardsMissingShards1325948251/001/11: 11 shards present, 3 missing [2 5 11], config 10+4 (total: 14) I0824 22:19:10.951129 fix.go:407 volume 11: no .dat size in .vif; reconstructing padded .dat (10485760 bytes) from 10 data shards I0824 22:19:10.981830 fix.go:426 volume 11: wrote /tmp/TestFixEcxFromShardsMissingShards1325948251/001/11.ecx from 10 data shards I0824 22:19:10.982758 fix.go:447 volume 11: wrote /tmp/TestFixEcxFromShardsMissingShards1325948251/001/11.vif (version 3, datFileSize 3000, ec 10+4) --- PASS: TestFixEcxFromShardsMissingShards (0.27s) === RUN TestSaveToIdxWritesOffsetOrder --- PASS: TestSaveToIdxWritesOffsetOrder (0.01s) === RUN TestRunFuseDoesNotSkipOptionAfterConcurrentWriters --- PASS: TestRunFuseDoesNotSkipOptionAfterConcurrentWriters (0.01s) === RUN TestDefaultVolumeSizeLimitAlignsWithErasureCodingRows --- PASS: TestDefaultVolumeSizeLimitAlignsWithErasureCodingRows (0.00s) === RUN TestPeerIndexIgnoresGrpcPort --- PASS: TestPeerIndexIgnoresGrpcPort (0.00s) === RUN TestCheckPeersAddsSelfWhenGrpcPortMismatches I0824 22:19:11.021109 master.go:389 current: 127.0.0.1:9000 peers:127.0.0.1:9002,127.0.0.1:9003 --- PASS: TestCheckPeersAddsSelfWhenGrpcPortMismatches (0.00s) === RUN TestCheckPeersCanonicalizesSelfEntry I0824 22:19:11.021498 master.go:389 current: 127.0.0.1:9000 peers:127.0.0.1:9000,127.0.0.1:9002,127.0.0.1:9003 --- PASS: TestCheckPeersCanonicalizesSelfEntry (0.00s) === RUN TestCheckPeersDeduplicatesAliasPeers I0824 22:19:11.028341 master.go:389 current: 127.0.0.1:9000 peers:127.0.0.1:9002,127.0.0.1:9002.19002,127.0.0.1:9003 --- PASS: TestCheckPeersDeduplicatesAliasPeers (0.01s) === RUN TestMetadataEventDirectoryMembershipUsesDirectoryBoundaries --- PASS: TestMetadataEventDirectoryMembershipUsesDirectoryBoundaries (0.00s) === RUN TestMetadataEventUpdatesAndRemovesDirectory === RUN TestMetadataEventUpdatesAndRemovesDirectory/nil_response === RUN TestMetadataEventUpdatesAndRemovesDirectory/create_event === RUN TestMetadataEventUpdatesAndRemovesDirectory/rename_out === RUN TestMetadataEventUpdatesAndRemovesDirectory/rename_into === RUN TestMetadataEventUpdatesAndRemovesDirectory/rename_within === RUN TestMetadataEventUpdatesAndRemovesDirectory/delete --- PASS: TestMetadataEventUpdatesAndRemovesDirectory (0.00s) --- PASS: TestMetadataEventUpdatesAndRemovesDirectory/nil_response (0.00s) --- PASS: TestMetadataEventUpdatesAndRemovesDirectory/create_event (0.00s) --- PASS: TestMetadataEventUpdatesAndRemovesDirectory/rename_out (0.00s) --- PASS: TestMetadataEventUpdatesAndRemovesDirectory/rename_into (0.00s) --- PASS: TestMetadataEventUpdatesAndRemovesDirectory/rename_within (0.00s) --- PASS: TestMetadataEventUpdatesAndRemovesDirectory/delete (0.00s) === RUN TestApplyMiniAdminCredentialFallbackFromEnv --- PASS: TestApplyMiniAdminCredentialFallbackFromEnv (0.00s) === RUN TestParseBucketList === RUN TestParseBucketList/empty === RUN TestParseBucketList/single === RUN TestParseBucketList/multi === RUN TestParseBucketList/trims_whitespace === RUN TestParseBucketList/drops_empty_entries === RUN TestParseBucketList/dedupes_preserving_order === RUN TestParseBucketList/only_commas --- PASS: TestParseBucketList (0.01s) --- PASS: TestParseBucketList/empty (0.01s) --- PASS: TestParseBucketList/single (0.00s) --- PASS: TestParseBucketList/multi (0.00s) --- PASS: TestParseBucketList/trims_whitespace (0.00s) --- PASS: TestParseBucketList/drops_empty_entries (0.00s) --- PASS: TestParseBucketList/dedupes_preserving_order (0.00s) --- PASS: TestParseBucketList/only_commas (0.00s) === RUN TestParseTableBucketList === RUN TestParseTableBucketList/empty === RUN TestParseTableBucketList/defaults_to_iceberg === RUN TestParseTableBucketList/explicit_format === RUN TestParseTableBucketList/format_is_case_insensitive === RUN TestParseTableBucketList/mixed_formats === RUN TestParseTableBucketList/trims_whitespace === RUN TestParseTableBucketList/drops_unsupported_format W0824 22:19:11.040449 mini.go:2163 not creating table bucket vectors: unsupported format "DELTA" === RUN TestParseTableBucketList/empty_format_is_the_default === RUN TestParseTableBucketList/dedupes_by_name === RUN TestParseTableBucketList/drops_empty_entries --- PASS: TestParseTableBucketList (0.00s) --- PASS: TestParseTableBucketList/empty (0.00s) --- PASS: TestParseTableBucketList/defaults_to_iceberg (0.00s) --- PASS: TestParseTableBucketList/explicit_format (0.00s) --- PASS: TestParseTableBucketList/format_is_case_insensitive (0.00s) --- PASS: TestParseTableBucketList/mixed_formats (0.00s) --- PASS: TestParseTableBucketList/trims_whitespace (0.00s) --- PASS: TestParseTableBucketList/drops_unsupported_format (0.00s) --- PASS: TestParseTableBucketList/empty_format_is_the_default (0.00s) --- PASS: TestParseTableBucketList/dedupes_by_name (0.00s) --- PASS: TestParseTableBucketList/drops_empty_entries (0.00s) === RUN TestMiniServesTableFormat --- PASS: TestMiniServesTableFormat (0.00s) === RUN TestIcebergRoutingNames === RUN TestIcebergRoutingNames/none === RUN TestIcebergRoutingNames/iceberg_only === RUN TestIcebergRoutingNames/lance_is_not_routable === RUN TestIcebergRoutingNames/lance_first === RUN TestIcebergRoutingNames/order_preserved --- PASS: TestIcebergRoutingNames (0.00s) --- PASS: TestIcebergRoutingNames/none (0.00s) --- PASS: TestIcebergRoutingNames/iceberg_only (0.00s) --- PASS: TestIcebergRoutingNames/lance_is_not_routable (0.00s) --- PASS: TestIcebergRoutingNames/lance_first (0.00s) --- PASS: TestIcebergRoutingNames/order_preserved (0.00s) === RUN TestIcebergRoutingNamesFromEnvSpec --- PASS: TestIcebergRoutingNamesFromEnvSpec (0.00s) === RUN TestMiniDefaultPluginJobTypes --- PASS: TestMiniDefaultPluginJobTypes (0.00s) === RUN Test_bucketPathForMountRoot === RUN Test_bucketPathForMountRoot/bucket_root_mount === RUN Test_bucketPathForMountRoot/bucket_root_with_trailing_slash === RUN Test_bucketPathForMountRoot/subdirectory_mount === RUN Test_bucketPathForMountRoot/non-bucket_mount --- PASS: Test_bucketPathForMountRoot (0.00s) --- PASS: Test_bucketPathForMountRoot/bucket_root_mount (0.00s) --- PASS: Test_bucketPathForMountRoot/bucket_root_with_trailing_slash (0.00s) --- PASS: Test_bucketPathForMountRoot/subdirectory_mount (0.00s) --- PASS: Test_bucketPathForMountRoot/non-bucket_mount (0.00s) === RUN TestBuildPluginWorkerHandlerExplicitTypes --- PASS: TestBuildPluginWorkerHandlerExplicitTypes (0.00s) === RUN TestBuildPluginWorkerHandlerAliases --- PASS: TestBuildPluginWorkerHandlerAliases (0.00s) === RUN TestBuildPluginWorkerHandlerUnknown --- PASS: TestBuildPluginWorkerHandlerUnknown (0.00s) === RUN TestBuildPluginWorkerHandlers --- PASS: TestBuildPluginWorkerHandlers (0.00s) === RUN TestBuildPluginWorkerHandlersCategories --- PASS: TestBuildPluginWorkerHandlersCategories (0.00s) === RUN TestPluginWorkerDefaultJobTypes --- PASS: TestPluginWorkerDefaultJobTypes (0.00s) === RUN TestResolvePluginWorkerID I0824 22:19:11.073635 worker.go:148 Generated new worker ID: w-arch-n-e26a I0824 22:19:11.074126 worker.go:155 Saved worker ID to /tmp/TestResolvePluginWorkerID1535971858/001/worker.id I0824 22:19:11.074396 worker.go:110 Loaded existing worker ID from /tmp/TestResolvePluginWorkerID1535971858/001/worker.id: w-arch-n-e26a --- PASS: TestResolvePluginWorkerID (0.03s) === RUN TestParsePluginWorkerAdminAddress --- PASS: TestParsePluginWorkerAdminAddress (0.00s) === RUN TestResolvePluginWorkerAdminServerUsesStatusGrpcPort --- PASS: TestResolvePluginWorkerAdminServerUsesStatusGrpcPort (0.01s) === RUN TestResolvePluginWorkerAdminServerKeepsDefaultGrpcOffset --- PASS: TestResolvePluginWorkerAdminServerKeepsDefaultGrpcOffset (0.01s) === RUN TestDeriveS3AdvertisedEndpoint === RUN TestDeriveS3AdvertisedEndpoint/wildcard_bind_IP_with_no_externalUrl_does_not_advertise_an_endpoint === RUN TestDeriveS3AdvertisedEndpoint/empty_bind_IP_with_no_externalUrl_does_not_advertise_an_endpoint === RUN TestDeriveS3AdvertisedEndpoint/explicit_bind_IP_is_kept === RUN TestDeriveS3AdvertisedEndpoint/IPv6_literals_are_bracketed === RUN TestDeriveS3AdvertisedEndpoint/TLS_key_+_https_port_switches_scheme_and_port === RUN TestDeriveS3AdvertisedEndpoint/TLS-only_(no_separate_https_port)_still_uses_https_on_the_HTTP_port === RUN TestDeriveS3AdvertisedEndpoint/externalUrl_wins_and_trailing_slash_is_stripped --- PASS: TestDeriveS3AdvertisedEndpoint (0.00s) --- PASS: TestDeriveS3AdvertisedEndpoint/wildcard_bind_IP_with_no_externalUrl_does_not_advertise_an_endpoint (0.00s) --- PASS: TestDeriveS3AdvertisedEndpoint/empty_bind_IP_with_no_externalUrl_does_not_advertise_an_endpoint (0.00s) --- PASS: TestDeriveS3AdvertisedEndpoint/explicit_bind_IP_is_kept (0.00s) --- PASS: TestDeriveS3AdvertisedEndpoint/IPv6_literals_are_bracketed (0.00s) --- PASS: TestDeriveS3AdvertisedEndpoint/TLS_key_+_https_port_switches_scheme_and_port (0.00s) --- PASS: TestDeriveS3AdvertisedEndpoint/TLS-only_(no_separate_https_port)_still_uses_https_on_the_HTTP_port (0.00s) --- PASS: TestDeriveS3AdvertisedEndpoint/externalUrl_wins_and_trailing_slash_is_stripped (0.00s) === RUN TestReadingTomlConfiguration database is map[connection_max:5000 enabled:true ports:[8001 8001 8002] server:192.168.1.1] servers is map[alpha:map[dc:eqdc10 ip:10.0.0.1] beta:map[dc:eqdc10 ip:10.0.0.2]] alpha ip is 10.0.0.1 --- PASS: TestReadingTomlConfiguration (0.00s) === RUN TestXYZ I0824 22:19:11.101071 volume_test.go:12 Last-Modified Mon, 08 Jul 2013 08:53:16 GMT --- PASS: TestXYZ (0.00s) === RUN TestWorkerDefaultJobTypes --- PASS: TestWorkerDefaultJobTypes (0.00s) FAIL FAIL github.com/seaweedfs/seaweedfs/weed/command 62.566s ? github.com/seaweedfs/seaweedfs/weed/command/scaffold [no test files] === RUN TestCredentialStoreInterface --- PASS: TestCredentialStoreInterface (0.00s) === RUN TestCredentialManagerCreation --- PASS: TestCredentialManagerCreation (0.09s) === RUN TestCredentialInterface --- PASS: TestCredentialInterface (0.01s) === RUN TestCredentialManagerIntegration --- PASS: TestCredentialManagerIntegration (0.03s) === RUN TestErrorTypes --- PASS: TestErrorTypes (0.00s) === RUN TestGetAvailableStores credential_test.go:317: Available stores: [memory filer_etc] (count: 2) --- PASS: TestGetAvailableStores (0.00s) === RUN TestValidateServiceAccountId === RUN TestValidateServiceAccountId/simple === RUN TestValidateServiceAccountId/uppercase_parent === RUN TestValidateServiceAccountId/hyphenated_uuid_suffix === RUN TestValidateServiceAccountId/with_hyphen === RUN TestValidateServiceAccountId/with_underscore === RUN TestValidateServiceAccountId/email-style === RUN TestValidateServiceAccountId/with_dot === RUN TestValidateServiceAccountId/with_plus === RUN TestValidateServiceAccountId/with_equals === RUN TestValidateServiceAccountId/with_comma === RUN TestValidateServiceAccountId/empty === RUN TestValidateServiceAccountId/missing_prefix === RUN TestValidateServiceAccountId/wrong_prefix === RUN TestValidateServiceAccountId/uppercase_uuid === RUN TestValidateServiceAccountId/missing_uuid === RUN TestValidateServiceAccountId/missing_user === RUN TestValidateServiceAccountId/colon_in_user --- PASS: TestValidateServiceAccountId (0.01s) --- PASS: TestValidateServiceAccountId/simple (0.00s) --- PASS: TestValidateServiceAccountId/uppercase_parent (0.00s) --- PASS: TestValidateServiceAccountId/hyphenated_uuid_suffix (0.00s) --- PASS: TestValidateServiceAccountId/with_hyphen (0.00s) --- PASS: TestValidateServiceAccountId/with_underscore (0.00s) --- PASS: TestValidateServiceAccountId/email-style (0.00s) --- PASS: TestValidateServiceAccountId/with_dot (0.00s) --- PASS: TestValidateServiceAccountId/with_plus (0.00s) --- PASS: TestValidateServiceAccountId/with_equals (0.00s) --- PASS: TestValidateServiceAccountId/with_comma (0.00s) --- PASS: TestValidateServiceAccountId/empty (0.00s) --- PASS: TestValidateServiceAccountId/missing_prefix (0.00s) --- PASS: TestValidateServiceAccountId/wrong_prefix (0.00s) --- PASS: TestValidateServiceAccountId/uppercase_uuid (0.00s) --- PASS: TestValidateServiceAccountId/missing_uuid (0.00s) --- PASS: TestValidateServiceAccountId/missing_user (0.00s) --- PASS: TestValidateServiceAccountId/colon_in_user (0.00s) === RUN TestSaveConfigurationDelegatesWithoutMaster --- PASS: TestSaveConfigurationDelegatesWithoutMaster (0.01s) === RUN TestSaveConfigurationDiffSnapshotTakenBeforeWrite --- PASS: TestSaveConfigurationDiffSnapshotTakenBeforeWrite (0.00s) PASS ok github.com/seaweedfs/seaweedfs/weed/credential 18.056s === RUN TestFilerEtcStoreListPolicyNamesIncludesLegacyPolicies --- PASS: TestFilerEtcStoreListPolicyNamesIncludesLegacyPolicies (0.62s) === RUN TestFilerEtcStoreDeletePolicyRemovesLegacyManagedCopy --- PASS: TestFilerEtcStoreDeletePolicyRemovesLegacyManagedCopy (0.20s) === RUN TestFilerEtcStoreDeletePolicySerializesLegacyUpdates --- PASS: TestFilerEtcStoreDeletePolicySerializesLegacyUpdates (0.49s) === RUN TestFilerEtcStoreLoadManagedPoliciesRespectsReadContext --- PASS: TestFilerEtcStoreLoadManagedPoliciesRespectsReadContext (0.07s) PASS ok github.com/seaweedfs/seaweedfs/weed/credential/filer_etc 18.409s === RUN TestIamGrpcStore_AdminBearerToken === RUN TestIamGrpcStore_AdminBearerToken/matching_key_succeeds === RUN TestIamGrpcStore_AdminBearerToken/wrong_key_unauthenticated === RUN TestIamGrpcStore_AdminBearerToken/no_key_unauthenticated --- PASS: TestIamGrpcStore_AdminBearerToken (0.29s) --- PASS: TestIamGrpcStore_AdminBearerToken/matching_key_succeeds (0.24s) --- PASS: TestIamGrpcStore_AdminBearerToken/wrong_key_unauthenticated (0.01s) --- PASS: TestIamGrpcStore_AdminBearerToken/no_key_unauthenticated (0.01s) PASS ok github.com/seaweedfs/seaweedfs/weed/credential/grpc 31.244s === RUN TestMemoryStore --- PASS: TestMemoryStore (0.09s) === RUN TestMemoryStoreConcurrency --- PASS: TestMemoryStoreConcurrency (0.01s) === RUN TestMemoryStoreReset --- PASS: TestMemoryStoreReset (0.04s) === RUN TestMemoryStoreConfigurationSaveLoad --- PASS: TestMemoryStoreConfigurationSaveLoad (0.00s) === RUN TestMemoryStoreServiceAccountByAccessKey --- PASS: TestMemoryStoreServiceAccountByAccessKey (0.00s) PASS ok github.com/seaweedfs/seaweedfs/weed/credential/memory 15.380s ? github.com/seaweedfs/seaweedfs/weed/credential/postgres [no test files] === RUN TestInlinePolicyOperations --- PASS: TestInlinePolicyOperations (0.00s) === RUN TestMemoryRenameUserMovesIdentityAndPolicies --- PASS: TestMemoryRenameUserMovesIdentityAndPolicies (0.04s) === RUN TestStoreRegistration integration_test.go:40: Available stores: [memory postgres filer_etc] --- PASS: TestStoreRegistration (0.00s) === RUN TestMemoryStoreIntegration --- PASS: TestMemoryStoreIntegration (0.00s) === RUN TestPolicyManagement --- PASS: TestPolicyManagement (0.00s) === RUN TestPolicyManagementWithFilerEtc policy_test.go:140: Filer connection required for filer_etc store testing --- SKIP: TestPolicyManagementWithFilerEtc (0.00s) === RUN TestPolicyManagementWithPostgres policy_test.go:146: PostgreSQL connection required for postgres store testing --- SKIP: TestPolicyManagementWithPostgres (0.00s) PASS ok github.com/seaweedfs/seaweedfs/weed/credential/test 19.062s === RUN TestEcBalanceMigratesCrossDiskTypeShards dn1 moves ec shard 1.0 to dn2 dn1 moves ec shard 1.1 to dn3 dn1 moves ec shard 1.2 to dn2 dn1 moves ec shard 1.3 to dn3 dn1 moves ec shard 1.4 to dn2 dn1 moves ec shard 1.5 to dn3 dn1 moves ec shard 1.10 to dn2 dn1 moves ec shard 1.11 to dn3 --- PASS: TestEcBalanceMigratesCrossDiskTypeShards (0.05s) === RUN TestEcCheckReplication --- PASS: TestEcCheckReplication (0.00s) === RUN TestEcCheckReplicationNoVolumes === RUN TestEcCheckReplicationNoVolumes/unfiltered === RUN TestEcCheckReplicationNoVolumes/filtered_no_match --- PASS: TestEcCheckReplicationNoVolumes (0.00s) --- PASS: TestEcCheckReplicationNoVolumes/unfiltered (0.00s) --- PASS: TestEcCheckReplicationNoVolumes/filtered_no_match (0.00s) === RUN TestEcCheckReplicationUnexpectedShard --- PASS: TestEcCheckReplicationUnexpectedShard (0.04s) === RUN TestEcCheckReplicationMixedAnomalies --- PASS: TestEcCheckReplicationMixedAnomalies (0.00s) === RUN TestPickBestDiskOnNodeSelectsPhysicalDiskZero --- PASS: TestPickBestDiskOnNodeSelectsPhysicalDiskZero (0.00s) === RUN TestCommandEcBalanceSmall dn1 moves ec shard 1.0 to dn2 dn1 moves ec shard 1.1 to dn2 dn1 moves ec shard 1.2 to dn2 dn1 moves ec shard 1.3 to dn2 dn1 moves ec shard 1.4 to dn2 dn1 moves ec shard 1.10 to dn2 dn1 moves ec shard 1.11 to dn2 dn2 moves ec shard 2.0 to dn1 dn2 moves ec shard 2.1 to dn1 dn2 moves ec shard 2.2 to dn1 dn2 moves ec shard 2.3 to dn1 dn2 moves ec shard 2.4 to dn1 dn2 moves ec shard 2.10 to dn1 dn2 moves ec shard 2.11 to dn1 --- PASS: TestCommandEcBalanceSmall (0.01s) === RUN TestCommandEcBalanceNothingToMove dn1 moves ec shard 1.0 to dn2 dn1 moves ec shard 1.1 to dn2 dn2 moves ec shard 1.10 to dn1 dn2 moves ec shard 1.11 to dn1 dn2 moves ec shard 2.0 to dn1 dn2 moves ec shard 2.1 to dn1 dn1 moves ec shard 2.10 to dn2 dn1 moves ec shard 2.11 to dn2 --- PASS: TestCommandEcBalanceNothingToMove (0.00s) === RUN TestCommandEcBalanceAddNewServers dn1 moves ec shard 1.0 to dn3 dn1 moves ec shard 1.1 to dn4 dn1 moves ec shard 1.2 to dn3 dn1 moves ec shard 1.3 to dn4 dn2 moves ec shard 1.10 to dn1 dn2 moves ec shard 1.11 to dn3 dn2 moves ec shard 1.12 to dn4 dn2 moves ec shard 2.0 to dn3 dn2 moves ec shard 2.1 to dn4 dn2 moves ec shard 2.2 to dn3 dn2 moves ec shard 2.3 to dn4 dn1 moves ec shard 2.10 to dn2 dn1 moves ec shard 2.11 to dn3 dn1 moves ec shard 2.12 to dn4 dn1 moves ec shard 1.4 to dn3 dn2 moves ec shard 1.7 to dn4 --- PASS: TestCommandEcBalanceAddNewServers (0.01s) === RUN TestCommandEcBalanceAddNewRacks dn1 moves ec shard 1.0 to dn3 dn1 moves ec shard 1.1 to dn4 dn1 moves ec shard 1.2 to dn3 dn1 moves ec shard 1.3 to dn4 dn1 moves ec shard 1.4 to dn3 dn2 moves ec shard 1.10 to dn4 dn2 moves ec shard 1.11 to dn3 dn2 moves ec shard 2.0 to dn3 dn2 moves ec shard 2.1 to dn4 dn2 moves ec shard 2.2 to dn3 dn2 moves ec shard 2.3 to dn4 dn2 moves ec shard 2.4 to dn3 dn1 moves ec shard 2.10 to dn4 dn1 moves ec shard 2.11 to dn3 dn2 moves ec shard 1.12 to dn1 dn1 moves ec shard 2.12 to dn2 dn3 moves ec shard 1.0 to dn4 --- PASS: TestCommandEcBalanceAddNewRacks (0.00s) === RUN TestCommandEcBalanceVolumeEvenButRackUneven dn_shared moves ec shard 1.0 to dn3 --- PASS: TestCommandEcBalanceVolumeEvenButRackUneven (0.01s) === RUN TestCommandEcBalanceEvenDataAndParityDistribution dn1 moves ec shard 1.0 to dn2 dn1 moves ec shard 1.1 to dn3 dn1 moves ec shard 1.2 to dn4 dn1 moves ec shard 1.3 to dn5 dn1 moves ec shard 1.4 to dn6 dn1 moves ec shard 1.5 to dn2 dn1 moves ec shard 1.6 to dn3 dn1 moves ec shard 1.7 to dn4 dn1 moves ec shard 1.10 to dn2 dn1 moves ec shard 1.11 to dn3 dn1 moves ec shard 1.12 to dn4 ec_common_test.go:239: Distribution after balancing: ec_common_test.go:240: Data shards per rack: map[rack1:2 rack2:2 rack3:2 rack4:2 rack5:1 rack6:1] (max allowed: 2) ec_common_test.go:241: Parity shards per rack: map[rack1:1 rack2:1 rack3:1 rack4:1] (max allowed: 1) --- PASS: TestCommandEcBalanceEvenDataAndParityDistribution (0.01s) === RUN TestCommandEcBalanceMultipleVolumesEvenDistribution dn1 moves ec shard 1.0 to dn2 dn1 moves ec shard 1.1 to dn3 dn1 moves ec shard 1.2 to dn4 dn1 moves ec shard 1.3 to dn5 dn1 moves ec shard 1.4 to dn6 dn1 moves ec shard 1.5 to dn2 dn1 moves ec shard 1.6 to dn3 dn1 moves ec shard 1.7 to dn4 dn1 moves ec shard 1.10 to dn2 dn1 moves ec shard 1.11 to dn3 dn1 moves ec shard 1.12 to dn4 dn2 moves ec shard 2.0 to dn1 dn2 moves ec shard 2.1 to dn3 dn2 moves ec shard 2.2 to dn4 dn2 moves ec shard 2.3 to dn5 dn2 moves ec shard 2.4 to dn6 dn2 moves ec shard 2.5 to dn1 dn2 moves ec shard 2.6 to dn3 dn2 moves ec shard 2.7 to dn4 dn2 moves ec shard 2.10 to dn1 dn2 moves ec shard 2.11 to dn3 dn2 moves ec shard 2.12 to dn4 ec_common_test.go:303: Volume 1 - Data: map[rack1:2 rack2:2 rack3:2 rack4:2 rack5:1 rack6:1], Parity: map[rack1:1 rack2:1 rack3:1 rack4:1] ec_common_test.go:303: Volume 2 - Data: map[rack1:2 rack2:2 rack3:2 rack4:2 rack5:1 rack6:1], Parity: map[rack1:1 rack2:1 rack3:1 rack4:1] --- PASS: TestCommandEcBalanceMultipleVolumesEvenDistribution (0.01s) === RUN TestCommandEcBalanceAllNodesShareAllVolumes dn1 moves ec shard 1.0 to dn3 dn1 moves ec shard 1.1 to dn4 dn2 moves ec shard 1.5 to dn3 dn2 moves ec shard 1.6 to dn4 dn3 moves ec shard 1.10 to dn1 dn3 moves ec shard 1.11 to dn2 dn1 moves ec shard 2.0 to dn3 dn1 moves ec shard 2.1 to dn4 dn2 moves ec shard 2.5 to dn3 dn2 moves ec shard 2.6 to dn4 dn3 moves ec shard 2.10 to dn1 dn3 moves ec shard 2.11 to dn2 dn1 moves ec shard 1.2 to dn3 dn2 moves ec shard 1.7 to dn4 ec_common_test.go:351: node dn1: 7 shards ec_common_test.go:351: node dn2: 7 shards ec_common_test.go:351: node dn3: 7 shards ec_common_test.go:351: node dn4: 7 shards --- PASS: TestCommandEcBalanceAllNodesShareAllVolumes (0.01s) === RUN TestCommandEcBalanceIssue8793Topology ec_common_test.go:417: BEFORE node 192.168.0.12:8332 (max 80): 36 shards ec_common_test.go:417: BEFORE node 192.168.0.12:8333 (max 80): 36 shards ec_common_test.go:417: BEFORE node 192.168.0.12:8334 (max 80): 36 shards ec_common_test.go:417: BEFORE node 192.168.0.12:8335 (max 80): 36 shards ec_common_test.go:417: BEFORE node 192.168.0.12:8336 (max 80): 22 shards ec_common_test.go:417: BEFORE node 192.168.0.12:8337 (max 80): 22 shards ec_common_test.go:417: BEFORE node 192.168.0.12:8338 (max 80): 22 shards ec_common_test.go:417: BEFORE node 192.168.0.12:8339 (max 80): 22 shards ec_common_test.go:417: BEFORE node 192.168.0.12:8340 (max 80): 22 shards ec_common_test.go:417: BEFORE node 192.168.0.12:8341 (max 33): 22 shards ec_common_test.go:417: BEFORE node 192.168.0.12:8342 (max 33): 22 shards ec_common_test.go:417: BEFORE node 192.168.0.12:8343 (max 33): 22 shards ec_common_test.go:417: BEFORE node 192.168.0.25:8350 (max 33): 22 shards ec_common_test.go:417: BEFORE node 192.168.0.25:8351 (max 33): 22 shards 192.168.0.12:8332 moves ec shard 23.0 to 192.168.0.12:8333 192.168.0.12:8332 moves ec shard 23.1 to 192.168.0.12:8334 192.168.0.12:8332 moves ec shard 23.2 to 192.168.0.12:8335 192.168.0.12:8332 moves ec shard 23.3 to 192.168.0.12:8336 192.168.0.12:8332 moves ec shard 23.4 to 192.168.0.12:8337 192.168.0.12:8332 moves ec shard 23.5 to 192.168.0.12:8338 192.168.0.12:8332 moves ec shard 23.6 to 192.168.0.12:8339 192.168.0.12:8332 moves ec shard 23.7 to 192.168.0.12:8340 192.168.0.12:8332 moves ec shard 23.8 to 192.168.0.12:8341 192.168.0.12:8332 moves ec shard 23.10 to 192.168.0.12:8342 192.168.0.12:8332 moves ec shard 23.11 to 192.168.0.12:8343 192.168.0.12:8332 moves ec shard 23.12 to 192.168.0.25:8350 192.168.0.12:8333 moves ec shard 24.0 to 192.168.0.12:8332 192.168.0.12:8333 moves ec shard 24.1 to 192.168.0.12:8334 192.168.0.12:8333 moves ec shard 24.2 to 192.168.0.12:8335 192.168.0.12:8333 moves ec shard 24.3 to 192.168.0.12:8336 192.168.0.12:8333 moves ec shard 24.4 to 192.168.0.12:8337 192.168.0.12:8333 moves ec shard 24.5 to 192.168.0.12:8338 192.168.0.12:8333 moves ec shard 24.6 to 192.168.0.12:8339 192.168.0.12:8333 moves ec shard 24.7 to 192.168.0.12:8340 192.168.0.12:8333 moves ec shard 24.8 to 192.168.0.12:8341 192.168.0.12:8333 moves ec shard 24.10 to 192.168.0.12:8342 192.168.0.12:8333 moves ec shard 24.11 to 192.168.0.12:8343 192.168.0.12:8333 moves ec shard 24.12 to 192.168.0.25:8350 192.168.0.12:8334 moves ec shard 25.0 to 192.168.0.12:8332 192.168.0.12:8334 moves ec shard 25.1 to 192.168.0.12:8333 192.168.0.12:8334 moves ec shard 25.2 to 192.168.0.12:8335 192.168.0.12:8334 moves ec shard 25.3 to 192.168.0.12:8336 192.168.0.12:8334 moves ec shard 25.4 to 192.168.0.12:8337 192.168.0.12:8334 moves ec shard 25.5 to 192.168.0.12:8338 192.168.0.12:8334 moves ec shard 25.6 to 192.168.0.12:8339 192.168.0.12:8334 moves ec shard 25.7 to 192.168.0.12:8340 192.168.0.12:8334 moves ec shard 25.8 to 192.168.0.12:8341 192.168.0.12:8334 moves ec shard 25.10 to 192.168.0.12:8342 192.168.0.12:8334 moves ec shard 25.11 to 192.168.0.12:8343 192.168.0.12:8334 moves ec shard 25.12 to 192.168.0.25:8350 192.168.0.12:8335 moves ec shard 26.0 to 192.168.0.12:8332 192.168.0.12:8335 moves ec shard 26.1 to 192.168.0.12:8333 192.168.0.12:8335 moves ec shard 26.2 to 192.168.0.12:8334 192.168.0.12:8335 moves ec shard 26.3 to 192.168.0.12:8336 192.168.0.12:8335 moves ec shard 26.4 to 192.168.0.12:8337 192.168.0.12:8335 moves ec shard 26.5 to 192.168.0.12:8338 192.168.0.12:8335 moves ec shard 26.6 to 192.168.0.12:8339 192.168.0.12:8335 moves ec shard 26.7 to 192.168.0.12:8340 192.168.0.12:8335 moves ec shard 26.8 to 192.168.0.12:8341 192.168.0.12:8335 moves ec shard 26.10 to 192.168.0.12:8342 192.168.0.12:8335 moves ec shard 26.11 to 192.168.0.12:8343 192.168.0.12:8335 moves ec shard 26.12 to 192.168.0.25:8350 192.168.0.12:8341 moves ec shard 1.9 to 192.168.0.12:8336 192.168.0.12:8342 moves ec shard 1.10 to 192.168.0.12:8337 192.168.0.12:8343 moves ec shard 1.11 to 192.168.0.12:8338 192.168.0.25:8350 moves ec shard 1.12 to 192.168.0.12:8339 192.168.0.12:8341 moves ec shard 2.9 to 192.168.0.12:8340 192.168.0.12:8342 moves ec shard 2.10 to 192.168.0.12:8332 192.168.0.12:8343 moves ec shard 2.11 to 192.168.0.12:8333 192.168.0.25:8350 moves ec shard 2.12 to 192.168.0.12:8334 192.168.0.12:8341 moves ec shard 3.9 to 192.168.0.12:8335 192.168.0.12:8342 moves ec shard 3.10 to 192.168.0.12:8336 192.168.0.12:8343 moves ec shard 3.11 to 192.168.0.12:8337 192.168.0.25:8350 moves ec shard 3.12 to 192.168.0.12:8338 192.168.0.12:8341 moves ec shard 4.9 to 192.168.0.12:8339 192.168.0.12:8342 moves ec shard 4.10 to 192.168.0.12:8340 192.168.0.12:8343 moves ec shard 4.11 to 192.168.0.12:8332 192.168.0.25:8350 moves ec shard 4.12 to 192.168.0.12:8333 192.168.0.12:8341 moves ec shard 5.9 to 192.168.0.12:8334 192.168.0.12:8342 moves ec shard 5.10 to 192.168.0.12:8335 192.168.0.12:8343 moves ec shard 5.11 to 192.168.0.12:8336 192.168.0.25:8350 moves ec shard 5.12 to 192.168.0.12:8337 192.168.0.25:8351 moves ec shard 1.13 to 192.168.0.12:8338 192.168.0.12:8341 moves ec shard 6.9 to 192.168.0.12:8339 192.168.0.12:8342 moves ec shard 6.10 to 192.168.0.12:8340 192.168.0.12:8343 moves ec shard 6.11 to 192.168.0.12:8332 192.168.0.25:8350 moves ec shard 6.12 to 192.168.0.12:8333 192.168.0.25:8351 moves ec shard 2.13 to 192.168.0.12:8334 192.168.0.12:8341 moves ec shard 7.9 to 192.168.0.12:8335 192.168.0.12:8342 moves ec shard 7.10 to 192.168.0.12:8336 192.168.0.12:8343 moves ec shard 7.11 to 192.168.0.12:8337 192.168.0.25:8350 moves ec shard 7.12 to 192.168.0.12:8338 192.168.0.25:8351 moves ec shard 3.13 to 192.168.0.12:8339 192.168.0.12:8341 moves ec shard 8.9 to 192.168.0.12:8340 192.168.0.12:8342 moves ec shard 8.10 to 192.168.0.12:8332 192.168.0.12:8343 moves ec shard 8.11 to 192.168.0.12:8333 192.168.0.25:8350 moves ec shard 8.12 to 192.168.0.12:8334 192.168.0.25:8351 moves ec shard 4.13 to 192.168.0.12:8335 192.168.0.12:8341 moves ec shard 9.9 to 192.168.0.12:8336 192.168.0.12:8342 moves ec shard 9.10 to 192.168.0.12:8337 192.168.0.12:8343 moves ec shard 9.11 to 192.168.0.12:8338 192.168.0.25:8350 moves ec shard 9.12 to 192.168.0.12:8339 192.168.0.25:8351 moves ec shard 5.13 to 192.168.0.12:8340 192.168.0.12:8341 moves ec shard 10.9 to 192.168.0.12:8332 192.168.0.12:8342 moves ec shard 10.10 to 192.168.0.12:8333 192.168.0.12:8343 moves ec shard 10.11 to 192.168.0.12:8334 192.168.0.25:8350 moves ec shard 10.12 to 192.168.0.12:8335 192.168.0.25:8351 moves ec shard 6.13 to 192.168.0.12:8336 192.168.0.12:8341 moves ec shard 11.9 to 192.168.0.12:8337 192.168.0.12:8342 moves ec shard 11.10 to 192.168.0.12:8338 192.168.0.12:8343 moves ec shard 11.11 to 192.168.0.12:8339 192.168.0.25:8350 moves ec shard 11.12 to 192.168.0.12:8340 192.168.0.25:8351 moves ec shard 7.13 to 192.168.0.12:8332 192.168.0.12:8341 moves ec shard 12.9 to 192.168.0.12:8333 192.168.0.12:8342 moves ec shard 12.10 to 192.168.0.12:8334 192.168.0.12:8343 moves ec shard 12.11 to 192.168.0.12:8335 192.168.0.25:8350 moves ec shard 12.12 to 192.168.0.12:8336 192.168.0.25:8351 moves ec shard 8.13 to 192.168.0.12:8337 ec_common_test.go:454: AFTER node 192.168.0.12:8332: 33/80 shards (41% full, overall 41%) ec_common_test.go:454: AFTER node 192.168.0.12:8333: 33/80 shards (41% full, overall 41%) ec_common_test.go:454: AFTER node 192.168.0.12:8334: 33/80 shards (41% full, overall 41%) ec_common_test.go:454: AFTER node 192.168.0.12:8335: 33/80 shards (41% full, overall 41%) ec_common_test.go:454: AFTER node 192.168.0.12:8336: 33/80 shards (41% full, overall 41%) ec_common_test.go:454: AFTER node 192.168.0.12:8337: 33/80 shards (41% full, overall 41%) ec_common_test.go:454: AFTER node 192.168.0.12:8338: 32/80 shards (40% full, overall 41%) ec_common_test.go:454: AFTER node 192.168.0.12:8339: 32/80 shards (40% full, overall 41%) ec_common_test.go:454: AFTER node 192.168.0.12:8340: 32/80 shards (40% full, overall 41%) ec_common_test.go:454: AFTER node 192.168.0.12:8341: 14/33 shards (42% full, overall 41%) ec_common_test.go:454: AFTER node 192.168.0.12:8342: 14/33 shards (42% full, overall 41%) ec_common_test.go:454: AFTER node 192.168.0.12:8343: 14/33 shards (42% full, overall 41%) ec_common_test.go:454: AFTER node 192.168.0.25:8350: 14/33 shards (42% full, overall 41%) ec_common_test.go:454: AFTER node 192.168.0.25:8351: 14/33 shards (42% full, overall 41%) --- PASS: TestCommandEcBalanceIssue8793Topology (0.06s) === RUN TestCommandEcBalanceVolumeIdsFilter balancing ec volume(s) [1] planned 11 ec shard move(s) and 0 ec shard deletion(s) dn1 moves ec shard 1.0 to dn2 dn1 moves ec shard 1.1 to dn3 dn1 moves ec shard 1.2 to dn4 dn1 moves ec shard 1.3 to dn5 dn1 moves ec shard 1.4 to dn6 dn1 moves ec shard 1.5 to dn2 dn1 moves ec shard 1.6 to dn3 dn1 moves ec shard 1.7 to dn4 dn1 moves ec shard 1.10 to dn2 dn1 moves ec shard 1.11 to dn3 dn1 moves ec shard 1.12 to dn4 --- PASS: TestCommandEcBalanceVolumeIdsFilter (0.01s) === RUN TestCommandEcBalanceVolumeIdsNotFound --- PASS: TestCommandEcBalanceVolumeIdsNotFound (0.00s) === RUN TestMissingDataShard === RUN TestMissingDataShard/complete_set_on_one_holder_decodes,_so_nothing_is_finished === RUN TestMissingDataShard/complete_set_spread_across_holders_is_still_complete === RUN TestMissingDataShard/parity_gone_is_not_the_deletion_phase:_the_volume_still_decodes === RUN TestMissingDataShard/a_data_shard_gone_is_a_decode_that_was_interrupted_mid-cleanup === RUN TestMissingDataShard/the_first_gap_is_reported,_not_the_last === RUN TestMissingDataShard/no_holders_at_all_leaves_every_data_shard_missing --- PASS: TestMissingDataShard (0.00s) --- PASS: TestMissingDataShard/complete_set_on_one_holder_decodes,_so_nothing_is_finished (0.00s) --- PASS: TestMissingDataShard/complete_set_spread_across_holders_is_still_complete (0.00s) --- PASS: TestMissingDataShard/parity_gone_is_not_the_deletion_phase:_the_volume_still_decodes (0.00s) --- PASS: TestMissingDataShard/a_data_shard_gone_is_a_decode_that_was_interrupted_mid-cleanup (0.00s) --- PASS: TestMissingDataShard/the_first_gap_is_reported,_not_the_last (0.00s) --- PASS: TestMissingDataShard/no_holders_at_all_leaves_every_data_shard_missing (0.00s) === RUN TestAssertEncodableRegularVolumes === RUN TestAssertEncodableRegularVolumes/regular_volume === RUN TestAssertEncodableRegularVolumes/already_EC === RUN TestAssertEncodableRegularVolumes/regular_plus_stale_orphan_shards === RUN TestAssertEncodableRegularVolumes/unknown_id --- PASS: TestAssertEncodableRegularVolumes (0.03s) --- PASS: TestAssertEncodableRegularVolumes/regular_volume (0.03s) --- PASS: TestAssertEncodableRegularVolumes/already_EC (0.00s) --- PASS: TestAssertEncodableRegularVolumes/regular_plus_stale_orphan_shards (0.00s) --- PASS: TestAssertEncodableRegularVolumes/unknown_id (0.00s) === RUN TestPrepareDataToRecover_DryRunRecoverableNoSideEffects --- PASS: TestPrepareDataToRecover_DryRunRecoverableNoSideEffects (0.00s) === RUN TestPrepareDataToRecover_UnionsLocalShardsAcrossDisks --- PASS: TestPrepareDataToRecover_UnionsLocalShardsAcrossDisks (0.00s) === RUN TestCountLocalShards_UnionsAcrossDisks --- PASS: TestCountLocalShards_UnionsAcrossDisks (0.00s) === RUN TestPrepareDataToRecover_SelfSourceNotCopied --- PASS: TestPrepareDataToRecover_SelfSourceNotCopied (0.00s) === RUN TestPrepareDataToRecover_ApplyCopyFailureDoesNotCountAsRecoverable --- PASS: TestPrepareDataToRecover_ApplyCopyFailureDoesNotCountAsRecoverable (0.06s) === RUN TestRequireUniformShardSizes === RUN TestRequireUniformShardSizes/one_holder,_all_shards_the_same_length === RUN TestRequireUniformShardSizes/spread_across_holders,_still_one_length === RUN TestRequireUniformShardSizes/a_truncated_shard_is_named_with_its_holder === RUN TestRequireUniformShardSizes/two_copies_of_one_shard_that_disagree_are_still_a_disagreement === RUN TestRequireUniformShardSizes/unreported_sizes_are_skipped,_not_read_as_a_disagreement === RUN TestRequireUniformShardSizes/nothing_reported_at_all_verifies_vacuously === RUN TestRequireUniformShardSizes/no_holders_is_not_a_size_problem === RUN TestRequireUniformShardSizes/a_nil_inventory_is_skipped_rather_than_panicking --- PASS: TestRequireUniformShardSizes (0.00s) --- PASS: TestRequireUniformShardSizes/one_holder,_all_shards_the_same_length (0.00s) --- PASS: TestRequireUniformShardSizes/spread_across_holders,_still_one_length (0.00s) --- PASS: TestRequireUniformShardSizes/a_truncated_shard_is_named_with_its_holder (0.00s) --- PASS: TestRequireUniformShardSizes/two_copies_of_one_shard_that_disagree_are_still_a_disagreement (0.00s) --- PASS: TestRequireUniformShardSizes/unreported_sizes_are_skipped,_not_read_as_a_disagreement (0.00s) --- PASS: TestRequireUniformShardSizes/nothing_reported_at_all_verifies_vacuously (0.00s) --- PASS: TestRequireUniformShardSizes/no_holders_is_not_a_size_problem (0.00s) --- PASS: TestRequireUniformShardSizes/a_nil_inventory_is_skipped_rather_than_panicking (0.00s) === RUN TestShardSizeCheckIgnoresOlderEncodeGenerations --- PASS: TestShardSizeCheckIgnoresOlderEncodeGenerations (0.00s) === RUN TestShardSizeCheckStillCatchesShortShardInNewestGeneration --- PASS: TestShardSizeCheckStillCatchesShortShardInNewestGeneration (0.00s) === RUN TestShardSizeCheckHandlesUnstampedGenerations --- PASS: TestShardSizeCheckHandlesUnstampedGenerations (0.00s) === RUN TestSelectVolumeIdsFromTopology skip volume 1 on seaweedfs-volume-0.seaweedfs-volume.sea:8080: too small (size: 0.0 MB, threshold: 950.0 MB, 0.0% full) skip volume 2 on seaweedfs-volume-0.seaweedfs-volume.sea:8080: too small (size: 0.0 MB, threshold: 950.0 MB, 0.0% full) skip volume 3 on seaweedfs-volume-0.seaweedfs-volume.sea:8080: too small (size: 0.1 MB, threshold: 950.0 MB, 0.0% full) skip volume 4 on seaweedfs-volume-0.seaweedfs-volume.sea:8080: too small (size: 0.0 MB, threshold: 950.0 MB, 0.0% full) skip volume 5 on seaweedfs-volume-0.seaweedfs-volume.sea:8080: too small (size: 0.1 MB, threshold: 950.0 MB, 0.0% full) skip volume 6 on seaweedfs-volume-0.seaweedfs-volume.sea:8080: too small (size: 0.1 MB, threshold: 950.0 MB, 0.0% full) skip volume 7 on seaweedfs-volume-0.seaweedfs-volume.sea:8080: too small (size: 0.0 MB, threshold: 950.0 MB, 0.0% full) skip volume 8 on seaweedfs-volume-0.seaweedfs-volume.sea:8080: too small (size: 0.0 MB, threshold: 950.0 MB, 0.0% full) skip volume 9 on seaweedfs-volume-0.seaweedfs-volume.sea:8080: too small (size: 0.0 MB, threshold: 950.0 MB, 0.0% full) I0824 22:18:26.220080 ec_encode.go:974 replica test 10 on seaweedfs-volume-0.seaweedfs-volume.sea:8080 has no free disk skip replica of volume 10 on seaweedfs-volume-0.seaweedfs-volume.sea:8080: insufficient free disk space (free volumes: 1, required: 2) I0824 22:18:26.229027 ec_encode.go:974 replica test 11 on seaweedfs-volume-0.seaweedfs-volume.sea:8080 has no free disk skip replica of volume 11 on seaweedfs-volume-0.seaweedfs-volume.sea:8080: insufficient free disk space (free volumes: 1, required: 2) I0824 22:18:26.229181 ec_encode.go:974 replica test 12 on seaweedfs-volume-0.seaweedfs-volume.sea:8080 has no free disk skip replica of volume 12 on seaweedfs-volume-0.seaweedfs-volume.sea:8080: insufficient free disk space (free volumes: 1, required: 2) skip volume 13 on seaweedfs-volume-0.seaweedfs-volume.sea:8080: too small (size: 544.0 MB, threshold: 950.0 MB, 54.4% full) skip volume 14 on seaweedfs-volume-0.seaweedfs-volume.sea:8080: too small (size: 456.0 MB, threshold: 950.0 MB, 45.6% full) skip volume 1 on seaweedfs-volume-1.seaweedfs-volume.sea:8080: too small (size: 0.0 MB, threshold: 950.0 MB, 0.0% full) skip volume 2 on seaweedfs-volume-1.seaweedfs-volume.sea:8080: too small (size: 0.0 MB, threshold: 950.0 MB, 0.0% full) skip volume 3 on seaweedfs-volume-1.seaweedfs-volume.sea:8080: too small (size: 0.1 MB, threshold: 950.0 MB, 0.0% full) skip volume 4 on seaweedfs-volume-1.seaweedfs-volume.sea:8080: too small (size: 0.0 MB, threshold: 950.0 MB, 0.0% full) skip volume 5 on seaweedfs-volume-1.seaweedfs-volume.sea:8080: too small (size: 0.1 MB, threshold: 950.0 MB, 0.0% full) skip volume 6 on seaweedfs-volume-1.seaweedfs-volume.sea:8080: too small (size: 0.1 MB, threshold: 950.0 MB, 0.0% full) skip volume 7 on seaweedfs-volume-1.seaweedfs-volume.sea:8080: too small (size: 0.0 MB, threshold: 950.0 MB, 0.0% full) skip volume 8 on seaweedfs-volume-1.seaweedfs-volume.sea:8080: too small (size: 0.0 MB, threshold: 950.0 MB, 0.0% full) skip volume 9 on seaweedfs-volume-1.seaweedfs-volume.sea:8080: too small (size: 0.0 MB, threshold: 950.0 MB, 0.0% full) I0824 22:18:26.229492 ec_encode.go:974 replica test 10 on seaweedfs-volume-1.seaweedfs-volume.sea:8080 has no free disk skip replica of volume 10 on seaweedfs-volume-1.seaweedfs-volume.sea:8080: insufficient free disk space (free volumes: 1, required: 2) I0824 22:18:26.229539 ec_encode.go:974 replica test 11 on seaweedfs-volume-1.seaweedfs-volume.sea:8080 has no free disk skip replica of volume 11 on seaweedfs-volume-1.seaweedfs-volume.sea:8080: insufficient free disk space (free volumes: 1, required: 2) I0824 22:18:26.229571 ec_encode.go:974 replica test 12 on seaweedfs-volume-1.seaweedfs-volume.sea:8080 has no free disk skip replica of volume 12 on seaweedfs-volume-1.seaweedfs-volume.sea:8080: insufficient free disk space (free volumes: 1, required: 2) skip volume 13 on seaweedfs-volume-1.seaweedfs-volume.sea:8080: too small (size: 544.0 MB, threshold: 950.0 MB, 54.4% full) skip volume 14 on seaweedfs-volume-1.seaweedfs-volume.sea:8080: too small (size: 456.0 MB, threshold: 950.0 MB, 45.6% full) Volume selection summary: Total volumes examined: 28 Selected for encoding: 0 Collections matched: [ encrypt-data test] Reasons for exclusion: Too small (< 95.0% full): 22 Insufficient free disk space: 3 skip volume 1 on seaweedfs-volume-0.seaweedfs-volume.sea:8080: too small (size: 0.0 MB, threshold: 950.0 MB, 0.0% full) skip volume 2 on seaweedfs-volume-0.seaweedfs-volume.sea:8080: too small (size: 0.0 MB, threshold: 950.0 MB, 0.0% full) skip volume 3 on seaweedfs-volume-0.seaweedfs-volume.sea:8080: too small (size: 0.1 MB, threshold: 950.0 MB, 0.0% full) skip volume 4 on seaweedfs-volume-0.seaweedfs-volume.sea:8080: too small (size: 0.0 MB, threshold: 950.0 MB, 0.0% full) skip volume 5 on seaweedfs-volume-0.seaweedfs-volume.sea:8080: too small (size: 0.1 MB, threshold: 950.0 MB, 0.0% full) skip volume 6 on seaweedfs-volume-0.seaweedfs-volume.sea:8080: too small (size: 0.1 MB, threshold: 950.0 MB, 0.0% full) skip volume 7 on seaweedfs-volume-0.seaweedfs-volume.sea:8080: too small (size: 0.0 MB, threshold: 950.0 MB, 0.0% full) skip volume 8 on seaweedfs-volume-0.seaweedfs-volume.sea:8080: too small (size: 0.0 MB, threshold: 950.0 MB, 0.0% full) skip volume 9 on seaweedfs-volume-0.seaweedfs-volume.sea:8080: too small (size: 0.0 MB, threshold: 950.0 MB, 0.0% full) selected volume 10 on seaweedfs-volume-0.seaweedfs-volume.sea:8080: size 1000.0 MB (100.0% full), last modified 5025 seconds ago, free volumes: 2 selected volume 11 on seaweedfs-volume-0.seaweedfs-volume.sea:8080: size 1000.0 MB (100.0% full), last modified 4966 seconds ago, free volumes: 2 selected volume 12 on seaweedfs-volume-0.seaweedfs-volume.sea:8080: size 1000.0 MB (100.0% full), last modified 4933 seconds ago, free volumes: 2 skip volume 13 on seaweedfs-volume-0.seaweedfs-volume.sea:8080: too small (size: 544.0 MB, threshold: 950.0 MB, 54.4% full) skip volume 14 on seaweedfs-volume-0.seaweedfs-volume.sea:8080: too small (size: 456.0 MB, threshold: 950.0 MB, 45.6% full) skip volume 1 on seaweedfs-volume-1.seaweedfs-volume.sea:8080: too small (size: 0.0 MB, threshold: 950.0 MB, 0.0% full) skip volume 2 on seaweedfs-volume-1.seaweedfs-volume.sea:8080: too small (size: 0.0 MB, threshold: 950.0 MB, 0.0% full) skip volume 3 on seaweedfs-volume-1.seaweedfs-volume.sea:8080: too small (size: 0.1 MB, threshold: 950.0 MB, 0.0% full) skip volume 4 on seaweedfs-volume-1.seaweedfs-volume.sea:8080: too small (size: 0.0 MB, threshold: 950.0 MB, 0.0% full) skip volume 5 on seaweedfs-volume-1.seaweedfs-volume.sea:8080: too small (size: 0.1 MB, threshold: 950.0 MB, 0.0% full) skip volume 6 on seaweedfs-volume-1.seaweedfs-volume.sea:8080: too small (size: 0.1 MB, threshold: 950.0 MB, 0.0% full) skip volume 7 on seaweedfs-volume-1.seaweedfs-volume.sea:8080: too small (size: 0.0 MB, threshold: 950.0 MB, 0.0% full) skip volume 8 on seaweedfs-volume-1.seaweedfs-volume.sea:8080: too small (size: 0.0 MB, threshold: 950.0 MB, 0.0% full) skip volume 9 on seaweedfs-volume-1.seaweedfs-volume.sea:8080: too small (size: 0.0 MB, threshold: 950.0 MB, 0.0% full) selected volume 10 on seaweedfs-volume-1.seaweedfs-volume.sea:8080: size 1000.0 MB (100.0% full), last modified 5025 seconds ago, free volumes: 2 selected volume 11 on seaweedfs-volume-1.seaweedfs-volume.sea:8080: size 1000.0 MB (100.0% full), last modified 4966 seconds ago, free volumes: 2 selected volume 12 on seaweedfs-volume-1.seaweedfs-volume.sea:8080: size 1000.0 MB (100.0% full), last modified 4934 seconds ago, free volumes: 2 skip volume 13 on seaweedfs-volume-1.seaweedfs-volume.sea:8080: too small (size: 544.0 MB, threshold: 950.0 MB, 54.4% full) skip volume 14 on seaweedfs-volume-1.seaweedfs-volume.sea:8080: too small (size: 456.0 MB, threshold: 950.0 MB, 45.6% full) Volume selection summary: Total volumes examined: 28 Selected for encoding: 3 Collections matched: [ encrypt-data test] Reasons for exclusion: Too small (< 95.0% full): 22 --- PASS: TestSelectVolumeIdsFromTopology (0.01s) === RUN TestEcEncodeNodeCountCheck --- PASS: TestEcEncodeNodeCountCheck (0.00s) === RUN TestChunkVolumeIds --- PASS: TestChunkVolumeIds (0.00s) === RUN TestCollectEcShardBitsByNode --- PASS: TestCollectEcShardBitsByNode (0.00s) === RUN TestCollectEcShardBitsByNode_MixedGenerations --- PASS: TestCollectEcShardBitsByNode_MixedGenerations (0.00s) === RUN TestEcShardsClumpedOnOneNode --- PASS: TestEcShardsClumpedOnOneNode (0.00s) === RUN TestEcShardCountIgnoresDiskTypeOfTheShards --- PASS: TestEcShardCountIgnoresDiskTypeOfTheShards (0.00s) === RUN TestEcShardSummaryNamesTheShardIds --- PASS: TestEcShardSummaryNamesTheShardIds (0.00s) === RUN TestCountFreeShardSlotsPhysicalDiskGate --- PASS: TestCountFreeShardSlotsPhysicalDiskGate (0.00s) === RUN TestParseVolumeIdsFlag --- PASS: TestParseVolumeIdsFlag (0.00s) === RUN TestECLifecycleModelExhaustive ec_lifecycle_model_test.go:519: explored 6923 schedules reaching 17 distinct quiescent states, all recoverable --- PASS: TestECLifecycleModelExhaustive (0.28s) === RUN TestECRebalanceWithLimitedSlots ec_rebalance_slots_test.go:38: Topology summary: ec_rebalance_slots_test.go:39: Number of EC nodes: 6 ec_rebalance_slots_test.go:40: Total free EC slots: 432 ec_rebalance_slots_test.go:50: Node 127.0.0.1:8080 (rack rack0): 28 shards, 52 free slots ec_rebalance_slots_test.go:50: Node 127.0.0.1:8081 (rack rack1): 28 shards, 52 free slots ec_rebalance_slots_test.go:50: Node 127.0.0.1:8082 (rack rack2): 14 shards, 76 free slots ec_rebalance_slots_test.go:50: Node 127.0.0.1:8083 (rack rack3): 28 shards, 52 free slots ec_rebalance_slots_test.go:50: Node 127.0.0.1:8084 (rack rack4): 0 shards, 100 free slots ec_rebalance_slots_test.go:50: Node 127.0.0.1:8085 (rack rack5): 0 shards, 100 free slots ec_rebalance_slots_test.go:63: Total EC shards: 98 ec_rebalance_slots_test.go:105: totalFreeEcSlots = 432, rebalance should be possible --- PASS: TestECRebalanceWithLimitedSlots (0.01s) === RUN TestECRebalanceZeroFreeSlots ec_rebalance_slots_test.go:120: Zero free slots scenario: ec_rebalance_slots_test.go:128: Node 127.0.0.1:8080: 28 shards, 0 free slots, volumeCount=10, max=10 ec_rebalance_slots_test.go:128: Node 127.0.0.1:8081: 28 shards, 0 free slots, volumeCount=10, max=10 ec_rebalance_slots_test.go:128: Node 127.0.0.1:8082: 14 shards, 0 free slots, volumeCount=10, max=10 ec_rebalance_slots_test.go:128: Node 127.0.0.1:8083: 28 shards, 0 free slots, volumeCount=10, max=10 ec_rebalance_slots_test.go:133: Total free slots: 0 ec_rebalance_slots_test.go:136: ec_rebalance_slots_test.go:137: SCENARIO REPRODUCED: totalFreeEcSlots = 0 ec_rebalance_slots_test.go:138: This would trigger: 'no free ec shard slots. only 0 left' --- PASS: TestECRebalanceZeroFreeSlots (0.00s) === RUN TestEcShardMapRegister --- PASS: TestEcShardMapRegister (0.00s) === RUN TestEcShardMapShardCount === RUN TestEcShardMapShardCount/all_shards === RUN TestEcShardMapShardCount/data_shards_only === RUN TestEcShardMapShardCount/parity_shards_only === RUN TestEcShardMapShardCount/missing_some_shards === RUN TestEcShardMapShardCount/single_shard === RUN TestEcShardMapShardCount/no_shards --- PASS: TestEcShardMapShardCount (0.01s) --- PASS: TestEcShardMapShardCount/all_shards (0.00s) --- PASS: TestEcShardMapShardCount/data_shards_only (0.00s) --- PASS: TestEcShardMapShardCount/parity_shards_only (0.00s) --- PASS: TestEcShardMapShardCount/missing_some_shards (0.00s) --- PASS: TestEcShardMapShardCount/single_shard (0.00s) --- PASS: TestEcShardMapShardCount/no_shards (0.00s) === RUN TestRebuildEcVolumesInsufficientShards rebuildEcVolumes for "c1" --- PASS: TestRebuildEcVolumesInsufficientShards (0.00s) === RUN TestRebuildEcVolumesCompleteVolume rebuildEcVolumes for "c1" --- PASS: TestRebuildEcVolumesCompleteVolume (0.00s) === RUN TestRebuildEcVolumesInsufficientSpace rebuildEcVolumes for "c1" --- PASS: TestRebuildEcVolumesInsufficientSpace (0.00s) === RUN TestMultipleNodesWithShards --- PASS: TestMultipleNodesWithShards (0.00s) === RUN TestDuplicateShards --- PASS: TestDuplicateShards (0.00s) === RUN TestPrepareDataToRecoverTargetShardCount --- PASS: TestPrepareDataToRecoverTargetShardCount (0.00s) === RUN TestRebuilderVolumeIdMatches === RUN TestRebuilderVolumeIdMatches/no_target_volume_IDs_(nil) === RUN TestRebuilderVolumeIdMatches/no_target_volume_IDs === RUN TestRebuilderVolumeIdMatches/volume_ID_matches === RUN TestRebuilderVolumeIdMatches/volume_ID_mismatch --- PASS: TestRebuilderVolumeIdMatches (0.00s) --- PASS: TestRebuilderVolumeIdMatches/no_target_volume_IDs_(nil) (0.00s) --- PASS: TestRebuilderVolumeIdMatches/no_target_volume_IDs (0.00s) --- PASS: TestRebuilderVolumeIdMatches/volume_ID_matches (0.00s) --- PASS: TestRebuilderVolumeIdMatches/volume_ID_mismatch (0.00s) === RUN TestEcShardsFromString === RUN TestEcShardsFromString/single_id === RUN TestEcShardsFromString/list_of_ids === RUN TestEcShardsFromString/qualified_keeps_host:port === RUN TestEcShardsFromString/mixed_bare_and_qualified === RUN TestEcShardsFromString/colon_without_node_marker_is_not_a_shard_ID === RUN TestEcShardsFromString/shard_id_within_MaxShardCount === RUN TestEcShardsFromString/shard_id_at_MaxShardCount_is_rejected === RUN TestEcShardsFromString/negative_shard_id === RUN TestEcShardsFromString/out-of-range_shard_id_does_not_wrap === RUN TestEcShardsFromString/non-numeric_shard_id === RUN TestEcShardsFromString/empty_element --- PASS: TestEcShardsFromString (0.00s) --- PASS: TestEcShardsFromString/single_id (0.00s) --- PASS: TestEcShardsFromString/list_of_ids (0.00s) --- PASS: TestEcShardsFromString/qualified_keeps_host:port (0.00s) --- PASS: TestEcShardsFromString/mixed_bare_and_qualified (0.00s) --- PASS: TestEcShardsFromString/colon_without_node_marker_is_not_a_shard_ID (0.00s) --- PASS: TestEcShardsFromString/shard_id_within_MaxShardCount (0.00s) --- PASS: TestEcShardsFromString/shard_id_at_MaxShardCount_is_rejected (0.00s) --- PASS: TestEcShardsFromString/negative_shard_id (0.00s) --- PASS: TestEcShardsFromString/out-of-range_shard_id_does_not_wrap (0.00s) --- PASS: TestEcShardsFromString/non-numeric_shard_id (0.00s) --- PASS: TestEcShardsFromString/empty_element (0.00s) === RUN TestEcShardStringRoundTrips --- PASS: TestEcShardStringRoundTrips (0.00s) PASS ok github.com/seaweedfs/seaweedfs/weed/ec 18.020s === RUN TestEntryCodec_AtimeRoundTrip --- PASS: TestEntryCodec_AtimeRoundTrip (0.07s) === RUN TestEntryCodec_AtimeZeroFallsBackToMtime --- PASS: TestEntryCodec_AtimeZeroFallsBackToMtime (0.00s) === RUN TestEntryCodec_AtimeSubSecondEpochPreserved --- PASS: TestEntryCodec_AtimeSubSecondEpochPreserved (0.00s) === RUN TestDecodeAttributesOnlyMatchesFullDecode === RUN TestDecodeAttributesOnlyMatchesFullDecode/no_chunks === RUN TestDecodeAttributesOnlyMatchesFullDecode/directory === RUN TestDecodeAttributesOnlyMatchesFullDecode/one_chunk === RUN TestDecodeAttributesOnlyMatchesFullDecode/zero_FileSize,_size_comes_from_chunks === RUN TestDecodeAttributesOnlyMatchesFullDecode/chunks_out_of_order === RUN TestDecodeAttributesOnlyMatchesFullDecode/stored_FileSize_larger_than_chunks === RUN TestDecodeAttributesOnlyMatchesFullDecode/symlink === RUN TestDecodeAttributesOnlyMatchesFullDecode/hard_link === RUN TestDecodeAttributesOnlyMatchesFullDecode/inline_content === RUN TestDecodeAttributesOnlyMatchesFullDecode/extended_attributes === RUN TestDecodeAttributesOnlyMatchesFullDecode/remote_entry === RUN TestDecodeAttributesOnlyMatchesFullDecode/quota_and_worm --- PASS: TestDecodeAttributesOnlyMatchesFullDecode (0.23s) --- PASS: TestDecodeAttributesOnlyMatchesFullDecode/no_chunks (0.20s) --- PASS: TestDecodeAttributesOnlyMatchesFullDecode/directory (0.00s) --- PASS: TestDecodeAttributesOnlyMatchesFullDecode/one_chunk (0.01s) --- PASS: TestDecodeAttributesOnlyMatchesFullDecode/zero_FileSize,_size_comes_from_chunks (0.00s) --- PASS: TestDecodeAttributesOnlyMatchesFullDecode/chunks_out_of_order (0.00s) --- PASS: TestDecodeAttributesOnlyMatchesFullDecode/stored_FileSize_larger_than_chunks (0.00s) --- PASS: TestDecodeAttributesOnlyMatchesFullDecode/symlink (0.00s) --- PASS: TestDecodeAttributesOnlyMatchesFullDecode/hard_link (0.00s) --- PASS: TestDecodeAttributesOnlyMatchesFullDecode/inline_content (0.00s) --- PASS: TestDecodeAttributesOnlyMatchesFullDecode/extended_attributes (0.00s) --- PASS: TestDecodeAttributesOnlyMatchesFullDecode/remote_entry (0.00s) --- PASS: TestDecodeAttributesOnlyMatchesFullDecode/quota_and_worm (0.00s) === RUN TestDecodeAttributesOnlyRandomEntries --- PASS: TestDecodeAttributesOnlyRandomEntries (1.01s) === RUN TestDecodeAttributesOnlyRejectsGarbage --- PASS: TestDecodeAttributesOnlyRejectsGarbage (0.00s) === RUN TestProtoEntryWithoutChunksKeepsSize --- PASS: TestProtoEntryWithoutChunksKeepsSize (0.00s) === RUN TestChunkValidationCoversEveryField --- PASS: TestChunkValidationCoversEveryField (0.00s) === RUN TestDecodeAttributesOnlyRejectsWhatFullDecodeRejects === RUN TestDecodeAttributesOnlyRejectsWhatFullDecodeRejects/invalid_utf8_in_file_id === RUN TestDecodeAttributesOnlyRejectsWhatFullDecodeRejects/truncated_chunk === RUN TestDecodeAttributesOnlyRejectsWhatFullDecodeRejects/corrupt_nested_fid --- PASS: TestDecodeAttributesOnlyRejectsWhatFullDecodeRejects (0.03s) --- PASS: TestDecodeAttributesOnlyRejectsWhatFullDecodeRejects/invalid_utf8_in_file_id (0.00s) --- PASS: TestDecodeAttributesOnlyRejectsWhatFullDecodeRejects/truncated_chunk (0.00s) --- PASS: TestDecodeAttributesOnlyRejectsWhatFullDecodeRejects/corrupt_nested_fid (0.00s) === RUN TestDecodeAttributesOnlyManifestChunk --- PASS: TestDecodeAttributesOnlyManifestChunk (0.00s) === RUN TestDecodeAttributesOnlyFieldBeforeChunks --- PASS: TestDecodeAttributesOnlyFieldBeforeChunks (0.00s) === RUN TestApplyS3ExpiryMetadata === RUN TestApplyS3ExpiryMetadata/s3_entry_with_ttl === RUN TestApplyS3ExpiryMetadata/no_ttl === RUN TestApplyS3ExpiryMetadata/not_an_s3_entry === RUN TestApplyS3ExpiryMetadata/versioned_entry_keeps_crtime_expiry --- PASS: TestApplyS3ExpiryMetadata (0.00s) --- PASS: TestApplyS3ExpiryMetadata/s3_entry_with_ttl (0.00s) --- PASS: TestApplyS3ExpiryMetadata/no_ttl (0.00s) --- PASS: TestApplyS3ExpiryMetadata/not_an_s3_entry (0.00s) --- PASS: TestApplyS3ExpiryMetadata/versioned_entry_keeps_crtime_expiry (0.00s) === RUN TestChunkGroup_ReadDataAt_ErrorHandling === RUN TestChunkGroup_ReadDataAt_ErrorHandling/should_return_immediately_on_error === RUN TestChunkGroup_ReadDataAt_ErrorHandling/should_handle_EOF_correctly === RUN TestChunkGroup_ReadDataAt_ErrorHandling/should_return_EOF_when_offset_exceeds_file_size === RUN TestChunkGroup_ReadDataAt_ErrorHandling/should_demonstrate_the_GitHub_issue_fix_-_errors_should_not_be_masked === RUN TestChunkGroup_ReadDataAt_ErrorHandling/Context_Cancellation filechunk_group_test.go:145: Operation completed before context cancellation was checked - this is expected for empty ChunkGroup === RUN TestChunkGroup_ReadDataAt_ErrorHandling/Context_Cancellation_with_Timeout --- PASS: TestChunkGroup_ReadDataAt_ErrorHandling (0.03s) --- PASS: TestChunkGroup_ReadDataAt_ErrorHandling/should_return_immediately_on_error (0.00s) --- PASS: TestChunkGroup_ReadDataAt_ErrorHandling/should_handle_EOF_correctly (0.00s) --- PASS: TestChunkGroup_ReadDataAt_ErrorHandling/should_return_EOF_when_offset_exceeds_file_size (0.00s) --- PASS: TestChunkGroup_ReadDataAt_ErrorHandling/should_demonstrate_the_GitHub_issue_fix_-_errors_should_not_be_masked (0.00s) --- PASS: TestChunkGroup_ReadDataAt_ErrorHandling/Context_Cancellation (0.00s) --- PASS: TestChunkGroup_ReadDataAt_ErrorHandling/Context_Cancellation_with_Timeout (0.03s) === RUN TestChunkGroup_SearchChunks_Cancellation === RUN TestChunkGroup_SearchChunks_Cancellation/Context_Cancellation_in_SearchChunks filechunk_group_test.go:207: SearchChunks completed with cancelled context - context threading verified === RUN TestChunkGroup_SearchChunks_Cancellation/Context_with_Timeout_in_SearchChunks --- PASS: TestChunkGroup_SearchChunks_Cancellation (0.00s) --- PASS: TestChunkGroup_SearchChunks_Cancellation/Context_Cancellation_in_SearchChunks (0.00s) --- PASS: TestChunkGroup_SearchChunks_Cancellation/Context_with_Timeout_in_SearchChunks (0.00s) === RUN TestChunkGroup_doSearchChunks --- PASS: TestChunkGroup_doSearchChunks (0.00s) === RUN TestDoMaybeManifestize test 0 test 1 test 2 test 3 --- PASS: TestDoMaybeManifestize (0.00s) === RUN TestManifestRoundTripPreservesChunks --- PASS: TestManifestRoundTripPreservesChunks (0.03s) === RUN TestCompactResolvedOverlappingManifests --- PASS: TestCompactResolvedOverlappingManifests (0.00s) === RUN TestDoMinusChunksWithResolvedManifests --- PASS: TestDoMinusChunksWithResolvedManifests (0.03s) === RUN TestManifestizeSmallBatchWithRemainder --- PASS: TestManifestizeSmallBatchWithRemainder (0.00s) === RUN TestCompactMultipleOverlappingManifestGenerations --- PASS: TestCompactMultipleOverlappingManifestGenerations (0.00s) === RUN TestManifestBloatDetection filechunk_manifest_test.go:493: n=1 manifests: totalStored=20000 fileSize=10000 ratio=2.0x expectMerge=false filechunk_manifest_test.go:493: n=2 manifests: totalStored=30000 fileSize=10000 ratio=3.0x expectMerge=true filechunk_manifest_test.go:493: n=3 manifests: totalStored=40000 fileSize=10000 ratio=4.0x expectMerge=true filechunk_manifest_test.go:493: n=4 manifests: totalStored=50000 fileSize=10000 ratio=5.0x expectMerge=true filechunk_manifest_test.go:493: n=5 manifests: totalStored=60000 fileSize=10000 ratio=6.0x expectMerge=true filechunk_manifest_test.go:493: n=6 manifests: totalStored=70000 fileSize=10000 ratio=7.0x expectMerge=true filechunk_manifest_test.go:493: n=7 manifests: totalStored=80000 fileSize=10000 ratio=8.0x expectMerge=true filechunk_manifest_test.go:493: n=8 manifests: totalStored=90000 fileSize=10000 ratio=9.0x expectMerge=true filechunk_manifest_test.go:493: n=9 manifests: totalStored=100000 fileSize=10000 ratio=10.0x expectMerge=true filechunk_manifest_test.go:493: n=10 manifests: totalStored=110000 fileSize=10000 ratio=11.0x expectMerge=true --- PASS: TestManifestBloatDetection (0.00s) === RUN Test_removeGarbageChunks --- PASS: Test_removeGarbageChunks (0.00s) === RUN TestDoMinusChunks 2026/08/24 22:18:26 first deleted chunks: [file_id:"1" size:3 modified_ts_ns:100 source_file_id:"11" file_id:"2" offset:3 size:3 modified_ts_ns:200 file_id:"3" offset:6 size:3 modified_ts_ns:300 source_file_id:"33"] 2026/08/24 22:18:26 clusterA synced empty chunks event result: [] --- PASS: TestDoMinusChunks (0.01s) === RUN TestCompactFileChunksRealCase I0824 22:18:26.910658 filechunks2_test.go:84 before chunk 2,512f31f2c0700a [ 0, 25) I0824 22:18:26.921737 filechunks2_test.go:84 before chunk 6,512f2c2e24e9e8 [ 868352, 917585) I0824 22:18:26.922129 filechunks2_test.go:84 before chunk 7,514468dd5954ca [ 884736, 901120) I0824 22:18:26.922287 filechunks2_test.go:84 before chunk 5,5144463173fe77 [ 917504, 2297856) I0824 22:18:26.922419 filechunks2_test.go:84 before chunk 4,51444c7ab54e2d [ 2301952, 2367488) I0824 22:18:26.922730 filechunks2_test.go:84 before chunk 4,514450e643ad22 [ 2371584, 2420736) I0824 22:18:26.922804 filechunks2_test.go:84 before chunk 6,514456a5e9e4d7 [ 2449408, 2490368) I0824 22:18:26.922850 filechunks2_test.go:84 before chunk 3,51444f8d53eebe [ 2494464, 2555904) I0824 22:18:26.922882 filechunks2_test.go:84 before chunk 4,5144578b097c7e [ 2560000, 2596864) I0824 22:18:26.922911 filechunks2_test.go:84 before chunk 3,51445500b6b4ac [ 2637824, 2678784) I0824 22:18:26.922934 filechunks2_test.go:84 before chunk 1,51446285e52a61 [ 2695168, 2715648) I0824 22:18:26.923304 filechunks2_test.go:84 compacted chunk 2,512f31f2c0700a [ 0, 25) I0824 22:18:26.923454 filechunks2_test.go:84 compacted chunk 6,512f2c2e24e9e8 [ 868352, 917585) I0824 22:18:26.923506 filechunks2_test.go:84 compacted chunk 7,514468dd5954ca [ 884736, 901120) I0824 22:18:26.923568 filechunks2_test.go:84 compacted chunk 5,5144463173fe77 [ 917504, 2297856) I0824 22:18:26.923594 filechunks2_test.go:84 compacted chunk 4,51444c7ab54e2d [ 2301952, 2367488) I0824 22:18:26.923640 filechunks2_test.go:84 compacted chunk 4,514450e643ad22 [ 2371584, 2420736) I0824 22:18:26.923665 filechunks2_test.go:84 compacted chunk 6,514456a5e9e4d7 [ 2449408, 2490368) I0824 22:18:26.923691 filechunks2_test.go:84 compacted chunk 3,51444f8d53eebe [ 2494464, 2555904) I0824 22:18:26.923720 filechunks2_test.go:84 compacted chunk 4,5144578b097c7e [ 2560000, 2596864) I0824 22:18:26.923748 filechunks2_test.go:84 compacted chunk 3,51445500b6b4ac [ 2637824, 2678784) I0824 22:18:26.923769 filechunks2_test.go:84 compacted chunk 1,51446285e52a61 [ 2695168, 2715648) --- PASS: TestCompactFileChunksRealCase (0.04s) === RUN TestReadResolvedChunks resolved to 4 visible intervales [0,50) a 1 [50,150) b 2 [175,275) e 5 [275,300) d 4 --- PASS: TestReadResolvedChunks (0.00s) === RUN TestReadResolvedChunks2 resolved to 2 visible intervales [200,225) e 5 [225,250) c 3 --- PASS: TestReadResolvedChunks2 (0.00s) === RUN TestRandomizedReadResolvedChunks --- PASS: TestRandomizedReadResolvedChunks (0.09s) === RUN TestSequentialReadResolvedChunks visibles 13--- PASS: TestSequentialReadResolvedChunks (0.00s) === RUN TestActualReadResolvedChunks [0,2097152) 5,e7b96fef48 1634447487595823000 [2097152,4194304) 5,e5562640b9 1634447487595826000 [4194304,6291456) 5,df033e0fe4 1634447487595827000 [6291456,8388608) 7,eb08148a9b 1634447487595827000 [8388608,10485760) 7,e0f92d1604 1634447487595828000 [10485760,12582912) 7,e33cb63262 1634447487595828000 [12582912,14680064) 5,ea98e40e93 1634447487595829000 [14680064,16777216) 5,e165661172 1634447487595829000 [16777216,18874368) 3,e692097486 1634447487595830000 [18874368,20971520) 3,e28e2e3cbd 1634447487595830000 [20971520,23068672) 3,e443974d4e 1634447487595830000 [23068672,25165824) 2,e815bed597 1634447487595831000 [25165824,27140560) 5,e94715199e 1634447487595832000 --- PASS: TestActualReadResolvedChunks (0.00s) === RUN TestActualReadResolvedChunks2 [0,184320) 1,e7b96fef48 1 [184320,188416) 2,33562640b9 4 [188416,2285568) 4,df033e0fe4 3 --- PASS: TestActualReadResolvedChunks2 (0.00s) === RUN TestCompactFileChunks --- PASS: TestCompactFileChunks (0.00s) === RUN TestCompactFileChunks2 --- PASS: TestCompactFileChunks2 (0.00s) === RUN TestRandomFileChunksCompact --- PASS: TestRandomFileChunksCompact (0.00s) === RUN TestIntervalMerging 2026/08/24 22:18:27 ++++++++++ merged test case 0 ++++++++++++++++++++ 2026/08/24 22:18:27 test case 0, interval start=0, stop=100, fileId=abc 2026/08/24 22:18:27 test case 0, interval start=100, stop=200, fileId=asdf 2026/08/24 22:18:27 test case 0, interval start=200, stop=300, fileId=fsad 2026/08/24 22:18:27 ++++++++++ merged test case 1 ++++++++++++++++++++ 2026/08/24 22:18:27 test case 1, interval start=0, stop=200, fileId=asdf 2026/08/24 22:18:27 ++++++++++ merged test case 2 ++++++++++++++++++++ 2026/08/24 22:18:27 test case 2, interval start=0, stop=70, fileId=b 2026/08/24 22:18:27 test case 2, interval start=70, stop=100, fileId=a 2026/08/24 22:18:27 ++++++++++ merged test case 3 ++++++++++++++++++++ 2026/08/24 22:18:27 test case 3, interval start=0, stop=50, fileId=asdf 2026/08/24 22:18:27 test case 3, interval start=50, stop=300, fileId=xxxx 2026/08/24 22:18:27 ++++++++++ merged test case 4 ++++++++++++++++++++ 2026/08/24 22:18:27 test case 4, interval start=0, stop=200, fileId=asdf 2026/08/24 22:18:27 test case 4, interval start=250, stop=500, fileId=xxxx 2026/08/24 22:18:27 ++++++++++ merged test case 5 ++++++++++++++++++++ 2026/08/24 22:18:27 test case 5, interval start=0, stop=200, fileId=d 2026/08/24 22:18:27 test case 5, interval start=200, stop=220, fileId=c 2026/08/24 22:18:27 ++++++++++ merged test case 6 ++++++++++++++++++++ 2026/08/24 22:18:27 test case 6, interval start=0, stop=100, fileId=xyz 2026/08/24 22:18:27 ++++++++++ merged test case 7 ++++++++++++++++++++ 2026/08/24 22:18:27 test case 7, interval start=0, stop=2097152, fileId=3,029565bf3092 2026/08/24 22:18:27 test case 7, interval start=2097152, stop=5242880, fileId=6,029632f47ae2 2026/08/24 22:18:27 test case 7, interval start=5242880, stop=8388608, fileId=2,029734c5aa10 2026/08/24 22:18:27 test case 7, interval start=8388608, stop=11534336, fileId=5,02982f80de50 2026/08/24 22:18:27 test case 7, interval start=11534336, stop=14376529, fileId=7,0299ad723803 2026/08/24 22:18:27 ++++++++++ merged test case 8 ++++++++++++++++++++ 2026/08/24 22:18:27 test case 8, interval start=0, stop=77824, fileId=4,0b3df938e301 2026/08/24 22:18:27 test case 8, interval start=77824, stop=208896, fileId=4,0b3f0c7202f0 2026/08/24 22:18:27 test case 8, interval start=208896, stop=339968, fileId=2,0b4031a72689 2026/08/24 22:18:27 test case 8, interval start=339968, stop=471040, fileId=3,0b416a557362 2026/08/24 22:18:27 test case 8, interval start=471040, stop=472225, fileId=6,0b3e0650019c --- PASS: TestIntervalMerging (0.01s) === RUN TestChunksReading 2026/08/24 22:18:27 ++++++++++ read test case 0 ++++++++++++++++++++ 2026/08/24 22:18:27 read case 0, chunk 0, offset=0, size=100, fileId=abc 2026/08/24 22:18:27 read case 0, chunk 1, offset=0, size=100, fileId=asdf 2026/08/24 22:18:27 read case 0, chunk 2, offset=0, size=50, fileId=fsad 2026/08/24 22:18:27 ++++++++++ read test case 1 ++++++++++++++++++++ 2026/08/24 22:18:27 read case 1, chunk 0, offset=50, size=100, fileId=asdf 2026/08/24 22:18:27 ++++++++++ read test case 2 ++++++++++++++++++++ 2026/08/24 22:18:27 read case 2, chunk 0, offset=20, size=30, fileId=b 2026/08/24 22:18:27 read case 2, chunk 1, offset=57, size=10, fileId=a 2026/08/24 22:18:27 ++++++++++ read test case 3 ++++++++++++++++++++ 2026/08/24 22:18:27 read case 3, chunk 0, offset=0, size=50, fileId=asdf 2026/08/24 22:18:27 read case 3, chunk 1, offset=0, size=150, fileId=xxxx 2026/08/24 22:18:27 ++++++++++ read test case 4 ++++++++++++++++++++ 2026/08/24 22:18:27 read case 4, chunk 0, offset=0, size=200, fileId=asdf 2026/08/24 22:18:27 read case 4, chunk 1, offset=0, size=150, fileId=xxxx 2026/08/24 22:18:27 ++++++++++ read test case 5 ++++++++++++++++++++ 2026/08/24 22:18:27 read case 5, chunk 0, offset=0, size=200, fileId=c 2026/08/24 22:18:27 read case 5, chunk 1, offset=130, size=20, fileId=b 2026/08/24 22:18:27 ++++++++++ read test case 6 ++++++++++++++++++++ 2026/08/24 22:18:27 read case 6, chunk 0, offset=0, size=100, fileId=xyz 2026/08/24 22:18:27 ++++++++++ read test case 7 ++++++++++++++++++++ 2026/08/24 22:18:27 read case 7, chunk 0, offset=0, size=100, fileId=abc 2026/08/24 22:18:27 read case 7, chunk 1, offset=0, size=100, fileId=asdf 2026/08/24 22:18:27 ++++++++++ read test case 8 ++++++++++++++++++++ 2026/08/24 22:18:27 read case 8, chunk 0, offset=0, size=90, fileId=abc 2026/08/24 22:18:27 read case 8, chunk 1, offset=0, size=100, fileId=asdf 2026/08/24 22:18:27 read case 8, chunk 2, offset=0, size=110, fileId=fsad 2026/08/24 22:18:27 ++++++++++ read test case 9 ++++++++++++++++++++ 2026/08/24 22:18:27 read case 9, chunk 0, offset=0, size=43175936, fileId=2,111fc2cbfac1 2026/08/24 22:18:27 read case 9, chunk 1, offset=0, size=9805824, fileId=2,112a36ea7f85 2026/08/24 22:18:27 read case 9, chunk 2, offset=0, size=19582976, fileId=4,112d5f31c5e7 2026/08/24 22:18:27 read case 9, chunk 3, offset=0, size=60690432, fileId=1,113245f0cdb6 2026/08/24 22:18:27 read case 9, chunk 4, offset=0, size=4014080, fileId=3,1141a70733b5 2026/08/24 22:18:27 read case 9, chunk 5, offset=0, size=16309588, fileId=1,114201d5bbdb --- PASS: TestChunksReading (0.00s) === RUN TestViewFromVisibleIntervals --- PASS: TestViewFromVisibleIntervals (0.00s) === RUN TestViewFromVisibleIntervals2 --- PASS: TestViewFromVisibleIntervals2 (0.00s) === RUN TestViewFromVisibleIntervals3 --- PASS: TestViewFromVisibleIntervals3 (0.00s) === RUN TestCompactFileChunks3 --- PASS: TestCompactFileChunks3 (0.00s) === RUN TestFilerConf --- PASS: TestFilerConf (0.00s) === RUN TestWormInheritance --- PASS: TestWormInheritance (0.00s) === RUN TestWormLegacyFalseIsNotAnOverride --- PASS: TestWormLegacyFalseIsNotAnOverride (0.00s) === RUN TestWormSurvivesRoundTrip --- PASS: TestWormSurvivesRoundTrip (0.04s) === RUN TestClonePathConf --- PASS: TestClonePathConf (0.00s) === RUN TestClonePathConfNil --- PASS: TestClonePathConfNil (0.00s) === RUN TestApplyBucketQuotaReadOnly --- PASS: TestApplyBucketQuotaReadOnly (0.00s) === RUN TestClearReadOnly --- PASS: TestClearReadOnly (0.00s) === RUN TestClearBucketLifecycleDayTTLs_NoFilerConf --- PASS: TestClearBucketLifecycleDayTTLs_NoFilerConf (0.02s) === RUN TestClearBucketLifecycleDayTTLs_NoMatchingRule --- PASS: TestClearBucketLifecycleDayTTLs_NoMatchingRule (0.00s) === RUN TestClearBucketLifecycleDayTTLs_RemovesDayTTLUnderBucket I0824 22:18:27.130681 filer_conf.go:461 lifecycle migration: dropping legacy day-TTL rule /buckets/mybucket/ ttl=7d --- PASS: TestClearBucketLifecycleDayTTLs_RemovesDayTTLUnderBucket (0.00s) === RUN TestClearBucketLifecycleDayTTLs_KeepsNonDayTTL --- PASS: TestClearBucketLifecycleDayTTLs_KeepsNonDayTTL (0.00s) === RUN TestClearBucketLifecycleDayTTLs_KeepsOperatorSettingsAndClearsOnlyTheTTL I0824 22:18:27.133424 filer_conf.go:464 lifecycle migration: clearing legacy day-TTL 30d on operator rule /buckets/mybucket/logs/ --- PASS: TestClearBucketLifecycleDayTTLs_KeepsOperatorSettingsAndClearsOnlyTheTTL (0.00s) === RUN TestClearBucketLifecycleDayTTLs_KeepsWormSettings I0824 22:18:27.136029 filer_conf.go:464 lifecycle migration: clearing legacy day-TTL 7d on operator rule /buckets/mybucket/locked/ --- PASS: TestClearBucketLifecycleDayTTLs_KeepsWormSettings (0.00s) === RUN TestClearBucketLifecycleDayTTLs_KeepsOtherBucketsAndCollections I0824 22:18:27.140612 filer_conf.go:461 lifecycle migration: dropping legacy day-TTL rule /buckets/mybucket/ ttl=7d --- PASS: TestClearBucketLifecycleDayTTLs_KeepsOtherBucketsAndCollections (0.00s) === RUN TestSaveFilerConfConditionally_RejectsUpdateModifiedConcurrently --- PASS: TestSaveFilerConfConditionally_RejectsUpdateModifiedConcurrently (0.00s) === RUN TestSaveFilerConfConditionally_ExactContentCheckRejectsConcurrentModification --- PASS: TestSaveFilerConfConditionally_ExactContentCheckRejectsConcurrentModification (0.00s) === RUN TestSaveFilerConfConditionally_RejectsCreateWhenCreatedConcurrently --- PASS: TestSaveFilerConfConditionally_RejectsCreateWhenCreatedConcurrently (0.00s) === RUN TestDeletionRetryQueue_AddAndRetrieve --- PASS: TestDeletionRetryQueue_AddAndRetrieve (0.00s) === RUN TestDeletionRetryQueue_ExponentialBackoff filer_deletion_test.go:80: Exponential backoff delays: [5m0.00000652s 10m0.00003402s 20m0.00000402s 40m0.00000492s 1h20m0.00000436s] --- PASS: TestDeletionRetryQueue_ExponentialBackoff (0.00s) === RUN TestDeletionRetryQueue_OverflowProtection --- PASS: TestDeletionRetryQueue_OverflowProtection (0.00s) === RUN TestDeletionRetryQueue_MaxAttemptsReached W0824 22:18:27.161386 filer_deletion.go:247 max retry attempts (10) reached for file1, last error: final error --- PASS: TestDeletionRetryQueue_MaxAttemptsReached (0.03s) === RUN TestCalculateBackoff --- PASS: TestCalculateBackoff (0.00s) === RUN TestIsRetryableError --- PASS: TestIsRetryableError (0.00s) === RUN TestDeletionRetryQueue_HeapOrdering --- PASS: TestDeletionRetryQueue_HeapOrdering (0.00s) === RUN TestDeletionRetryQueue_DuplicateFileIds --- PASS: TestDeletionRetryQueue_DuplicateFileIds (0.00s) === RUN TestEnsureEntryInodeMatchesFuseDerivation --- PASS: TestEnsureEntryInodeMatchesFuseDerivation (0.00s) === RUN TestEnsureEntryInodeSharesAcrossHardLinks --- PASS: TestEnsureEntryInodeSharesAcrossHardLinks (0.00s) === RUN TestCreateEntryAssignsInodeWhenMissing --- PASS: TestCreateEntryAssignsInodeWhenMissing (0.15s) === RUN TestCreateEntryAssignsInodesToAutoCreatedParents --- PASS: TestCreateEntryAssignsInodesToAutoCreatedParents (0.03s) === RUN TestUpdateEntryPreservesExistingInode E0824 22:18:27.382758 filer.go:489 existing /doc.txt is a file --- PASS: TestUpdateEntryPreservesExistingInode (0.01s) === RUN TestUpdateEntryBackfillsMissingLegacyInode --- PASS: TestUpdateEntryBackfillsMissingLegacyInode (0.01s) === RUN TestMaybeLazyFetchFromRemote_HitsRemoteAndPersists --- PASS: TestMaybeLazyFetchFromRemote_HitsRemoteAndPersists (0.06s) === RUN TestMaybeLazyFetchFromRemote_NotUnderMount --- PASS: TestMaybeLazyFetchFromRemote_NotUnderMount (0.00s) === RUN TestMaybeLazyFetchFromRemote_RemoteObjectNotFound --- PASS: TestMaybeLazyFetchFromRemote_RemoteObjectNotFound (0.03s) === RUN TestMaybeLazyFetchFromRemote_CreateEntryFailureReturnsInMemoryEntry E0824 22:18:27.499409 filer.go:272 insert entry /buckets/mybucket/failfile.txt: simulated store failure W0824 22:18:27.499745 filer_lazy_remote.go:102 maybeLazyFetchFromRemote: failed to persist filer entry for /buckets/mybucket/failfile.txt: insert entry /buckets/mybucket/failfile.txt: simulated store failure --- PASS: TestMaybeLazyFetchFromRemote_CreateEntryFailureReturnsInMemoryEntry (0.01s) === RUN TestMaybeLazyFetchFromRemote_LongestPrefixMount --- PASS: TestMaybeLazyFetchFromRemote_LongestPrefixMount (0.01s) === RUN TestMaybeLazyFetchFromRemote_ContextGuardPreventsRecursion --- PASS: TestMaybeLazyFetchFromRemote_ContextGuardPreventsRecursion (0.01s) === RUN TestFindEntry_LazyFetchOnMiss --- PASS: TestFindEntry_LazyFetchOnMiss (0.00s) === RUN TestDeleteEntryMetaAndData_RemoteOnlyFileDeletesRemoteAndMetadata --- PASS: TestDeleteEntryMetaAndData_RemoteOnlyFileDeletesRemoteAndMetadata (0.01s) === RUN TestDeleteEntryMetaAndData_IsFromOtherClusterSkipsRemoteDelete --- PASS: TestDeleteEntryMetaAndData_IsFromOtherClusterSkipsRemoteDelete (0.04s) === RUN TestDeleteEntryMetaAndData_UnmountedDirectorySkipsRemoteDelete --- PASS: TestDeleteEntryMetaAndData_UnmountedDirectorySkipsRemoteDelete (0.05s) === RUN TestDeleteEntryMetaAndData_RemoteOnlyFileNotUnderMountSkipsRemoteDelete --- PASS: TestDeleteEntryMetaAndData_RemoteOnlyFileNotUnderMountSkipsRemoteDelete (0.01s) === RUN TestDeleteEntryMetaAndData_RemoteMountWithoutClientResolutionKeepsMetadata --- PASS: TestDeleteEntryMetaAndData_RemoteMountWithoutClientResolutionKeepsMetadata (0.01s) === RUN TestDeleteEntryMetaAndData_LocalDeleteFailurePreservesMetadata --- PASS: TestDeleteEntryMetaAndData_LocalDeleteFailurePreservesMetadata (0.01s) === RUN TestDeleteEntryMetaAndData_RemoteDeleteNotFoundStillDeletesMetadata --- PASS: TestDeleteEntryMetaAndData_RemoteDeleteNotFoundStillDeletesMetadata (0.05s) === RUN TestDeleteEntryMetaAndData_RemoteDeleteErrorKeepsMetadata --- PASS: TestDeleteEntryMetaAndData_RemoteDeleteErrorKeepsMetadata (0.01s) === RUN TestDeleteEntryMetaAndData_DirectoryUnderMountDeletesRemoteDirectory --- PASS: TestDeleteEntryMetaAndData_DirectoryUnderMountDeletesRemoteDirectory (0.01s) === RUN TestDeleteEntryMetaAndData_RecursiveFolderDeleteRemotesChildren --- PASS: TestDeleteEntryMetaAndData_RecursiveFolderDeleteRemotesChildren (0.01s) === RUN TestMaybeLazyListFromRemote_PopulatesStoreFromRemote --- PASS: TestMaybeLazyListFromRemote_PopulatesStoreFromRemote (0.01s) === RUN TestMaybeLazyListFromRemote_DisabledWhenTTLZero --- PASS: TestMaybeLazyListFromRemote_DisabledWhenTTLZero (0.01s) === RUN TestMaybeLazyListFromRemote_TTLCachePreventsSecondCall --- PASS: TestMaybeLazyListFromRemote_TTLCachePreventsSecondCall (0.01s) === RUN TestMaybeLazyListFromRemote_NotUnderMount --- PASS: TestMaybeLazyListFromRemote_NotUnderMount (0.03s) === RUN TestMaybeLazyListFromRemote_SkipsLocalOnlyEntries --- PASS: TestMaybeLazyListFromRemote_SkipsLocalOnlyEntries (0.05s) === RUN TestMaybeLazyListFromRemote_MergesExistingRemoteEntry --- PASS: TestMaybeLazyListFromRemote_MergesExistingRemoteEntry (0.01s) === RUN TestMaybeLazyListFromRemote_ContextGuardPreventsRecursion --- PASS: TestMaybeLazyListFromRemote_ContextGuardPreventsRecursion (0.01s) === RUN TestResolveMetadataLogAssignDiskTypeUsesPathRule --- PASS: TestResolveMetadataLogAssignDiskTypeUsesPathRule (0.00s) === RUN TestResolveMetadataLogAssignDiskTypeFallsBackToFilerDefault --- PASS: TestResolveMetadataLogAssignDiskTypeFallsBackToFilerDefault (0.00s) === RUN TestResolveMetadataLogAssignDiskTypeNilFilerConf --- PASS: TestResolveMetadataLogAssignDiskTypeNilFilerConf (0.00s) === RUN TestMetaLogInflightFloor --- PASS: TestMetaLogInflightFloor (0.00s) === RUN TestLocalFlushedThroughTsNsBoundsInflight --- PASS: TestLocalFlushedThroughTsNsBoundsInflight (0.01s) === RUN TestLocalDeliveredThroughTsNsBoundsInflight --- PASS: TestLocalDeliveredThroughTsNsBoundsInflight (0.01s) === RUN TestClaimFencesFutureStamps --- PASS: TestClaimFencesFutureStamps (0.01s) === RUN TestMetaLogInflightStampMonotonic --- PASS: TestMetaLogInflightStampMonotonic (0.00s) === RUN TestPersistedLogScanStartCoversSpanningWindow --- PASS: TestPersistedLogScanStartCoversSpanningWindow (0.00s) === RUN TestPersistedLogScanStartCrossesMidnight --- PASS: TestPersistedLogScanStartCrossesMidnight (0.00s) === RUN TestSpanningLogFileIsNotSkipped --- PASS: TestSpanningLogFileIsNotSkipped (0.00s) === RUN TestClampLogRefsCursorNeverRewinds --- PASS: TestClampLogRefsCursorNeverRewinds (0.00s) === RUN TestPersistedLogScanStartTsNsMinuteAligned --- PASS: TestPersistedLogScanStartTsNsMinuteAligned (0.00s) === RUN TestLastShippedLogEntryTsNs === RUN TestLastShippedLogEntryTsNs/a_fully_readable_file_answers_with_its_tail,_complete === RUN TestLastShippedLogEntryTsNs/the_prefix_ends_at_the_first_missing_chunk === RUN TestLastShippedLogEntryTsNs/nothing_readable_is_a_clean_no-answer === RUN TestLastShippedLogEntryTsNs/an_entirely_missing_last_file_falls_back_to_earlier_files === RUN TestLastShippedLogEntryTsNs/a_cached_chunk_on_a_dead_volume_ends_the_prefix === RUN TestLastShippedLogEntryTsNs/a_dead_needle_behind_a_warm_cache_is_the_accepted_residual === RUN TestLastShippedLogEntryTsNs/spanning_records_stream_the_shipped_list --- PASS: TestLastShippedLogEntryTsNs (0.03s) --- PASS: TestLastShippedLogEntryTsNs/a_fully_readable_file_answers_with_its_tail,_complete (0.00s) --- PASS: TestLastShippedLogEntryTsNs/the_prefix_ends_at_the_first_missing_chunk (0.00s) --- PASS: TestLastShippedLogEntryTsNs/nothing_readable_is_a_clean_no-answer (0.00s) --- PASS: TestLastShippedLogEntryTsNs/an_entirely_missing_last_file_falls_back_to_earlier_files (0.00s) --- PASS: TestLastShippedLogEntryTsNs/a_cached_chunk_on_a_dead_volume_ends_the_prefix (0.00s) --- PASS: TestLastShippedLogEntryTsNs/a_dead_needle_behind_a_warm_cache_is_the_accepted_residual (0.00s) --- PASS: TestLastShippedLogEntryTsNs/spanning_records_stream_the_shipped_list (0.02s) === RUN TestProtoMarshal x to: 234,2423423422 * 2342342354223234,2342342342"6 0Ø: text/jsonPØѝ --- PASS: TestProtoMarshal (0.00s) === RUN TestNextLogPiece === RUN TestNextLogPiece/fits_in_one_piece === RUN TestNextLogPiece/many_small_records === RUN TestNextLogPiece/one_record_far_over_the_limit === RUN TestNextLogPiece/oversized_record_between_small_ones === RUN TestNextLogPiece/record_exactly_at_the_limit --- PASS: TestNextLogPiece (0.01s) --- PASS: TestNextLogPiece/fits_in_one_piece (0.00s) --- PASS: TestNextLogPiece/many_small_records (0.00s) --- PASS: TestNextLogPiece/one_record_far_over_the_limit (0.00s) --- PASS: TestNextLogPiece/oversized_record_between_small_ones (0.00s) --- PASS: TestNextLogPiece/record_exactly_at_the_limit (0.00s) === RUN TestNextLogPieceKeepsRecordBoundaries --- PASS: TestNextLogPieceKeepsRecordBoundaries (0.00s) === RUN TestNextLogPieceTruncatedTail --- PASS: TestNextLogPieceTruncatedTail (0.00s) === RUN TestVolumeFileSizeLimit === RUN TestVolumeFileSizeLimit/volume_server_size_rejection === RUN TestVolumeFileSizeLimit/default_limit === RUN TestVolumeFileSizeLimit/unrelated_failure === RUN TestVolumeFileSizeLimit/no_byte_count --- PASS: TestVolumeFileSizeLimit (0.04s) --- PASS: TestVolumeFileSizeLimit/volume_server_size_rejection (0.03s) --- PASS: TestVolumeFileSizeLimit/default_limit (0.00s) --- PASS: TestVolumeFileSizeLimit/unrelated_failure (0.00s) --- PASS: TestVolumeFileSizeLimit/no_byte_count (0.00s) === RUN TestNextLogPieceHonorsLoweredLimit --- PASS: TestNextLogPieceHonorsLoweredLimit (1.00s) === RUN TestOnBucketEventsRenameIntoBucketsRootCreatesBucket --- PASS: TestOnBucketEventsRenameIntoBucketsRootCreatesBucket (0.00s) === RUN TestDetectBucket === RUN TestDetectBucket/object_in_bucket === RUN TestDetectBucket/bucket_root === RUN TestDetectBucket/not_under_buckets_path === RUN TestDetectBucket/buckets_path_itself === RUN TestDetectBucket/custom_buckets_path === RUN TestDetectBucket/nested_object --- PASS: TestDetectBucket (0.00s) --- PASS: TestDetectBucket/object_in_bucket (0.00s) --- PASS: TestDetectBucket/bucket_root (0.00s) --- PASS: TestDetectBucket/not_under_buckets_path (0.00s) --- PASS: TestDetectBucket/buckets_path_itself (0.00s) --- PASS: TestDetectBucket/custom_buckets_path (0.00s) --- PASS: TestDetectBucket/nested_object (0.00s) === RUN TestFilerStoreWrapperMimeNormalization === RUN TestFilerStoreWrapperMimeNormalization/files_strip_octet-stream/insert === RUN TestFilerStoreWrapperMimeNormalization/files_strip_octet-stream/update === RUN TestFilerStoreWrapperMimeNormalization/directories_keep_octet-stream/insert === RUN TestFilerStoreWrapperMimeNormalization/directories_keep_octet-stream/update --- PASS: TestFilerStoreWrapperMimeNormalization (0.01s) --- PASS: TestFilerStoreWrapperMimeNormalization/files_strip_octet-stream/insert (0.00s) --- PASS: TestFilerStoreWrapperMimeNormalization/files_strip_octet-stream/update (0.00s) --- PASS: TestFilerStoreWrapperMimeNormalization/directories_keep_octet-stream/insert (0.00s) --- PASS: TestFilerStoreWrapperMimeNormalization/directories_keep_octet-stream/update (0.00s) === RUN TestFilerStoreWrapperWriteOpsRejectCancelledContext === RUN TestFilerStoreWrapperWriteOpsRejectCancelledContext/InsertEntry/cancelled === RUN TestFilerStoreWrapperWriteOpsRejectCancelledContext/InsertEntry/deadline_exceeded === RUN TestFilerStoreWrapperWriteOpsRejectCancelledContext/InsertEntryKnownAbsent/cancelled === RUN TestFilerStoreWrapperWriteOpsRejectCancelledContext/InsertEntryKnownAbsent/deadline_exceeded === RUN TestFilerStoreWrapperWriteOpsRejectCancelledContext/UpdateEntry/cancelled === RUN TestFilerStoreWrapperWriteOpsRejectCancelledContext/UpdateEntry/deadline_exceeded === RUN TestFilerStoreWrapperWriteOpsRejectCancelledContext/DeleteEntry/cancelled === RUN TestFilerStoreWrapperWriteOpsRejectCancelledContext/DeleteEntry/deadline_exceeded === RUN TestFilerStoreWrapperWriteOpsRejectCancelledContext/DeleteOneEntry/cancelled === RUN TestFilerStoreWrapperWriteOpsRejectCancelledContext/DeleteOneEntry/deadline_exceeded === RUN TestFilerStoreWrapperWriteOpsRejectCancelledContext/DeleteFolderChildren/cancelled === RUN TestFilerStoreWrapperWriteOpsRejectCancelledContext/DeleteFolderChildren/deadline_exceeded === RUN TestFilerStoreWrapperWriteOpsRejectCancelledContext/BeginTransaction/cancelled === RUN TestFilerStoreWrapperWriteOpsRejectCancelledContext/BeginTransaction/deadline_exceeded === RUN TestFilerStoreWrapperWriteOpsRejectCancelledContext/CommitTransaction/cancelled === RUN TestFilerStoreWrapperWriteOpsRejectCancelledContext/CommitTransaction/deadline_exceeded === RUN TestFilerStoreWrapperWriteOpsRejectCancelledContext/KvPut/cancelled === RUN TestFilerStoreWrapperWriteOpsRejectCancelledContext/KvPut/deadline_exceeded === RUN TestFilerStoreWrapperWriteOpsRejectCancelledContext/KvDelete/cancelled === RUN TestFilerStoreWrapperWriteOpsRejectCancelledContext/KvDelete/deadline_exceeded --- PASS: TestFilerStoreWrapperWriteOpsRejectCancelledContext (0.04s) --- PASS: TestFilerStoreWrapperWriteOpsRejectCancelledContext/InsertEntry/cancelled (0.00s) --- PASS: TestFilerStoreWrapperWriteOpsRejectCancelledContext/InsertEntry/deadline_exceeded (0.00s) --- PASS: TestFilerStoreWrapperWriteOpsRejectCancelledContext/InsertEntryKnownAbsent/cancelled (0.00s) --- PASS: TestFilerStoreWrapperWriteOpsRejectCancelledContext/InsertEntryKnownAbsent/deadline_exceeded (0.00s) --- PASS: TestFilerStoreWrapperWriteOpsRejectCancelledContext/UpdateEntry/cancelled (0.00s) --- PASS: TestFilerStoreWrapperWriteOpsRejectCancelledContext/UpdateEntry/deadline_exceeded (0.00s) --- PASS: TestFilerStoreWrapperWriteOpsRejectCancelledContext/DeleteEntry/cancelled (0.00s) --- PASS: TestFilerStoreWrapperWriteOpsRejectCancelledContext/DeleteEntry/deadline_exceeded (0.00s) --- PASS: TestFilerStoreWrapperWriteOpsRejectCancelledContext/DeleteOneEntry/cancelled (0.00s) --- PASS: TestFilerStoreWrapperWriteOpsRejectCancelledContext/DeleteOneEntry/deadline_exceeded (0.00s) --- PASS: TestFilerStoreWrapperWriteOpsRejectCancelledContext/DeleteFolderChildren/cancelled (0.00s) --- PASS: TestFilerStoreWrapperWriteOpsRejectCancelledContext/DeleteFolderChildren/deadline_exceeded (0.00s) --- PASS: TestFilerStoreWrapperWriteOpsRejectCancelledContext/BeginTransaction/cancelled (0.00s) --- PASS: TestFilerStoreWrapperWriteOpsRejectCancelledContext/BeginTransaction/deadline_exceeded (0.00s) --- PASS: TestFilerStoreWrapperWriteOpsRejectCancelledContext/CommitTransaction/cancelled (0.00s) --- PASS: TestFilerStoreWrapperWriteOpsRejectCancelledContext/CommitTransaction/deadline_exceeded (0.00s) --- PASS: TestFilerStoreWrapperWriteOpsRejectCancelledContext/KvPut/cancelled (0.00s) --- PASS: TestFilerStoreWrapperWriteOpsRejectCancelledContext/KvPut/deadline_exceeded (0.00s) --- PASS: TestFilerStoreWrapperWriteOpsRejectCancelledContext/KvDelete/cancelled (0.00s) --- PASS: TestFilerStoreWrapperWriteOpsRejectCancelledContext/KvDelete/deadline_exceeded (0.00s) === RUN TestFilerStoreWrapperWriteOpsSucceedWithActiveContext --- PASS: TestFilerStoreWrapperWriteOpsSucceedWithActiveContext (0.00s) === RUN TestFilerStoreWrapperReadOpsSucceedWithCancelledContext --- PASS: TestFilerStoreWrapperReadOpsSucceedWithCancelledContext (0.00s) === RUN TestFilerStoreWrapperRollbackSucceedsWithCancelledContext --- PASS: TestFilerStoreWrapperRollbackSucceedsWithCancelledContext (0.00s) === RUN TestIntervalList_Overlay [0,25) 6 6 [25,50) 1 1 [50,150) 2 2 [175,210) 5 5 [210,225) 3 3 [225,250) 4 4 [0,25) 6 6 [25,50) 1 1 [50,150) 7 7 [175,210) 5 5 [210,225) 3 3 [225,250) 4 4 --- PASS: TestIntervalList_Overlay (0.00s) === RUN TestIntervalList_Overlay2 [0,50) 2 2 [50,100) 1 1 --- PASS: TestIntervalList_Overlay2 (0.00s) === RUN TestIntervalList_Overlay3 [0,60) 2 2 [60,100) 1 1 --- PASS: TestIntervalList_Overlay3 (0.00s) === RUN TestIntervalList_Overlay4 [0,100) 2 2 --- PASS: TestIntervalList_Overlay4 (0.00s) === RUN TestIntervalList_Overlay5 [0,110) 2 2 --- PASS: TestIntervalList_Overlay5 (0.00s) === RUN TestIntervalList_Overlay6 [50,110) 2 2 --- PASS: TestIntervalList_Overlay6 (0.00s) === RUN TestIntervalList_Overlay7 [50,90) 2 2 [90,100) 1 1 --- PASS: TestIntervalList_Overlay7 (0.00s) === RUN TestIntervalList_Overlay8 [50,60) 1 1 [60,90) 2 2 [90,100) 1 1 --- PASS: TestIntervalList_Overlay8 (0.00s) === RUN TestIntervalList_Overlay9 [50,60) 1 1 [60,100) 2 2 --- PASS: TestIntervalList_Overlay9 (0.00s) === RUN TestIntervalList_Overlay10 [50,60) 1 1 [60,110) 2 2 --- PASS: TestIntervalList_Overlay10 (0.00s) === RUN TestIntervalList_Overlay11 [0,90) 5 5 [90,100) 1 1 [100,110) 2 2 --- PASS: TestIntervalList_Overlay11 (0.00s) === RUN TestIntervalList_insertInterval1 [50,150) 2 2 [200,250) 3 3 --- PASS: TestIntervalList_insertInterval1 (0.00s) === RUN TestIntervalList_insertInterval2 [0,25) 3 3 [50,150) 2 2 --- PASS: TestIntervalList_insertInterval2 (0.00s) === RUN TestIntervalList_insertInterval3 [0,75) 3 3 [75,150) 2 2 [200,250) 4 4 --- PASS: TestIntervalList_insertInterval3 (0.00s) === RUN TestIntervalList_insertInterval4 [0,200) 3 3 [200,250) 4 4 --- PASS: TestIntervalList_insertInterval4 (0.00s) === RUN TestIntervalList_insertInterval5 [0,225) 5 5 [225,250) 4 4 --- PASS: TestIntervalList_insertInterval5 (0.00s) === RUN TestIntervalList_insertInterval6 [0,50) 1 1 [50,150) 2 2 [150,200) 1 1 [200,250) 4 4 [250,275) 1 1 --- PASS: TestIntervalList_insertInterval6 (0.00s) === RUN TestIntervalList_insertInterval7 [50,150) 2 2 [150,200) 1 1 [200,250) 4 4 [250,275) 1 1 --- PASS: TestIntervalList_insertInterval7 (0.00s) === RUN TestIntervalList_insertInterval8 [50,75) 2 2 [75,200) 3 3 [200,250) 4 4 [250,275) 3 3 --- PASS: TestIntervalList_insertInterval8 (0.00s) === RUN TestIntervalList_insertInterval9 [50,150) 3 3 [200,250) 4 4 --- PASS: TestIntervalList_insertInterval9 (0.00s) === RUN TestIntervalList_insertInterval10 [50,100) 2 2 [100,200) 5 5 [200,300) 4 4 --- PASS: TestIntervalList_insertInterval10 (0.00s) === RUN TestIntervalList_insertInterval11 [0,64) 1 1 [64,68) 2 2 [68,72) 4 4 [72,136) 3 3 --- PASS: TestIntervalList_insertInterval11 (0.00s) === RUN TestIntervalList_insertIntervalStruct [0,64) 1 {1 0 0} [64,68) 4 {4 0 0} [68,72) 2 {2 0 0} [72,136) 3 {3 0 0} --- PASS: TestIntervalList_insertIntervalStruct (0.00s) === RUN TestReplicateMetadataChangeRetriesTransientFailure I0824 22:18:29.070587 retry.go:95 retry replicate metadata change from peer-transient:1: err: i/o timeout talking to store I0824 22:18:30.077172 retry.go:88 retry replicate metadata change from peer-transient:1 successfully --- PASS: TestReplicateMetadataChangeRetriesTransientFailure (1.01s) === RUN TestReplicateMetadataChangeGivesUpLoudlyOnPermanentFailure E0824 22:18:30.092252 meta_aggregator.go:563 giving up replicating metadata change from peer-permanent:1 for /buckets/poison-bucket (ts=1787609910091941712): entry checksum mismatch --- PASS: TestReplicateMetadataChangeGivesUpLoudlyOnPermanentFailure (0.01s) === RUN TestOnPeerUpdateRepeatedAdd --- PASS: TestOnPeerUpdateRepeatedAdd (0.00s) === RUN TestPeerWatermarkBookkeeping --- PASS: TestPeerWatermarkBookkeeping (0.00s) === RUN TestPeerWatermarkRemovalGrace --- PASS: TestPeerWatermarkRemovalGrace (0.00s) === RUN TestPeerFlushWatermarkBookkeeping --- PASS: TestPeerFlushWatermarkBookkeeping (0.00s) === RUN TestWatermarkAdvancedChansAreScoped --- PASS: TestWatermarkAdvancedChansAreScoped (0.00s) === RUN TestNotifyUpdateEventRecordsRequestMetadataEvent I0824 22:18:30.101287 meta_aggregator.go:304 loopSubscribeToOneFiler read 127.0.0.1:1 start from 2026-08-24 22:18:30.099978129 +0000 UTC 1787609910099978129 --- PASS: TestNotifyUpdateEventRecordsRequestMetadataEvent (0.01s) === RUN TestNotifyUpdateEventReloadsLocalFilerConfiguration I0824 22:18:30.126508 meta_aggregator.go:310 stop subscribing peer 127.0.0.1:1 meta change I0824 22:18:30.149431 filer_on_meta_event.go:92 processing directory:"/etc/seaweedfs" event_notification:{new_entry:{name:"filer.conf" attributes:{file_size:460 mtime:-62135596800 crtime:-62135596800 ctime:-62135596800} content:"{\n \"version\": 1,\n \"locations\": [\n {\n \"locationPrefix\": \"/data/\",\n \"collection\": \"hot\",\n \"replication\": \"\",\n \"ttl\": \"\",\n \"diskType\": \"\",\n \"fsync\": false,\n \"volumeGrowthCount\": 0,\n \"readOnly\": false,\n \"dataCenter\": \"\",\n \"rack\": \"\",\n \"dataNode\": \"\",\n \"maxFileNameLength\": 0,\n \"disableChunkDeletion\": false,\n \"wormGracePeriodSeconds\": \"0\",\n \"wormRetentionTimeSeconds\": \"0\"\n }\n ]\n}"} new_parent_path:"/etc/seaweedfs" signatures:42} ts_ns:1787609910148914612 --- PASS: TestNotifyUpdateEventReloadsLocalFilerConfiguration (0.04s) === RUN TestMountPeerRegistry_RegisterAndList --- PASS: TestMountPeerRegistry_RegisterAndList (0.00s) === RUN TestMountPeerRegistry_RenewExtendsExpiry --- PASS: TestMountPeerRegistry_RenewExtendsExpiry (0.00s) === RUN TestMountPeerRegistry_ExpirationDropsEntry --- PASS: TestMountPeerRegistry_ExpirationDropsEntry (0.00s) === RUN TestMountPeerRegistry_SweepCountsEvictions --- PASS: TestMountPeerRegistry_SweepCountsEvictions (0.00s) === RUN TestMountPeerRegistry_NegativeTTLFallsBackToDefault --- PASS: TestMountPeerRegistry_NegativeTTLFallsBackToDefault (0.00s) === RUN TestMountPeerRegistry_EmptyList --- PASS: TestMountPeerRegistry_EmptyList (0.00s) === RUN TestMountPeerRegistry_EmptyPeerAddrRejected --- PASS: TestMountPeerRegistry_EmptyPeerAddrRejected (0.00s) === RUN TestMountPeerRegistry_TTLCapped --- PASS: TestMountPeerRegistry_TTLCapped (0.00s) === RUN TestMountPeerRegistry_CapacityLimit --- PASS: TestMountPeerRegistry_CapacityLimit (0.14s) === RUN TestPersistedLogCacheHitMiss --- PASS: TestPersistedLogCacheHitMiss (0.00s) === RUN TestPersistedLogCacheNotCachedWhenUncacheable --- PASS: TestPersistedLogCacheNotCachedWhenUncacheable (0.00s) === RUN TestPersistedLogCacheSingleFlight --- PASS: TestPersistedLogCacheSingleFlight (0.02s) === RUN TestPersistedLogCacheEviction --- PASS: TestPersistedLogCacheEviction (0.00s) === RUN TestDecodeLogRecords --- PASS: TestDecodeLogRecords (0.00s) === RUN TestDecodeLogRecordsIncomplete --- PASS: TestDecodeLogRecordsIncomplete (0.00s) === RUN TestLogFileIteratorChunkedFiltering --- PASS: TestLogFileIteratorChunkedFiltering (0.00s) === RUN TestLogFileIteratorSkipsColdChunks --- PASS: TestLogFileIteratorSkipsColdChunks (0.00s) === RUN TestLogFileIteratorSharesDecodedChunks --- PASS: TestLogFileIteratorSharesDecodedChunks (0.00s) === RUN TestLogFileIteratorStreamsWhenRecordsSpanChunks --- PASS: TestLogFileIteratorStreamsWhenRecordsSpanChunks (0.00s) === RUN TestLogFileIteratorStreamFallbackResumesAfterYielded --- PASS: TestLogFileIteratorStreamFallbackResumesAfterYielded (0.00s) === RUN TestLogFileIteratorLoadErrorPropagates --- PASS: TestLogFileIteratorLoadErrorPropagates (0.00s) === RUN TestPersistedLogCacheIdleEviction --- PASS: TestPersistedLogCacheIdleEviction (0.00s) === RUN TestDecodeLogRecordsRejectsImplausibleRecords --- PASS: TestDecodeLogRecordsRejectsImplausibleRecords (0.00s) === RUN TestPersistedLogCacheLoadLargerThanBudget --- PASS: TestPersistedLogCacheLoadLargerThanBudget (0.00s) === RUN TestMergeRemoteContentEncoding --- PASS: TestMergeRemoteContentEncoding (0.00s) === RUN TestReaderAt --- PASS: TestReaderAt (0.00s) === RUN TestReaderAt0 --- PASS: TestReaderAt0 (0.00s) === RUN TestReaderAt1 --- PASS: TestReaderAt1 (0.00s) === RUN TestReaderAtGappedChunksDoNotLeak --- PASS: TestReaderAtGappedChunksDoNotLeak (0.00s) === RUN TestReaderAtSparseFileDoesNotLeak --- PASS: TestReaderAtSparseFileDoesNotLeak (0.00s) === RUN TestReaderAtParallelFailureReturnsContiguousPrefix --- PASS: TestReaderAtParallelFailureReturnsContiguousPrefix (0.00s) === RUN TestReaderCacheRetryAfterCacheInvalidation I0824 22:18:30.365466 stream.go:114 read chunk 425141,ef8914e9bdbbe8cb6838191f failed, invalidating cache and retrying: 404 Not Found: not found I0824 22:18:30.366173 stream.go:131 retrying read chunk 425141,ef8914e9bdbbe8cb6838191f with 1 new locations --- PASS: TestReaderCacheRetryAfterCacheInvalidation (0.01s) === RUN TestReaderCacheRemovesFailedDownloader --- PASS: TestReaderCacheRemovesFailedDownloader (0.00s) === RUN TestReaderCacheContextCancellation --- PASS: TestReaderCacheContextCancellation (0.00s) === RUN TestReaderCacheFallbackToChunkCache --- PASS: TestReaderCacheFallbackToChunkCache (0.00s) === RUN TestReaderCacheMultipleReadersWaitForSameChunk --- PASS: TestReaderCacheMultipleReadersWaitForSameChunk (0.00s) === RUN TestReaderCachePartialRead === RUN TestReaderCachePartialRead/read_from_start === RUN TestReaderCachePartialRead/read_from_middle === RUN TestReaderCachePartialRead/read_to_end === RUN TestReaderCachePartialRead/read_single_byte --- PASS: TestReaderCachePartialRead (0.00s) --- PASS: TestReaderCachePartialRead/read_from_start (0.00s) --- PASS: TestReaderCachePartialRead/read_from_middle (0.00s) --- PASS: TestReaderCachePartialRead/read_to_end (0.00s) --- PASS: TestReaderCachePartialRead/read_single_byte (0.00s) === RUN TestReaderCacheCleanup --- PASS: TestReaderCacheCleanup (0.00s) === RUN TestSingleChunkCacherDoneSignal --- PASS: TestSingleChunkCacherDoneSignal (0.00s) === RUN TestSingleChunkCacherLookupError --- PASS: TestSingleChunkCacherLookupError (0.00s) === RUN TestSingleChunkCacherContextCancellationDuringLookup --- PASS: TestSingleChunkCacherContextCancellationDuringLookup (0.00s) === RUN TestSingleChunkCacherMultipleReadersWaitForDownload --- PASS: TestSingleChunkCacherMultipleReadersWaitForDownload (0.00s) === RUN TestReaderCacheDownloaderDedup --- PASS: TestReaderCacheDownloaderDedup (0.00s) === RUN TestSingleChunkCacherOneReaderCancelsOthersContinue --- PASS: TestSingleChunkCacherOneReaderCancelsOthersContinue (0.00s) === RUN TestReaderPatternSequentialAndHysteresis --- PASS: TestReaderPatternSequentialAndHysteresis (0.00s) === RUN TestReaderPatternFarFirstReadIsRandom --- PASS: TestReaderPatternFarFirstReadIsRandom (0.00s) === RUN TestReaderPatternToleranceAbsorbsReorder --- PASS: TestReaderPatternToleranceAbsorbsReorder (0.00s) === RUN TestReaderPatternRandomRecoveryNeedsHysteresis --- PASS: TestReaderPatternRandomRecoveryNeedsHysteresis (0.00s) === RUN TestReaderPatternFrontierNeverRegresses --- PASS: TestReaderPatternFrontierNeverRegresses (0.00s) === RUN TestReaderPatternToleranceBoundary --- PASS: TestReaderPatternToleranceBoundary (0.00s) === RUN TestReaderPatternRecoversFromRandom --- PASS: TestReaderPatternRecoversFromRandom (0.00s) === RUN TestFindMountedRemoteMapping === RUN TestFindMountedRemoteMapping//buckets/b === RUN TestFindMountedRemoteMapping//buckets/b/other === RUN TestFindMountedRemoteMapping//buckets/b/sub === RUN TestFindMountedRemoteMapping//buckets/b/sub/deep === RUN TestFindMountedRemoteMapping//buckets/foo/x === RUN TestFindMountedRemoteMapping//buckets/foobar === RUN TestFindMountedRemoteMapping//buckets/other --- PASS: TestFindMountedRemoteMapping (0.02s) --- PASS: TestFindMountedRemoteMapping//buckets/b (0.00s) --- PASS: TestFindMountedRemoteMapping//buckets/b/other (0.00s) --- PASS: TestFindMountedRemoteMapping//buckets/b/sub (0.00s) --- PASS: TestFindMountedRemoteMapping//buckets/b/sub/deep (0.00s) --- PASS: TestFindMountedRemoteMapping//buckets/foo/x (0.00s) --- PASS: TestFindMountedRemoteMapping//buckets/foobar (0.00s) --- PASS: TestFindMountedRemoteMapping//buckets/other (0.00s) === RUN TestFilerRemoteStorage_FindRemoteStorageClient --- PASS: TestFilerRemoteStorage_FindRemoteStorageClient (0.00s) === RUN TestFilerRemoteStorage_FindMountDirectory_LongestPrefixWins --- PASS: TestFilerRemoteStorage_FindMountDirectory_LongestPrefixWins (0.00s) === RUN TestS3Conf --- PASS: TestS3Conf (0.04s) === RUN TestCheckDuplicateAccessKey --- PASS: TestCheckDuplicateAccessKey (0.00s) === RUN TestPrefetchInOrderDelivery --- PASS: TestPrefetchInOrderDelivery (0.10s) === RUN TestPrefetchSingleChunk --- PASS: TestPrefetchSingleChunk (0.02s) === RUN TestPrefetchFallbackToSequential --- PASS: TestPrefetchFallbackToSequential (0.03s) === RUN TestPrefetchContextCancellation I0824 22:18:30.686656 filechunk_manifest.go:185 read http://127.0.0.1:37411/1,0 failed, err: Get "http://127.0.0.1:37411/1,0?readDeleted=true": context deadline exceeded I0824 22:18:30.686805 filechunk_manifest.go:185 read http://127.0.0.1:37411/1,3 failed, err: Get "http://127.0.0.1:37411/1,3?readDeleted=true": context deadline exceeded I0824 22:18:30.687144 filechunk_manifest.go:185 read http://127.0.0.1:37411/1,1 failed, err: Get "http://127.0.0.1:37411/1,1?readDeleted=true": context deadline exceeded I0824 22:18:30.686978 filechunk_manifest.go:185 read http://127.0.0.1:37411/1,2 failed, err: Get "http://127.0.0.1:37411/1,2?readDeleted=true": context deadline exceeded --- PASS: TestPrefetchContextCancellation (0.12s) === RUN TestPrefetchRangeRequest --- PASS: TestPrefetchRangeRequest (0.02s) === RUN TestPrefetchLargePrefetchCount --- PASS: TestPrefetchLargePrefetchCount (0.03s) === RUN TestPrefetchConcurrentDownloads --- PASS: TestPrefetchConcurrentDownloads (0.31s) PASS ok github.com/seaweedfs/seaweedfs/weed/filer 22.794s === RUN TestGetTxOrDBInternalSystemFolder --- PASS: TestGetTxOrDBInternalSystemFolder (0.00s) === RUN TestGetTxOrDBRealBucket --- PASS: TestGetTxOrDBRealBucket (0.00s) PASS ok github.com/seaweedfs/seaweedfs/weed/filer/abstract_sql 15.441s === RUN TestListDirectoryPrefixedEntriesAqlInjection arangodb_store_test.go:80: arangodb tests are disabled. Start an arangodb server and set RUN_ARANGODB_TESTS=1 to enable, ARANGODB_ADDR defaults to http://127.0.0.1:8529. --- SKIP: TestListDirectoryPrefixedEntriesAqlInjection (0.10s) === RUN TestListDirectoryPrefixedEntriesPrefixNotEvaluated arangodb_store_test.go:105: arangodb tests are disabled. Start an arangodb server and set RUN_ARANGODB_TESTS=1 to enable, ARANGODB_ADDR defaults to http://127.0.0.1:8529. --- SKIP: TestListDirectoryPrefixedEntriesPrefixNotEvaluated (0.00s) === RUN TestListDirectoryPrefixedEntriesQuotedNames arangodb_store_test.go:122: arangodb tests are disabled. Start an arangodb server and set RUN_ARANGODB_TESTS=1 to enable, ARANGODB_ADDR defaults to http://127.0.0.1:8529. --- SKIP: TestListDirectoryPrefixedEntriesQuotedNames (0.00s) === RUN TestDeleteFolderChildrenAqlInjection arangodb_store_test.go:142: arangodb tests are disabled. Start an arangodb server and set RUN_ARANGODB_TESTS=1 to enable, ARANGODB_ADDR defaults to http://127.0.0.1:8529. --- SKIP: TestDeleteFolderChildrenAqlInjection (0.00s) === RUN TestDeleteFolderChildrenRemovesSubtree arangodb_store_test.go:160: arangodb tests are disabled. Start an arangodb server and set RUN_ARANGODB_TESTS=1 to enable, ARANGODB_ADDR defaults to http://127.0.0.1:8529. --- SKIP: TestDeleteFolderChildrenRemovesSubtree (0.00s) === RUN TestListDirectoryEntriesQuotedStartFileName arangodb_store_test.go:178: arangodb tests are disabled. Start an arangodb server and set RUN_ARANGODB_TESTS=1 to enable, ARANGODB_ADDR defaults to http://127.0.0.1:8529. --- SKIP: TestListDirectoryEntriesQuotedStartFileName (0.00s) PASS ok github.com/seaweedfs/seaweedfs/weed/filer/arangodb 14.101s ? github.com/seaweedfs/seaweedfs/weed/filer/cassandra [no test files] ? github.com/seaweedfs/seaweedfs/weed/filer/cassandra2 [no test files] ? github.com/seaweedfs/seaweedfs/weed/filer/elastic/v7 [no test files] === RUN TestCleanupQueue_Add --- PASS: TestCleanupQueue_Add (0.00s) === RUN TestCleanupQueue_Add_OutOfOrder --- PASS: TestCleanupQueue_Add_OutOfOrder (0.00s) === RUN TestCleanupQueue_Add_DuplicateWithOlderTime --- PASS: TestCleanupQueue_Add_DuplicateWithOlderTime (0.00s) === RUN TestCleanupQueue_Remove --- PASS: TestCleanupQueue_Remove (0.00s) === RUN TestCleanupQueue_Pop --- PASS: TestCleanupQueue_Pop (0.00s) === RUN TestCleanupQueue_Peek --- PASS: TestCleanupQueue_Peek (0.00s) === RUN TestCleanupQueue_Contains --- PASS: TestCleanupQueue_Contains (0.00s) === RUN TestCleanupQueue_ShouldProcess_MaxSize --- PASS: TestCleanupQueue_ShouldProcess_MaxSize (0.00s) === RUN TestCleanupQueue_ShouldProcess_MaxAge --- PASS: TestCleanupQueue_ShouldProcess_MaxAge (0.00s) === RUN TestCleanupQueue_Clear --- PASS: TestCleanupQueue_Clear (0.00s) === RUN TestCleanupQueue_OldestAge --- PASS: TestCleanupQueue_OldestAge (0.00s) === RUN TestCleanupQueue_TimeOrder --- PASS: TestCleanupQueue_TimeOrder (0.00s) === RUN TestCleanupQueue_DuplicateWithNewerTime --- PASS: TestCleanupQueue_DuplicateWithNewerTime (0.00s) === RUN TestCleanupQueue_Concurrent --- PASS: TestCleanupQueue_Concurrent (0.00s) === RUN Test_isUnderPath === RUN Test_isUnderPath/child_under_parent === RUN Test_isUnderPath/child_is_parent === RUN Test_isUnderPath/child_not_under_parent === RUN Test_isUnderPath/empty_parent === RUN Test_isUnderPath/root_parent === RUN Test_isUnderPath/parent_with_trailing_slash === RUN Test_isUnderPath/similar_prefix_but_not_under === RUN Test_isUnderPath/deeply_nested --- PASS: Test_isUnderPath (0.03s) --- PASS: Test_isUnderPath/child_under_parent (0.00s) --- PASS: Test_isUnderPath/child_is_parent (0.00s) --- PASS: Test_isUnderPath/child_not_under_parent (0.00s) --- PASS: Test_isUnderPath/empty_parent (0.00s) --- PASS: Test_isUnderPath/root_parent (0.00s) --- PASS: Test_isUnderPath/parent_with_trailing_slash (0.00s) --- PASS: Test_isUnderPath/similar_prefix_but_not_under (0.00s) --- PASS: Test_isUnderPath/deeply_nested (0.00s) === RUN Test_isUnderBucketPath === RUN Test_isUnderBucketPath/bucket_path_itself === RUN Test_isUnderBucketPath/bucket_directory === RUN Test_isUnderBucketPath/folder_in_bucket === RUN Test_isUnderBucketPath/nested_folder === RUN Test_isUnderBucketPath/outside_buckets === RUN Test_isUnderBucketPath/empty_bucket_path --- PASS: Test_isUnderBucketPath (0.01s) --- PASS: Test_isUnderBucketPath/bucket_path_itself (0.00s) --- PASS: Test_isUnderBucketPath/bucket_directory (0.00s) --- PASS: Test_isUnderBucketPath/folder_in_bucket (0.00s) --- PASS: Test_isUnderBucketPath/nested_folder (0.00s) --- PASS: Test_isUnderBucketPath/outside_buckets (0.00s) --- PASS: Test_isUnderBucketPath/empty_bucket_path (0.00s) === RUN Test_isMultipartUploadsPath === RUN Test_isMultipartUploadsPath/staging_root === RUN Test_isMultipartUploadsPath/upload_marker_dir === RUN Test_isMultipartUploadsPath/nested_under_staging === RUN Test_isMultipartUploadsPath/normal_folder === RUN Test_isMultipartUploadsPath/registry_own_uploads_dir === RUN Test_isMultipartUploadsPath/bucket_itself === RUN Test_isMultipartUploadsPath/outside_buckets === RUN Test_isMultipartUploadsPath/empty_bucket_path --- PASS: Test_isMultipartUploadsPath (0.00s) --- PASS: Test_isMultipartUploadsPath/staging_root (0.00s) --- PASS: Test_isMultipartUploadsPath/upload_marker_dir (0.00s) --- PASS: Test_isMultipartUploadsPath/nested_under_staging (0.00s) --- PASS: Test_isMultipartUploadsPath/normal_folder (0.00s) --- PASS: Test_isMultipartUploadsPath/registry_own_uploads_dir (0.00s) --- PASS: Test_isMultipartUploadsPath/bucket_itself (0.00s) --- PASS: Test_isMultipartUploadsPath/outside_buckets (0.00s) --- PASS: Test_isMultipartUploadsPath/empty_bucket_path (0.00s) === RUN TestEmptyFolderCleaner_OnDeleteEvent_skipsMultipartUploads --- PASS: TestEmptyFolderCleaner_OnDeleteEvent_skipsMultipartUploads (0.03s) === RUN TestEmptyFolderCleaner_executeCleanup_skipsMultipartUploads I0824 22:18:23.951066 empty_folder_cleaner.go:543 EmptyFolderCleaner: deleting empty folder /buckets/mybucket/folder (triggered by file.txt) --- PASS: TestEmptyFolderCleaner_executeCleanup_skipsMultipartUploads (0.01s) === RUN TestEmptyFolderCleaner_restoreFoldersWrittenDuringDelete === RUN TestEmptyFolderCleaner_restoreFoldersWrittenDuringDelete/a_create_event_for_the_deleted_folder_puts_it_back I0824 22:18:23.966402 empty_folder_cleaner.go:543 EmptyFolderCleaner: deleting empty folder /buckets/mybucket/folder (triggered by file.txt) === RUN TestEmptyFolderCleaner_restoreFoldersWrittenDuringDelete/a_create_event_elsewhere_is_ignored I0824 22:18:23.966787 empty_folder_cleaner.go:543 EmptyFolderCleaner: deleting empty folder /buckets/mybucket/folder (triggered by file.txt) === RUN TestEmptyFolderCleaner_restoreFoldersWrittenDuringDelete/an_entry_that_went_away_again_is_not_restored I0824 22:18:23.967137 empty_folder_cleaner.go:543 EmptyFolderCleaner: deleting empty folder /buckets/mybucket/folder (triggered by file.txt) === RUN TestEmptyFolderCleaner_restoreFoldersWrittenDuringDelete/observation_expires I0824 22:18:23.967499 empty_folder_cleaner.go:543 EmptyFolderCleaner: deleting empty folder /buckets/mybucket/folder (triggered by file.txt) === RUN TestEmptyFolderCleaner_restoreFoldersWrittenDuringDelete/a_failed_restore_is_retried I0824 22:18:23.967767 empty_folder_cleaner.go:543 EmptyFolderCleaner: deleting empty folder /buckets/mybucket/folder (triggered by file.txt) === RUN TestEmptyFolderCleaner_restoreFoldersWrittenDuringDelete/a_folder_whose_restore_keeps_failing_stops_being_observed I0824 22:18:23.968435 empty_folder_cleaner.go:543 EmptyFolderCleaner: deleting empty folder /buckets/mybucket/folder (triggered by file.txt) === RUN TestEmptyFolderCleaner_restoreFoldersWrittenDuringDelete/an_ancestor_taken_by_the_cascade_is_restored_with_its_own_attributes I0824 22:18:23.969125 empty_folder_cleaner.go:543 EmptyFolderCleaner: deleting empty folder /buckets/mybucket/data/abc (triggered by file.txt) I0824 22:18:23.969289 empty_folder_cleaner.go:543 EmptyFolderCleaner: deleting empty folder /buckets/mybucket/data (triggered by file.txt) I0824 22:18:23.969368 empty_folder_cleaner.go:543 EmptyFolderCleaner: deleting empty folder /buckets/mybucket/data (triggered by abc) === RUN TestEmptyFolderCleaner_restoreFoldersWrittenDuringDelete/a_delete_that_reports_failure_is_still_observed I0824 22:18:23.969738 empty_folder_cleaner.go:543 EmptyFolderCleaner: deleting empty folder /buckets/mybucket/folder (triggered by file.txt) --- PASS: TestEmptyFolderCleaner_restoreFoldersWrittenDuringDelete (0.00s) --- PASS: TestEmptyFolderCleaner_restoreFoldersWrittenDuringDelete/a_create_event_for_the_deleted_folder_puts_it_back (0.00s) --- PASS: TestEmptyFolderCleaner_restoreFoldersWrittenDuringDelete/a_create_event_elsewhere_is_ignored (0.00s) --- PASS: TestEmptyFolderCleaner_restoreFoldersWrittenDuringDelete/an_entry_that_went_away_again_is_not_restored (0.00s) --- PASS: TestEmptyFolderCleaner_restoreFoldersWrittenDuringDelete/observation_expires (0.00s) --- PASS: TestEmptyFolderCleaner_restoreFoldersWrittenDuringDelete/a_failed_restore_is_retried (0.00s) --- PASS: TestEmptyFolderCleaner_restoreFoldersWrittenDuringDelete/a_folder_whose_restore_keeps_failing_stops_being_observed (0.00s) --- PASS: TestEmptyFolderCleaner_restoreFoldersWrittenDuringDelete/an_ancestor_taken_by_the_cascade_is_restored_with_its_own_attributes (0.00s) --- PASS: TestEmptyFolderCleaner_restoreFoldersWrittenDuringDelete/a_delete_that_reports_failure_is_still_observed (0.00s) === RUN Test_autoRemoveEmptyFoldersEnabled === RUN Test_autoRemoveEmptyFoldersEnabled/no_attrs_defaults_enabled === RUN Test_autoRemoveEmptyFoldersEnabled/missing_key_defaults_enabled === RUN Test_autoRemoveEmptyFoldersEnabled/allow-empty_disables_cleanup === RUN Test_autoRemoveEmptyFoldersEnabled/explicit_false_keeps_cleanup_enabled --- PASS: Test_autoRemoveEmptyFoldersEnabled (0.00s) --- PASS: Test_autoRemoveEmptyFoldersEnabled/no_attrs_defaults_enabled (0.00s) --- PASS: Test_autoRemoveEmptyFoldersEnabled/missing_key_defaults_enabled (0.00s) --- PASS: Test_autoRemoveEmptyFoldersEnabled/allow-empty_disables_cleanup (0.00s) --- PASS: Test_autoRemoveEmptyFoldersEnabled/explicit_false_keeps_cleanup_enabled (0.00s) === RUN TestEmptyFolderCleaner_ownsFolder --- PASS: TestEmptyFolderCleaner_ownsFolder (0.00s) === RUN TestEmptyFolderCleaner_ownsFolder_singleServer --- PASS: TestEmptyFolderCleaner_ownsFolder_singleServer (0.00s) === RUN TestEmptyFolderCleaner_ownsFolder_emptyRing --- PASS: TestEmptyFolderCleaner_ownsFolder_emptyRing (0.00s) === RUN TestEmptyFolderCleaner_OnCreateEvent_cancelsCleanup --- PASS: TestEmptyFolderCleaner_OnCreateEvent_cancelsCleanup (0.00s) === RUN TestEmptyFolderCleaner_OnDeleteEvent_deduplication --- PASS: TestEmptyFolderCleaner_OnDeleteEvent_deduplication (0.00s) === RUN TestEmptyFolderCleaner_OnDeleteEvent_multipleFolders --- PASS: TestEmptyFolderCleaner_OnDeleteEvent_multipleFolders (0.00s) === RUN TestEmptyFolderCleaner_OnDeleteEvent_notOwner --- PASS: TestEmptyFolderCleaner_OnDeleteEvent_notOwner (0.00s) === RUN TestEmptyFolderCleaner_OnDeleteEvent_disabled --- PASS: TestEmptyFolderCleaner_OnDeleteEvent_disabled (0.00s) === RUN TestEmptyFolderCleaner_OnDeleteEvent_directoryDeletion --- PASS: TestEmptyFolderCleaner_OnDeleteEvent_directoryDeletion (0.00s) === RUN TestEmptyFolderCleaner_cachedCounts --- PASS: TestEmptyFolderCleaner_cachedCounts (0.00s) === RUN TestEmptyFolderCleaner_Stop --- PASS: TestEmptyFolderCleaner_Stop (0.00s) === RUN TestEmptyFolderCleaner_cacheEviction --- PASS: TestEmptyFolderCleaner_cacheEviction (0.00s) === RUN TestEmptyFolderCleaner_cacheEviction_skipsEntriesInQueue --- PASS: TestEmptyFolderCleaner_cacheEviction_skipsEntriesInQueue (0.00s) === RUN TestEmptyFolderCleaner_queueFIFOOrder --- PASS: TestEmptyFolderCleaner_queueFIFOOrder (0.00s) === RUN TestEmptyFolderCleaner_processCleanupQueue_onlyProcessesAgedItems I0824 22:18:23.981582 empty_folder_cleaner.go:543 EmptyFolderCleaner: deleting empty folder /buckets/test/folder1 (triggered by i1) I0824 22:18:23.981648 empty_folder_cleaner.go:543 EmptyFolderCleaner: deleting empty folder /buckets/test/folder2 (triggered by i2) --- PASS: TestEmptyFolderCleaner_processCleanupQueue_onlyProcessesAgedItems (0.00s) === RUN TestEmptyFolderCleaner_executeCleanup_bucketPolicyDisabledSkips --- PASS: TestEmptyFolderCleaner_executeCleanup_bucketPolicyDisabledSkips (0.00s) === RUN TestEmptyFolderCleaner_executeCleanup_directoryMarker === RUN TestEmptyFolderCleaner_executeCleanup_directoryMarker/skips_explicit_directory_marker === RUN TestEmptyFolderCleaner_executeCleanup_directoryMarker/deletes_implicit_empty_folder I0824 22:18:23.982735 empty_folder_cleaner.go:543 EmptyFolderCleaner: deleting empty folder /buckets/test/folder (triggered by triggered_item) --- PASS: TestEmptyFolderCleaner_executeCleanup_directoryMarker (0.00s) --- PASS: TestEmptyFolderCleaner_executeCleanup_directoryMarker/skips_explicit_directory_marker (0.00s) --- PASS: TestEmptyFolderCleaner_executeCleanup_directoryMarker/deletes_implicit_empty_folder (0.00s) === RUN TestEmptyFolderCleaner_restoreIfWrittenTo === RUN TestEmptyFolderCleaner_restoreIfWrittenTo/an_entry_that_landed_during_the_delete_is_restored_without_waiting_for_its_event I0824 22:18:23.985239 empty_folder_cleaner.go:543 EmptyFolderCleaner: deleting empty folder /buckets/mybucket/a/b (triggered by obj) === RUN TestEmptyFolderCleaner_restoreIfWrittenTo/a_folder_still_empty_after_the_delete_is_not_restored_and_the_parent_is_cascaded I0824 22:18:23.985495 empty_folder_cleaner.go:543 EmptyFolderCleaner: deleting empty folder /buckets/mybucket/a/b (triggered by obj) I0824 22:18:23.985592 empty_folder_cleaner.go:543 EmptyFolderCleaner: deleting empty folder /buckets/mybucket/a (triggered by obj) === RUN TestEmptyFolderCleaner_restoreIfWrittenTo/a_restore_that_fails_is_left_marked_for_the_deferred_pass I0824 22:18:23.985848 empty_folder_cleaner.go:543 EmptyFolderCleaner: deleting empty folder /buckets/mybucket/a/b (triggered by obj) I0824 22:18:23.985960 empty_folder_cleaner.go:543 EmptyFolderCleaner: deleting empty folder /buckets/mybucket/a (triggered by obj) === RUN TestEmptyFolderCleaner_restoreIfWrittenTo/a_re-check_that_fails_leaves_the_folder_to_the_event_path I0824 22:18:23.986128 empty_folder_cleaner.go:543 EmptyFolderCleaner: deleting empty folder /buckets/mybucket/a/b (triggered by obj) --- PASS: TestEmptyFolderCleaner_restoreIfWrittenTo (0.00s) --- PASS: TestEmptyFolderCleaner_restoreIfWrittenTo/an_entry_that_landed_during_the_delete_is_restored_without_waiting_for_its_event (0.00s) --- PASS: TestEmptyFolderCleaner_restoreIfWrittenTo/a_folder_still_empty_after_the_delete_is_not_restored_and_the_parent_is_cascaded (0.00s) --- PASS: TestEmptyFolderCleaner_restoreIfWrittenTo/a_restore_that_fails_is_left_marked_for_the_deferred_pass (0.00s) --- PASS: TestEmptyFolderCleaner_restoreIfWrittenTo/a_re-check_that_fails_leaves_the_folder_to_the_event_path (0.00s) PASS ok github.com/seaweedfs/seaweedfs/weed/filer/empty_folder_cleanup 15.447s === RUN TestStore --- PASS: TestStore (0.00s) PASS ok github.com/seaweedfs/seaweedfs/weed/filer/etcd 14.510s ? github.com/seaweedfs/seaweedfs/weed/filer/foundationdb [no test files] ? github.com/seaweedfs/seaweedfs/weed/filer/hbase [no test files] === RUN TestCreateAndFind I0824 22:18:25.202515 leveldb_store.go:48 filer store dir: /tmp/TestCreateAndFind3995604822/001 I0824 22:18:25.218058 file_util.go:29 Folder /tmp/TestCreateAndFind3995604822/001 Permission: -rwxr-xr-x I0824 22:18:25.301644 filer.go:201 create filer.store.id to 174650223 --- PASS: TestCreateAndFind (0.66s) === RUN TestEmptyRoot I0824 22:18:25.564292 leveldb_store.go:48 filer store dir: /tmp/TestEmptyRoot717550661/001 I0824 22:18:25.565245 file_util.go:29 Folder /tmp/TestEmptyRoot717550661/001 Permission: -rwxr-xr-x I0824 22:18:25.579916 filer.go:201 create filer.store.id to 950163352 --- PASS: TestEmptyRoot (0.02s) === RUN TestCreateEntryRecordsObjectSize I0824 22:18:25.691146 leveldb_store.go:48 filer store dir: /tmp/TestCreateEntryRecordsObjectSize2596018923/001 I0824 22:18:25.691911 file_util.go:29 Folder /tmp/TestCreateEntryRecordsObjectSize2596018923/001 Permission: -rwxr-xr-x I0824 22:18:25.721440 filer.go:201 create filer.store.id to 425687266 I0824 22:18:25.728419 filer_deletion.go:515 retry processor shutting down, 0 items remaining in queue I0824 22:18:25.728765 filer_deletion.go:306 deletion processor shutting down --- PASS: TestCreateEntryRecordsObjectSize (0.16s) PASS ok github.com/seaweedfs/seaweedfs/weed/filer/leveldb 17.157s === RUN TestNonRecursiveFolderDeleteKeepsRacingChild I0824 22:18:26.888522 leveldb2_store.go:43 filer store leveldb2 dir: /tmp/TestNonRecursiveFolderDeleteKeepsRacingChild3352747466/001 I0824 22:18:26.915588 file_util.go:29 Folder /tmp/TestNonRecursiveFolderDeleteKeepsRacingChild3352747466/001 Permission: -rwxr-xr-x I0824 22:18:26.968124 filer.go:201 create filer.store.id to -1408226418 --- PASS: TestNonRecursiveFolderDeleteKeepsRacingChild (0.37s) === RUN TestEnsureDirectoryEntryRestoresRacingParent I0824 22:18:27.118865 leveldb2_store.go:43 filer store leveldb2 dir: /tmp/TestEnsureDirectoryEntryRestoresRacingParent3944500624/001 I0824 22:18:27.119603 file_util.go:29 Folder /tmp/TestEnsureDirectoryEntryRestoresRacingParent3944500624/001 Permission: -rwxr-xr-x I0824 22:18:27.183190 filer.go:201 create filer.store.id to -342656568 --- PASS: TestEnsureDirectoryEntryRestoresRacingParent (0.08s) === RUN TestEnsureDirectoryEntryRestoresExactAttributes I0824 22:18:27.225080 leveldb2_store.go:43 filer store leveldb2 dir: /tmp/TestEnsureDirectoryEntryRestoresExactAttributes4026578933/001 I0824 22:18:27.225216 file_util.go:29 Folder /tmp/TestEnsureDirectoryEntryRestoresExactAttributes4026578933/001 Permission: -rwxr-xr-x I0824 22:18:27.243682 filer.go:201 create filer.store.id to -1396854304 --- PASS: TestEnsureDirectoryEntryRestoresExactAttributes (0.06s) === RUN TestCreateEntryCreatesParentAfterTheEntry I0824 22:18:27.290062 leveldb2_store.go:43 filer store leveldb2 dir: /tmp/TestCreateEntryCreatesParentAfterTheEntry2840506207/001 I0824 22:18:27.290565 file_util.go:29 Folder /tmp/TestCreateEntryCreatesParentAfterTheEntry2840506207/001 Permission: -rwxr-xr-x I0824 22:18:27.338296 filer.go:201 create filer.store.id to 1669589324 --- PASS: TestCreateEntryCreatesParentAfterTheEntry (0.10s) === RUN TestCreateEntryKeepsTheEntryWhenTheParentCannotBeCreated I0824 22:18:27.362268 leveldb2_store.go:43 filer store leveldb2 dir: /tmp/TestCreateEntryKeepsTheEntryWhenTheParentCannotBeCreated3573441176/001 I0824 22:18:27.363313 file_util.go:29 Folder /tmp/TestCreateEntryKeepsTheEntryWhenTheParentCannotBeCreated3573441176/001 Permission: -rwxr-xr-x I0824 22:18:27.408730 filer.go:201 create filer.store.id to 73933172 E0824 22:18:27.412486 filer.go:287 create parent directories of /buckets/Not_A_Valid_Bucket/obj: invalid bucket name Not_A_Valid_Bucket: bucket name can only contain lower case characters, numbers, dots, and hyphens --- PASS: TestCreateEntryKeepsTheEntryWhenTheParentCannotBeCreated (0.08s) === RUN TestCreateEntryDoesNotAnnounceAnEntryWhoseParentFailed I0824 22:18:27.436398 leveldb2_store.go:43 filer store leveldb2 dir: /tmp/TestCreateEntryDoesNotAnnounceAnEntryWhoseParentFailed2723511612/001 I0824 22:18:27.438663 file_util.go:29 Folder /tmp/TestCreateEntryDoesNotAnnounceAnEntryWhoseParentFailed2723511612/001 Permission: -rwxr-xr-x I0824 22:18:27.477429 filer.go:201 create filer.store.id to -1732871665 E0824 22:18:27.479865 filer.go:287 create parent directories of /buckets/Not_A_Valid_Bucket/obj: invalid bucket name Not_A_Valid_Bucket: bucket name can only contain lower case characters, numbers, dots, and hyphens --- PASS: TestCreateEntryDoesNotAnnounceAnEntryWhoseParentFailed (0.05s) === RUN TestEnsureDirectoryEntryNarrowsAWiderRestore I0824 22:18:27.485762 leveldb2_store.go:43 filer store leveldb2 dir: /tmp/TestEnsureDirectoryEntryNarrowsAWiderRestore3751413685/001 I0824 22:18:27.485959 file_util.go:29 Folder /tmp/TestEnsureDirectoryEntryNarrowsAWiderRestore3751413685/001 Permission: -rwxr-xr-x I0824 22:18:28.285054 filer.go:201 create filer.store.id to 1186343720 --- PASS: TestEnsureDirectoryEntryNarrowsAWiderRestore (0.85s) === RUN TestEnsureDirectoryEntryKeepsATightenedDirectory I0824 22:18:28.359001 leveldb2_store.go:43 filer store leveldb2 dir: /tmp/TestEnsureDirectoryEntryKeepsATightenedDirectory1053415724/001 I0824 22:18:28.359582 file_util.go:29 Folder /tmp/TestEnsureDirectoryEntryKeepsATightenedDirectory1053415724/001 Permission: -rwxr-xr-x I0824 22:18:28.435576 filer.go:201 create filer.store.id to 1463766528 --- PASS: TestEnsureDirectoryEntryKeepsATightenedDirectory (0.10s) === RUN TestEnsureDirectoryEntryIntersectsMixedModes I0824 22:18:28.450299 leveldb2_store.go:43 filer store leveldb2 dir: /tmp/TestEnsureDirectoryEntryIntersectsMixedModes203544742/001 I0824 22:18:28.450489 file_util.go:29 Folder /tmp/TestEnsureDirectoryEntryIntersectsMixedModes203544742/001 Permission: -rwxr-xr-x I0824 22:18:28.496258 filer.go:201 create filer.store.id to -552451201 --- PASS: TestEnsureDirectoryEntryIntersectsMixedModes (0.07s) === RUN TestCreateAndFind I0824 22:18:28.524594 leveldb2_store.go:43 filer store leveldb2 dir: /tmp/TestCreateAndFind1505857025/001 I0824 22:18:28.524761 file_util.go:29 Folder /tmp/TestCreateAndFind1505857025/001 Permission: -rwxr-xr-x I0824 22:18:28.533845 filer.go:201 create filer.store.id to -641036618 --- PASS: TestCreateAndFind (0.07s) === RUN TestEmptyRoot I0824 22:18:28.598368 leveldb2_store.go:43 filer store leveldb2 dir: /tmp/TestEmptyRoot1288807685/001 I0824 22:18:28.598615 file_util.go:29 Folder /tmp/TestEmptyRoot1288807685/001 Permission: -rwxr-xr-x I0824 22:18:28.611060 filer.go:201 create filer.store.id to -1516279925 --- PASS: TestEmptyRoot (0.02s) PASS ok github.com/seaweedfs/seaweedfs/weed/filer/leveldb2 20.059s === RUN TestCreateAndFind I0824 22:18:26.279135 leveldb3_store.go:50 filer store leveldb3 dir: /tmp/TestCreateAndFind1056664378/001 I0824 22:18:26.293789 file_util.go:29 Folder /tmp/TestCreateAndFind1056664378/001 Permission: -rwxr-xr-x I0824 22:18:26.370135 filer.go:201 create filer.store.id to -1651959825 --- PASS: TestCreateAndFind (0.52s) === RUN TestEmptyRoot I0824 22:18:26.606956 leveldb3_store.go:50 filer store leveldb3 dir: /tmp/TestEmptyRoot3772675656/001 I0824 22:18:26.607824 file_util.go:29 Folder /tmp/TestEmptyRoot3772675656/001 Permission: -rwxr-xr-x I0824 22:18:26.621807 filer.go:201 create filer.store.id to -1711941756 --- PASS: TestEmptyRoot (0.04s) PASS ok github.com/seaweedfs/seaweedfs/weed/filer/leveldb3 18.030s ? github.com/seaweedfs/seaweedfs/weed/filer/mongodb [no test files] === RUN TestDefaultUpsertQueryUsesOnDuplicateKey --- PASS: TestDefaultUpsertQueryUsesOnDuplicateKey (0.00s) === RUN TestDefaultCreateTableQueryRendersValidSql --- PASS: TestDefaultCreateTableQueryRendersValidSql (0.00s) === RUN TestEmptyUpsertTemplateFallsBackToPlainInsert --- PASS: TestEmptyUpsertTemplateFallsBackToPlainInsert (0.00s) === RUN TestListSqlDefaultOrderingFollowsColumn --- PASS: TestListSqlDefaultOrderingFollowsColumn (0.00s) === RUN TestListSqlBinaryOrderingOnNonBinaryColumn --- PASS: TestListSqlBinaryOrderingOnNonBinaryColumn (0.00s) === RUN TestIsBinaryCollation --- PASS: TestIsBinaryCollation (0.00s) PASS ok github.com/seaweedfs/seaweedfs/weed/filer/mysql 14.046s ? github.com/seaweedfs/seaweedfs/weed/filer/mysql2 [no test files] === RUN TestManagerGrantAndConflict --- PASS: TestManagerGrantAndConflict (0.00s) === RUN TestManagerUnlockCleansEmptyKeyAndIndex --- PASS: TestManagerUnlockCleansEmptyKeyAndIndex (0.00s) === RUN TestManagerPartialUnlockKeepsIndex --- PASS: TestManagerPartialUnlockKeepsIndex (0.00s) === RUN TestManagerGetLk --- PASS: TestManagerGetLk (0.00s) === RUN TestManagerReleasePosixOwnerKeepsFlockAndIndex --- PASS: TestManagerReleasePosixOwnerKeepsFlockAndIndex (0.00s) === RUN TestManagerReleaseSessionReapsAcrossKeys --- PASS: TestManagerReleaseSessionReapsAcrossKeys (0.00s) === RUN TestManagerReapsOnlyStaleLeasedSessions --- PASS: TestManagerReapsOnlyStaleLeasedSessions (0.00s) === RUN TestManagerConcurrentFlockMutualExclusion --- PASS: TestManagerConcurrentFlockMutualExclusion (0.37s) === RUN TestNonOverlappingLocksFromDifferentOwners --- PASS: TestNonOverlappingLocksFromDifferentOwners (0.00s) === RUN TestOverlappingReadLocksFromDifferentOwners --- PASS: TestOverlappingReadLocksFromDifferentOwners (0.00s) === RUN TestOverlappingWriteReadConflict --- PASS: TestOverlappingWriteReadConflict (0.00s) === RUN TestOverlappingWriteWriteConflict --- PASS: TestOverlappingWriteWriteConflict (0.00s) === RUN TestSameOwnerUpgradeReadToWrite --- PASS: TestSameOwnerUpgradeReadToWrite (0.00s) === RUN TestSameOwnerDowngradeWriteToRead --- PASS: TestSameOwnerDowngradeWriteToRead (0.00s) === RUN TestLockCoalescing --- PASS: TestLockCoalescing (0.00s) === RUN TestLockSplitting --- PASS: TestLockSplitting (0.00s) === RUN TestConflictReportsHolder --- PASS: TestConflictReportsHolder (0.00s) === RUN TestConflictNoneForSharedReads --- PASS: TestConflictNoneForSharedReads (0.00s) === RUN TestConflictSameOwnerNone --- PASS: TestConflictSameOwnerNone (0.00s) === RUN TestReleaseOwner --- PASS: TestReleaseOwner (0.00s) === RUN TestFlockAndFcntlDoNotConflict --- PASS: TestFlockAndFcntlDoNotConflict (0.00s) === RUN TestReleasePosixOwnerKeepsFlock --- PASS: TestReleasePosixOwnerKeepsFlock (0.00s) === RUN TestReleaseFlockOwnerKeepsPosix --- PASS: TestReleaseFlockOwnerKeepsPosix (0.00s) === RUN TestHasPosixIgnoresMissingOwnerAndFlock --- PASS: TestHasPosixIgnoresMissingOwnerAndFlock (0.00s) === RUN TestWholeFileLock --- PASS: TestWholeFileLock (0.00s) === RUN TestReleaseNoExistingLocks --- PASS: TestReleaseNoExistingLocks (0.00s) === RUN TestSameOwnerReplaceDifferentType --- PASS: TestSameOwnerReplaceDifferentType (0.00s) === RUN TestNonAdjacentRangesNotCoalesced --- PASS: TestNonAdjacentRangesNotCoalesced (0.00s) === RUN TestAdjacencyNoOverflowAtMaxUint64 --- PASS: TestAdjacencyNoOverflowAtMaxUint64 (0.00s) === RUN TestTwoSessionsSameOwnerDoNotAlias --- PASS: TestTwoSessionsSameOwnerDoNotAlias (0.00s) === RUN TestReleaseSessionReapsOnlyThatSession --- PASS: TestReleaseSessionReapsOnlyThatSession (0.00s) === RUN TestEmptyAfterReleasingAll --- PASS: TestEmptyAfterReleasingAll (0.00s) === RUN TestReassertRebuildsOnFreshOwner --- PASS: TestReassertRebuildsOnFreshOwner (0.00s) === RUN TestReassertIdempotent --- PASS: TestReassertIdempotent (0.00s) === RUN TestReassertReportsConflict --- PASS: TestReassertReportsConflict (0.00s) === RUN TestReassertRenewsLease --- PASS: TestReassertRenewsLease (0.00s) PASS ok github.com/seaweedfs/seaweedfs/weed/filer/posixlock 0.741s === RUN TestConfigureListOrderingSkipsCockroachDB --- PASS: TestConfigureListOrderingSkipsCockroachDB (0.10s) === RUN TestConfigureListOrderingForcesBinaryOnLocalePostgres W0824 22:18:24.503889 postgres_collation.go:29 postgres: filemeta.name collation "en_US.UTF-8" is not byte-ordered, so S3 list order is not byte-lexicographic and clients that merge sorted listings may report spurious diffs. Falling back to a slower COLLATE "C" sort; declare the name column COLLATE "C" (or use a C/C.UTF-8 database collation) for correct, indexed ordering. --- PASS: TestConfigureListOrderingForcesBinaryOnLocalePostgres (0.01s) === RUN TestDefaultUpsertQueryIsConflictSafe --- PASS: TestDefaultUpsertQueryIsConflictSafe (0.00s) === RUN TestDefaultCreateTableQueryRendersValidSql --- PASS: TestDefaultCreateTableQueryRendersValidSql (0.00s) === RUN TestEmptyUpsertTemplateFallsBackToPlainInsert --- PASS: TestEmptyUpsertTemplateFallsBackToPlainInsert (0.00s) === RUN TestListSqlDefaultOrderingFollowsColumn --- PASS: TestListSqlDefaultOrderingFollowsColumn (0.00s) === RUN TestListSqlBinaryOrderingOnNonBinaryColumn --- PASS: TestListSqlBinaryOrderingOnNonBinaryColumn (0.00s) === RUN TestIsByteOrderedCollation --- PASS: TestIsByteOrderedCollation (0.00s) PASS ok github.com/seaweedfs/seaweedfs/weed/filer/postgres 15.874s ? github.com/seaweedfs/seaweedfs/weed/filer/postgres2 [no test files] === RUN TestListDirectoryEntriesRemovesOrphanedIndexMembers universal_redis_store_test.go:93: redis tests are disabled. Start a redis-server and set RUN_REDIS_TESTS=1 to enable, REDIS_ADDR defaults to 127.0.0.1:6379. --- SKIP: TestListDirectoryEntriesRemovesOrphanedIndexMembers (0.07s) === RUN TestRemoveOrphanedDirectoryListMemberKeepsRecreatedEntry universal_redis_store_test.go:112: redis tests are disabled. Start a redis-server and set RUN_REDIS_TESTS=1 to enable, REDIS_ADDR defaults to 127.0.0.1:6379. --- SKIP: TestRemoveOrphanedDirectoryListMemberKeepsRecreatedEntry (0.04s) === RUN TestRemoveOrphanedDirectoryListMemberKeepsDirectoryWithChildren universal_redis_store_test.go:128: redis tests are disabled. Start a redis-server and set RUN_REDIS_TESTS=1 to enable, REDIS_ADDR defaults to 127.0.0.1:6379. --- SKIP: TestRemoveOrphanedDirectoryListMemberKeepsDirectoryWithChildren (0.00s) === RUN TestDeleteFolderChildrenRemovesTheListing universal_redis_store_test.go:158: redis tests are disabled. Start a redis-server and set RUN_REDIS_TESTS=1 to enable, REDIS_ADDR defaults to 127.0.0.1:6379. --- SKIP: TestDeleteFolderChildrenRemovesTheListing (0.00s) PASS ok github.com/seaweedfs/seaweedfs/weed/filer/redis 16.951s === RUN TestListDirectoryEntriesRemovesOrphanedIndexMembers === RUN TestListDirectoryEntriesRemovesOrphanedIndexMembers/keyPrefix= universal_redis_store_test.go:93: redis2 tests are disabled. Start a redis-server and set RUN_REDIS_TESTS=1 to enable, REDIS_ADDR defaults to 127.0.0.1:6379. === RUN TestListDirectoryEntriesRemovesOrphanedIndexMembers/keyPrefix=sw: universal_redis_store_test.go:93: redis2 tests are disabled. Start a redis-server and set RUN_REDIS_TESTS=1 to enable, REDIS_ADDR defaults to 127.0.0.1:6379. --- PASS: TestListDirectoryEntriesRemovesOrphanedIndexMembers (0.03s) --- SKIP: TestListDirectoryEntriesRemovesOrphanedIndexMembers/keyPrefix= (0.03s) --- SKIP: TestListDirectoryEntriesRemovesOrphanedIndexMembers/keyPrefix=sw: (0.00s) === RUN TestRemoveOrphanedDirectoryListMemberKeepsRecreatedEntry === RUN TestRemoveOrphanedDirectoryListMemberKeepsRecreatedEntry/keyPrefix= universal_redis_store_test.go:116: redis2 tests are disabled. Start a redis-server and set RUN_REDIS_TESTS=1 to enable, REDIS_ADDR defaults to 127.0.0.1:6379. === RUN TestRemoveOrphanedDirectoryListMemberKeepsRecreatedEntry/keyPrefix=sw: universal_redis_store_test.go:116: redis2 tests are disabled. Start a redis-server and set RUN_REDIS_TESTS=1 to enable, REDIS_ADDR defaults to 127.0.0.1:6379. --- PASS: TestRemoveOrphanedDirectoryListMemberKeepsRecreatedEntry (0.03s) --- SKIP: TestRemoveOrphanedDirectoryListMemberKeepsRecreatedEntry/keyPrefix= (0.03s) --- SKIP: TestRemoveOrphanedDirectoryListMemberKeepsRecreatedEntry/keyPrefix=sw: (0.00s) === RUN TestRemoveOrphanedDirectoryListMemberKeepsDirectoryWithChildren === RUN TestRemoveOrphanedDirectoryListMemberKeepsDirectoryWithChildren/keyPrefix= universal_redis_store_test.go:137: redis2 tests are disabled. Start a redis-server and set RUN_REDIS_TESTS=1 to enable, REDIS_ADDR defaults to 127.0.0.1:6379. === RUN TestRemoveOrphanedDirectoryListMemberKeepsDirectoryWithChildren/keyPrefix=sw: universal_redis_store_test.go:137: redis2 tests are disabled. Start a redis-server and set RUN_REDIS_TESTS=1 to enable, REDIS_ADDR defaults to 127.0.0.1:6379. --- PASS: TestRemoveOrphanedDirectoryListMemberKeepsDirectoryWithChildren (0.03s) --- SKIP: TestRemoveOrphanedDirectoryListMemberKeepsDirectoryWithChildren/keyPrefix= (0.03s) --- SKIP: TestRemoveOrphanedDirectoryListMemberKeepsDirectoryWithChildren/keyPrefix=sw: (0.00s) === RUN TestRemoveOrphanedDirectoryListMemberSkipsSuperLargeDirectory universal_redis_store_test.go:169: redis2 tests are disabled. Start a redis-server and set RUN_REDIS_TESTS=1 to enable, REDIS_ADDR defaults to 127.0.0.1:6379. --- SKIP: TestRemoveOrphanedDirectoryListMemberSkipsSuperLargeDirectory (0.00s) === RUN TestListDirectoryEntriesRemovesIndexMembersExpiredByRedis universal_redis_store_test.go:193: redis2 tests are disabled. Start a redis-server and set RUN_REDIS_TESTS=1 to enable, REDIS_ADDR defaults to 127.0.0.1:6379. --- SKIP: TestListDirectoryEntriesRemovesIndexMembersExpiredByRedis (0.00s) === RUN TestListDirectoryEntriesDeletesLogicallyExpiredEntries === RUN TestListDirectoryEntriesDeletesLogicallyExpiredEntries/keyPrefix= universal_redis_store_test.go:239: redis2 tests are disabled. Start a redis-server and set RUN_REDIS_TESTS=1 to enable, REDIS_ADDR defaults to 127.0.0.1:6379. === RUN TestListDirectoryEntriesDeletesLogicallyExpiredEntries/keyPrefix=sw: universal_redis_store_test.go:239: redis2 tests are disabled. Start a redis-server and set RUN_REDIS_TESTS=1 to enable, REDIS_ADDR defaults to 127.0.0.1:6379. --- PASS: TestListDirectoryEntriesDeletesLogicallyExpiredEntries (0.00s) --- SKIP: TestListDirectoryEntriesDeletesLogicallyExpiredEntries/keyPrefix= (0.00s) --- SKIP: TestListDirectoryEntriesDeletesLogicallyExpiredEntries/keyPrefix=sw: (0.00s) === RUN TestDeleteExpiredEntryKeepsRecreatedValue universal_redis_store_test.go:260: redis2 tests are disabled. Start a redis-server and set RUN_REDIS_TESTS=1 to enable, REDIS_ADDR defaults to 127.0.0.1:6379. --- SKIP: TestDeleteExpiredEntryKeepsRecreatedValue (0.00s) === RUN TestDeleteIfUnchangedScriptOnlyDeletesSameBytes universal_redis_store_test.go:279: redis2 tests are disabled. Start a redis-server and set RUN_REDIS_TESTS=1 to enable, REDIS_ADDR defaults to 127.0.0.1:6379. --- SKIP: TestDeleteIfUnchangedScriptOnlyDeletesSameBytes (0.00s) === RUN TestDeleteEntryKeepsChildListing universal_redis_store_test.go:304: redis2 tests are disabled. Start a redis-server and set RUN_REDIS_TESTS=1 to enable, REDIS_ADDR defaults to 127.0.0.1:6379. --- SKIP: TestDeleteEntryKeepsChildListing (0.00s) PASS ok github.com/seaweedfs/seaweedfs/weed/filer/redis2 15.027s === RUN TestDeleteEntryKeepsChildListing universal_redis_store_test.go:70: redis3 tests are disabled. Start a redis-server and set RUN_REDIS_TESTS=1 to enable, REDIS_ADDR defaults to 127.0.0.1:6379. --- SKIP: TestDeleteEntryKeepsChildListing (0.08s) === RUN TestRemovingTheLastChildDropsTheListing universal_redis_store_test.go:107: redis3 tests are disabled. Start a redis-server and set RUN_REDIS_TESTS=1 to enable, REDIS_ADDR defaults to 127.0.0.1:6379. --- SKIP: TestRemovingTheLastChildDropsTheListing (0.00s) PASS ok github.com/seaweedfs/seaweedfs/weed/filer/redis3 19.020s === RUN TestUnsetKeepsGoRedisDefaults --- PASS: TestUnsetKeepsGoRedisDefaults (1.11s) === RUN TestRead --- PASS: TestRead (0.00s) === RUN TestNegativeDurationIsIgnored --- PASS: TestNegativeDurationIsIgnored (0.00s) PASS ok github.com/seaweedfs/seaweedfs/weed/filer/redis_conf 13.309s === RUN TestDisabled --- PASS: TestDisabled (0.00s) === RUN TestEnabledWithoutCertificates --- PASS: TestEnabledWithoutCertificates (0.00s) === RUN TestMutualTls --- PASS: TestMutualTls (0.08s) === RUN TestUnreadableCaCertificate --- PASS: TestUnreadableCaCertificate (0.00s) === RUN TestCaCertificateWithoutPem --- PASS: TestCaCertificateWithoutPem (0.02s) === RUN TestClientCertificateWithoutKey --- PASS: TestClientCertificateWithoutKey (0.05s) PASS ok github.com/seaweedfs/seaweedfs/weed/filer/redis_tls 12.764s ? github.com/seaweedfs/seaweedfs/weed/filer/sqlite [no test files] ? github.com/seaweedfs/seaweedfs/weed/filer/store_test [no test files] ? github.com/seaweedfs/seaweedfs/weed/filer/tarantool [no test files] ? github.com/seaweedfs/seaweedfs/weed/filer/tikv [no test files] ? github.com/seaweedfs/seaweedfs/weed/filer/ydb [no test files] ? github.com/seaweedfs/seaweedfs/weed/filer_client [no test files] === RUN TestCompressRotated_Toggle --- PASS: TestCompressRotated_Toggle (0.00s) === RUN TestCompressFile --- PASS: TestCompressFile (0.10s) === RUN TestCompressFile_AlreadyGz --- PASS: TestCompressFile_AlreadyGz (0.00s) === RUN TestCompressFile_NonExistent --- PASS: TestCompressFile_NonExistent (0.00s) === RUN TestCompressFile_EmptyFile --- PASS: TestCompressFile_EmptyFile (0.36s) === RUN TestCompressFile_LargeContent --- PASS: TestCompressFile_LargeContent (0.47s) === RUN TestJSONMode_Toggle --- PASS: TestJSONMode_Toggle (0.00s) === RUN TestJSONMode_Output {"ts":"2026-03-19T15:30:00Z","level":"INFO","file":"glog_json_test.go","line":39,"msg":"hello json"} --- PASS: TestJSONMode_Output (0.00s) === RUN TestJSONMode_AllLevels {"ts":"2026-08-24T22:18:12.60792555Z","level":"INFO","file":"glog_json_test.go","line":78,"msg":"info msg"} {"ts":"2026-08-24T22:18:12.60799711Z","level":"WARNING","file":"glog_json_test.go","line":79,"msg":"warn msg"} {"ts":"2026-08-24T22:18:12.60810017Z","level":"ERROR","file":"glog_json_test.go","line":80,"msg":"error msg"} --- PASS: TestJSONMode_AllLevels (0.00s) === RUN TestJSONMode_Infof {"ts":"2026-08-24T22:18:12.60857801Z","level":"INFO","file":"glog_json_test.go","line":106,"msg":"count=42 name=test"} --- PASS: TestJSONMode_Infof (0.00s) === RUN TestJSONEscapeString --- PASS: TestJSONEscapeString (0.00s) === RUN TestItoa --- PASS: TestItoa (0.00s) === RUN TestJSONMode_TextFallback I0824 22:18:12.609095 glog_json_test.go:167 text mode --- PASS: TestJSONMode_TextFallback (0.00s) === RUN TestShortHostname --- PASS: TestShortHostname (0.00s) === RUN TestInfo I0824 22:18:12.610686 glog_test.go:92 test --- PASS: TestInfo (0.00s) === RUN TestInfoDepth I0824 22:18:12.610904 glog_test.go:109 depth-test0 I0824 22:18:12.610931 glog_test.go:110 depth-test1 --- PASS: TestInfoDepth (0.00s) === RUN TestCopyStandardLogToPanic --- PASS: TestCopyStandardLogToPanic (0.00s) === RUN TestStandardLog I0824 22:18:12.614950 glog_test.go:163 test --- PASS: TestStandardLog (0.00s) === RUN TestHeader I0102 15:04:05.067890 glog_test.go:181 test --- PASS: TestHeader (0.00s) === RUN TestError E0824 22:18:12.620546 glog_test.go:202 test --- PASS: TestError (0.00s) === RUN TestWarning W0824 22:18:12.620876 glog_test.go:224 test --- PASS: TestWarning (0.00s) === RUN TestV I0824 22:18:12.623114 glog_test.go:243 test --- PASS: TestV (0.00s) === RUN TestVmoduleOn I0824 22:18:12.623567 glog_test.go:267 test --- PASS: TestVmoduleOn (0.00s) === RUN TestVmoduleOff --- PASS: TestVmoduleOff (0.00s) === RUN TestVmoduleGlob --- PASS: TestVmoduleGlob (0.00s) === RUN TestRollover I0824 22:18:12.624437 glog_test.go:337 x I0824 22:18:12.628942 glog_test.go:351 xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx I0824 22:18:13.647258 glog_test.go:364 x --- PASS: TestRollover (1.13s) === RUN TestTimeBasedRollover I0824 22:18:13.754585 glog_test.go:393 x I0824 22:48:13.647472 glog_test.go:407 still within interval I0824 23:19:13.647472 glog_test.go:417 past interval --- PASS: TestTimeBasedRollover (0.05s) === RUN TestLogBacktraceAt I0824 22:18:13.801238 glog_test.go:448 we want a stack trace here goroutine 195 [running]: github.com/seaweedfs/seaweedfs/weed/glog.stacks(0x0) /build/seaweedfs/src/seaweedfs-4.44/weed/glog/glog.go:832 +0x82 github.com/seaweedfs/seaweedfs/weed/glog.(*loggingT).output(0x4eb860, 0x0, 0x3f5c2181c0, {0x27d3ba?, 0x1?}, 0x0?, 0x0) /build/seaweedfs/src/seaweedfs-4.44/weed/glog/glog.go:730 +0xb4 github.com/seaweedfs/seaweedfs/weed/glog.(*loggingT).printDepth(0x4eb860, 0x0, 0x1, {0x3f5c19de48, 0x1, 0x1}) /build/seaweedfs/src/seaweedfs-4.44/weed/glog/glog.go:667 +0xc0 github.com/seaweedfs/seaweedfs/weed/glog.(*loggingT).print(...) /build/seaweedfs/src/seaweedfs-4.44/weed/glog/glog.go:654 github.com/seaweedfs/seaweedfs/weed/glog.Info(...) /build/seaweedfs/src/seaweedfs-4.44/weed/glog/glog.go:1149 github.com/seaweedfs/seaweedfs/weed/glog.TestLogBacktraceAt(0x3f5cc12248) /build/seaweedfs/src/seaweedfs-4.44/weed/glog/glog_test.go:448 +0x35e testing.tRunner(0x3f5cc12248, 0x48ea28) /usr/lib/go/src/testing/testing.go:2193 +0xba created by testing.(*T).Run in goroutine 1 /usr/lib/go/src/testing/testing.go:2258 +0x36c --- PASS: TestLogBacktraceAt (0.00s) PASS ok github.com/seaweedfs/seaweedfs/weed/glog 5.132s === RUN TestRedactSensitiveFormValues --- PASS: TestRedactSensitiveFormValues (0.04s) === RUN TestListUsersResponseXMLOrdering --- PASS: TestListUsersResponseXMLOrdering (0.00s) === RUN TestErrorResponseXMLUsesTopLevelRequestId --- PASS: TestErrorResponseXMLUsesTopLevelRequestId (0.04s) === RUN TestValidateCallerSuppliedAccessKeyId === RUN TestValidateCallerSuppliedAccessKeyId/empty === RUN TestValidateCallerSuppliedAccessKeyId/three_chars === RUN TestValidateCallerSuppliedAccessKeyId/four_chars_ok === RUN TestValidateCallerSuppliedAccessKeyId/mixed_case_alnum === RUN TestValidateCallerSuppliedAccessKeyId/128_chars_ok === RUN TestValidateCallerSuppliedAccessKeyId/129_chars_too_long === RUN TestValidateCallerSuppliedAccessKeyId/slash_rejected === RUN TestValidateCallerSuppliedAccessKeyId/equals_rejected === RUN TestValidateCallerSuppliedAccessKeyId/dash_rejected === RUN TestValidateCallerSuppliedAccessKeyId/underscore_rejected === RUN TestValidateCallerSuppliedAccessKeyId/unicode_rejected --- PASS: TestValidateCallerSuppliedAccessKeyId (0.00s) --- PASS: TestValidateCallerSuppliedAccessKeyId/empty (0.00s) --- PASS: TestValidateCallerSuppliedAccessKeyId/three_chars (0.00s) --- PASS: TestValidateCallerSuppliedAccessKeyId/four_chars_ok (0.00s) --- PASS: TestValidateCallerSuppliedAccessKeyId/mixed_case_alnum (0.00s) --- PASS: TestValidateCallerSuppliedAccessKeyId/128_chars_ok (0.00s) --- PASS: TestValidateCallerSuppliedAccessKeyId/129_chars_too_long (0.00s) --- PASS: TestValidateCallerSuppliedAccessKeyId/slash_rejected (0.00s) --- PASS: TestValidateCallerSuppliedAccessKeyId/equals_rejected (0.00s) --- PASS: TestValidateCallerSuppliedAccessKeyId/dash_rejected (0.00s) --- PASS: TestValidateCallerSuppliedAccessKeyId/underscore_rejected (0.00s) --- PASS: TestValidateCallerSuppliedAccessKeyId/unicode_rejected (0.00s) === RUN TestValidateCallerSuppliedSecretAccessKey === RUN TestValidateCallerSuppliedSecretAccessKey/empty === RUN TestValidateCallerSuppliedSecretAccessKey/seven_chars === RUN TestValidateCallerSuppliedSecretAccessKey/eight_chars_ok === RUN TestValidateCallerSuppliedSecretAccessKey/128_chars_ok === RUN TestValidateCallerSuppliedSecretAccessKey/129_chars_too_long === RUN TestValidateCallerSuppliedSecretAccessKey/non-alnum_allowed_in_secret --- PASS: TestValidateCallerSuppliedSecretAccessKey (0.00s) --- PASS: TestValidateCallerSuppliedSecretAccessKey/empty (0.00s) --- PASS: TestValidateCallerSuppliedSecretAccessKey/seven_chars (0.00s) --- PASS: TestValidateCallerSuppliedSecretAccessKey/eight_chars_ok (0.00s) --- PASS: TestValidateCallerSuppliedSecretAccessKey/128_chars_ok (0.00s) --- PASS: TestValidateCallerSuppliedSecretAccessKey/129_chars_too_long (0.00s) --- PASS: TestValidateCallerSuppliedSecretAccessKey/non-alnum_allowed_in_secret (0.00s) === RUN TestFindAccessKeyOwner === RUN TestFindAccessKeyOwner/matches_user === RUN TestFindAccessKeyOwner/matches_service_account === RUN TestFindAccessKeyOwner/no_match === RUN TestFindAccessKeyOwner/empty_key === RUN TestFindAccessKeyOwner/nil_config --- PASS: TestFindAccessKeyOwner (0.00s) --- PASS: TestFindAccessKeyOwner/matches_user (0.00s) --- PASS: TestFindAccessKeyOwner/matches_service_account (0.00s) --- PASS: TestFindAccessKeyOwner/no_match (0.00s) --- PASS: TestFindAccessKeyOwner/empty_key (0.00s) --- PASS: TestFindAccessKeyOwner/nil_config (0.00s) PASS ok github.com/seaweedfs/seaweedfs/weed/iam 11.591s === RUN TestGetProviderByIssuerAndAccount === RUN TestGetProviderByIssuerAndAccount/matching_account_returns_scoped_record === RUN TestGetProviderByIssuerAndAccount/unknown_account_is_rejected === RUN TestGetProviderByIssuerAndAccount/global_provider_satisfies_any_account === RUN TestGetProviderByIssuerAndAccount/empty_caller_account_accepts_any_record --- PASS: TestGetProviderByIssuerAndAccount (0.11s) --- PASS: TestGetProviderByIssuerAndAccount/matching_account_returns_scoped_record (0.02s) --- PASS: TestGetProviderByIssuerAndAccount/unknown_account_is_rejected (0.00s) --- PASS: TestGetProviderByIssuerAndAccount/global_provider_satisfies_any_account (0.00s) --- PASS: TestGetProviderByIssuerAndAccount/empty_caller_account_accepts_any_record (0.00s) === RUN TestPolicyVariableSubstitution --- PASS: TestPolicyVariableSubstitution (0.15s) === RUN TestConditionWithNumericComparison === RUN TestConditionWithNumericComparison/duration_within_limit === RUN TestConditionWithNumericComparison/duration_at_limit === RUN TestConditionWithNumericComparison/duration_exceeding_limit --- PASS: TestConditionWithNumericComparison (0.03s) --- PASS: TestConditionWithNumericComparison/duration_within_limit (0.01s) --- PASS: TestConditionWithNumericComparison/duration_at_limit (0.01s) --- PASS: TestConditionWithNumericComparison/duration_exceeding_limit (0.00s) === RUN TestMultipleConditionOperators === RUN TestMultipleConditionOperators/all_conditions_match === RUN TestMultipleConditionOperators/aud_mismatch === RUN TestMultipleConditionOperators/sub_pattern_mismatch --- PASS: TestMultipleConditionOperators (0.01s) --- PASS: TestMultipleConditionOperators/all_conditions_match (0.00s) --- PASS: TestMultipleConditionOperators/aud_mismatch (0.00s) --- PASS: TestMultipleConditionOperators/sub_pattern_mismatch (0.00s) === RUN TestFullOIDCWorkflow === RUN TestFullOIDCWorkflow/successful_role_assumption_with_policy_validation === RUN TestFullOIDCWorkflow/role_assumption_denied_by_trust_policy === RUN TestFullOIDCWorkflow/invalid_token_rejected --- PASS: TestFullOIDCWorkflow (0.02s) --- PASS: TestFullOIDCWorkflow/successful_role_assumption_with_policy_validation (0.01s) --- PASS: TestFullOIDCWorkflow/role_assumption_denied_by_trust_policy (0.00s) --- PASS: TestFullOIDCWorkflow/invalid_token_rejected (0.00s) === RUN TestFullLDAPWorkflow === RUN TestFullLDAPWorkflow/successful_LDAP_role_assumption === RUN TestFullLDAPWorkflow/invalid_LDAP_credentials --- PASS: TestFullLDAPWorkflow (0.02s) --- PASS: TestFullLDAPWorkflow/successful_LDAP_role_assumption (0.01s) --- PASS: TestFullLDAPWorkflow/invalid_LDAP_credentials (0.00s) === RUN TestPolicyEnforcement === RUN TestPolicyEnforcement/allow_read_access === RUN TestPolicyEnforcement/allow_list_bucket === RUN TestPolicyEnforcement/deny_write_access === RUN TestPolicyEnforcement/deny_delete_access === RUN TestPolicyEnforcement/deny_filer_access --- PASS: TestPolicyEnforcement (0.06s) --- PASS: TestPolicyEnforcement/allow_read_access (0.02s) --- PASS: TestPolicyEnforcement/allow_list_bucket (0.00s) --- PASS: TestPolicyEnforcement/deny_write_access (0.00s) --- PASS: TestPolicyEnforcement/deny_delete_access (0.01s) --- PASS: TestPolicyEnforcement/deny_filer_access (0.01s) === RUN TestSessionPolicyBoundary --- PASS: TestSessionPolicyBoundary (0.02s) === RUN TestAssumeRoleWithWebIdentitySessionPolicy --- PASS: TestAssumeRoleWithWebIdentitySessionPolicy (0.02s) === RUN TestAssumeRoleWithCredentialsSessionPolicy --- PASS: TestAssumeRoleWithCredentialsSessionPolicy (0.01s) === RUN TestSessionExpiration --- PASS: TestSessionExpiration (0.02s) === RUN TestTrustPolicyValidation === RUN TestTrustPolicyValidation/OIDC_user_allowed_by_trust_policy === RUN TestTrustPolicyValidation/LDAP_user_allowed_by_different_role === RUN TestTrustPolicyValidation/Wrong_provider_for_role --- PASS: TestTrustPolicyValidation (0.01s) --- PASS: TestTrustPolicyValidation/OIDC_user_allowed_by_trust_policy (0.00s) --- PASS: TestTrustPolicyValidation/LDAP_user_allowed_by_different_role (0.00s) --- PASS: TestTrustPolicyValidation/Wrong_provider_for_role (0.00s) === RUN TestTrustPolicyWildcardPrincipal === RUN TestTrustPolicyWildcardPrincipal/Wildcard_federated_principal_allows_any_provider === RUN TestTrustPolicyWildcardPrincipal/Wildcard_in_array_allows_any_provider === RUN TestTrustPolicyWildcardPrincipal/Plain_wildcard_allows_any_provider_(regression) === RUN TestTrustPolicyWildcardPrincipal/Non-wildcard_federated_principal_requires_matching_provider === RUN TestTrustPolicyWildcardPrincipal/Interface_array_principal_works_correctly --- PASS: TestTrustPolicyWildcardPrincipal (0.02s) --- PASS: TestTrustPolicyWildcardPrincipal/Wildcard_federated_principal_allows_any_provider (0.00s) --- PASS: TestTrustPolicyWildcardPrincipal/Wildcard_in_array_allows_any_provider (0.00s) --- PASS: TestTrustPolicyWildcardPrincipal/Plain_wildcard_allows_any_provider_(regression) (0.00s) --- PASS: TestTrustPolicyWildcardPrincipal/Non-wildcard_federated_principal_requires_matching_provider (0.00s) --- PASS: TestTrustPolicyWildcardPrincipal/Interface_array_principal_works_correctly (0.00s) === RUN TestOIDCClaimsTrustPolicy --- PASS: TestOIDCClaimsTrustPolicy (0.01s) === RUN TestIsActionAllowed_RegisteredUserWithoutPoliciesIsDenied --- PASS: TestIsActionAllowed_RegisteredUserWithoutPoliciesIsDenied (0.00s) === RUN TestIsActionAllowed_RegisteredRoleWithoutPoliciesIsDenied --- PASS: TestIsActionAllowed_RegisteredRoleWithoutPoliciesIsDenied (0.00s) === RUN TestIsActionAllowed_RegisteredUserWithNonMatchingPolicyIsDenied --- PASS: TestIsActionAllowed_RegisteredUserWithNonMatchingPolicyIsDenied (0.00s) === RUN TestOIDCAuditLifecycleEventsAreEmitted --- PASS: TestOIDCAuditLifecycleEventsAreEmitted (0.00s) === RUN TestOIDCAuditDefaultsToGlogSink I0824 22:18:25.685240 oidc_provider_audit.go:66 oidc-audit: {"type":"Create","arn":"arn:aws:iam::111122223333:oidc-provider/idp.example","url":"https://idp.example","occurredAt":"2026-08-24T22:18:25.683730614Z"} --- PASS: TestOIDCAuditDefaultsToGlogSink (0.01s) === RUN TestStaticConfigSeedsProviderStore --- PASS: TestStaticConfigSeedsProviderStore (0.00s) === RUN TestStaticConfigMirrorsThumbprintsAndAdvancedFields --- PASS: TestStaticConfigMirrorsThumbprintsAndAdvancedFields (0.00s) === RUN TestStoreNotConfiguredReturnsClearError --- PASS: TestStoreNotConfiguredReturnsClearError (0.00s) === RUN TestMemoryStoreCRUD --- PASS: TestMemoryStoreCRUD (0.00s) === RUN TestMemoryStoreGetByIssuerNormalizesHost --- PASS: TestMemoryStoreGetByIssuerNormalizesHost (0.00s) === RUN TestMemoryStoreGetByIssuerMissing --- PASS: TestMemoryStoreGetByIssuerMissing (0.00s) === RUN TestDeriveOIDCProviderARN === RUN TestDeriveOIDCProviderARN/google === RUN TestDeriveOIDCProviderARN/EKS_with_path === RUN TestDeriveOIDCProviderARN/uppercase_host_normalized === RUN TestDeriveOIDCProviderARN/trailing_slash_trimmed === RUN TestDeriveOIDCProviderARN/empty_account_allowed --- PASS: TestDeriveOIDCProviderARN (0.00s) --- PASS: TestDeriveOIDCProviderARN/google (0.00s) --- PASS: TestDeriveOIDCProviderARN/EKS_with_path (0.00s) --- PASS: TestDeriveOIDCProviderARN/uppercase_host_normalized (0.00s) --- PASS: TestDeriveOIDCProviderARN/trailing_slash_trimmed (0.00s) --- PASS: TestDeriveOIDCProviderARN/empty_account_allowed (0.00s) === RUN TestDeriveOIDCProviderARNRejectsBadInput --- PASS: TestDeriveOIDCProviderARNRejectsBadInput (0.01s) === RUN TestStoreRejectsEmptyARN --- PASS: TestStoreRejectsEmptyARN (0.00s) === RUN TestStoreRejectsNil --- PASS: TestStoreRejectsNil (0.00s) === RUN TestNormalizeIssuerRejectsEmpty --- PASS: TestNormalizeIssuerRejectsEmpty (0.00s) === RUN TestNormalizeIssuerHandlesNonURL --- PASS: TestNormalizeIssuerHandlesNonURL (0.00s) === RUN TestCapDurationByRole === RUN TestCapDurationByRole/no_cap,_no_request === RUN TestCapDurationByRole/no_cap,_with_request === RUN TestCapDurationByRole/cap_only,_no_request_->_nil_so_STS_default_applies === RUN TestCapDurationByRole/request_below_cap_->_request === RUN TestCapDurationByRole/request_equal_cap_->_request === RUN TestCapDurationByRole/request_above_cap_->_cap --- PASS: TestCapDurationByRole (0.00s) --- PASS: TestCapDurationByRole/no_cap,_no_request (0.00s) --- PASS: TestCapDurationByRole/no_cap,_with_request (0.00s) --- PASS: TestCapDurationByRole/cap_only,_no_request_->_nil_so_STS_default_applies (0.00s) --- PASS: TestCapDurationByRole/request_below_cap_->_request (0.00s) --- PASS: TestCapDurationByRole/request_equal_cap_->_request (0.00s) --- PASS: TestCapDurationByRole/request_above_cap_->_cap (0.00s) === RUN TestMemoryRoleStore --- PASS: TestMemoryRoleStore (0.00s) === RUN TestRoleStoreConfiguration --- PASS: TestRoleStoreConfiguration (0.00s) === RUN TestDistributedIAMManagerWithRoleStore --- PASS: TestDistributedIAMManagerWithRoleStore (0.00s) === RUN TestMemoryRevocationStoreCRUD --- PASS: TestMemoryRevocationStoreCRUD (0.00s) === RUN TestMemoryRevocationStoreRejectsBadInput --- PASS: TestMemoryRevocationStoreRejectsBadInput (0.00s) === RUN TestMemoryRevocationStorePurge --- PASS: TestMemoryRevocationStorePurge (0.00s) === RUN TestIAMManagerRevocationDefaultIsNoop --- PASS: TestIAMManagerRevocationDefaultIsNoop (0.00s) === RUN TestIAMManagerRevocationFlow --- PASS: TestIAMManagerRevocationFlow (0.00s) === RUN TestTrustPolicyAWSUserPrincipal === RUN TestTrustPolicyAWSUserPrincipal/matching_user_principal_should_be_allowed === RUN TestTrustPolicyAWSUserPrincipal/non-matching_user_principal_should_be_denied --- PASS: TestTrustPolicyAWSUserPrincipal (0.00s) --- PASS: TestTrustPolicyAWSUserPrincipal/matching_user_principal_should_be_allowed (0.00s) --- PASS: TestTrustPolicyAWSUserPrincipal/non-matching_user_principal_should_be_denied (0.00s) === RUN TestTrustPolicyAWSWildcardPrincipal --- PASS: TestTrustPolicyAWSWildcardPrincipal (0.00s) === RUN TestTrustPolicyAWSUserArrayPrincipal === RUN TestTrustPolicyAWSUserArrayPrincipal/first_listed_user_is_allowed === RUN TestTrustPolicyAWSUserArrayPrincipal/second_listed_user_is_allowed === RUN TestTrustPolicyAWSUserArrayPrincipal/unlisted_user_is_denied --- PASS: TestTrustPolicyAWSUserArrayPrincipal (0.00s) --- PASS: TestTrustPolicyAWSUserArrayPrincipal/first_listed_user_is_allowed (0.00s) --- PASS: TestTrustPolicyAWSUserArrayPrincipal/second_listed_user_is_allowed (0.00s) --- PASS: TestTrustPolicyAWSUserArrayPrincipal/unlisted_user_is_denied (0.00s) === RUN TestTrustPolicyPlainStringPrincipal --- PASS: TestTrustPolicyPlainStringPrincipal (0.00s) PASS ok github.com/seaweedfs/seaweedfs/weed/iam/integration 17.070s ? github.com/seaweedfs/seaweedfs/weed/iam/ldap [no test files] === RUN TestDiscoveryHappyPath --- PASS: TestDiscoveryHappyPath (0.15s) === RUN TestDiscoveryFallback404 --- PASS: TestDiscoveryFallback404 (0.05s) === RUN TestDiscoveryDisabledByExplicitJWKSUri --- PASS: TestDiscoveryDisabledByExplicitJWKSUri (0.01s) === RUN TestDiscoveryRejectsIssuerMismatch --- PASS: TestDiscoveryRejectsIssuerMismatch (0.04s) === RUN TestDiscoveryRejectsMissingIssuer --- PASS: TestDiscoveryRejectsMissingIssuer (0.01s) === RUN TestOIDCProviderInitialization === RUN TestOIDCProviderInitialization/valid_config === RUN TestOIDCProviderInitialization/missing_issuer === RUN TestOIDCProviderInitialization/missing_client_id === RUN TestOIDCProviderInitialization/invalid_issuer_url --- PASS: TestOIDCProviderInitialization (0.00s) --- PASS: TestOIDCProviderInitialization/valid_config (0.00s) --- PASS: TestOIDCProviderInitialization/missing_issuer (0.00s) --- PASS: TestOIDCProviderInitialization/missing_client_id (0.00s) --- PASS: TestOIDCProviderInitialization/invalid_issuer_url (0.00s) === RUN TestOIDCProviderJWTValidation === RUN TestOIDCProviderJWTValidation/valid_token === RUN TestOIDCProviderJWTValidation/valid_token_with_array_audience === RUN TestOIDCProviderJWTValidation/expired_token === RUN TestOIDCProviderJWTValidation/invalid_signature --- PASS: TestOIDCProviderJWTValidation (1.86s) --- PASS: TestOIDCProviderJWTValidation/valid_token (0.20s) --- PASS: TestOIDCProviderJWTValidation/valid_token_with_array_audience (0.01s) --- PASS: TestOIDCProviderJWTValidation/expired_token (0.01s) --- PASS: TestOIDCProviderJWTValidation/invalid_signature (1.30s) === RUN TestOIDCProviderJWTValidationECDSA === RUN TestOIDCProviderJWTValidationECDSA/valid_token === RUN TestOIDCProviderJWTValidationECDSA/expired_token === RUN TestOIDCProviderJWTValidationECDSA/invalid_signature === RUN TestOIDCProviderJWTValidationECDSA/invalid_issuer === RUN TestOIDCProviderJWTValidationECDSA/invalid_audience --- PASS: TestOIDCProviderJWTValidationECDSA (0.09s) --- PASS: TestOIDCProviderJWTValidationECDSA/valid_token (0.03s) --- PASS: TestOIDCProviderJWTValidationECDSA/expired_token (0.01s) --- PASS: TestOIDCProviderJWTValidationECDSA/invalid_signature (0.01s) --- PASS: TestOIDCProviderJWTValidationECDSA/invalid_issuer (0.02s) --- PASS: TestOIDCProviderJWTValidationECDSA/invalid_audience (0.00s) === RUN TestOIDCProviderAuthentication === RUN TestOIDCProviderAuthentication/successful_authentication === RUN TestOIDCProviderAuthentication/successful_authentication_with_additional_attributes === RUN TestOIDCProviderAuthentication/authentication_with_invalid_token --- PASS: TestOIDCProviderAuthentication (0.42s) --- PASS: TestOIDCProviderAuthentication/successful_authentication (0.02s) --- PASS: TestOIDCProviderAuthentication/successful_authentication_with_additional_attributes (0.02s) --- PASS: TestOIDCProviderAuthentication/authentication_with_invalid_token (0.00s) === RUN TestOIDCProviderUserInfo === RUN TestOIDCProviderUserInfo/get_user_info_with_access_token === RUN TestOIDCProviderUserInfo/get_admin_user_info === RUN TestOIDCProviderUserInfo/get_user_info_without_token === RUN TestOIDCProviderUserInfo/get_user_info_with_invalid_token === RUN TestOIDCProviderUserInfo/get_user_info_with_custom_claims_mapping === RUN TestOIDCProviderUserInfo/get_user_info_with_empty_id --- PASS: TestOIDCProviderUserInfo (0.03s) --- PASS: TestOIDCProviderUserInfo/get_user_info_with_access_token (0.02s) --- PASS: TestOIDCProviderUserInfo/get_admin_user_info (0.00s) --- PASS: TestOIDCProviderUserInfo/get_user_info_without_token (0.00s) --- PASS: TestOIDCProviderUserInfo/get_user_info_with_invalid_token (0.00s) --- PASS: TestOIDCProviderUserInfo/get_user_info_with_custom_claims_mapping (0.00s) --- PASS: TestOIDCProviderUserInfo/get_user_info_with_empty_id (0.00s) === RUN TestExtractClaimPoliciesString --- PASS: TestExtractClaimPoliciesString (0.00s) === RUN TestExtractClaimPoliciesCommaSeparated --- PASS: TestExtractClaimPoliciesCommaSeparated (0.00s) === RUN TestExtractClaimPoliciesArray --- PASS: TestExtractClaimPoliciesArray (0.00s) === RUN TestExtractClaimPoliciesMissing --- PASS: TestExtractClaimPoliciesMissing (0.00s) === RUN TestExtractClaimPoliciesEmptyClaimName --- PASS: TestExtractClaimPoliciesEmptyClaimName (0.00s) === RUN TestExtractClaimPoliciesUnsupportedShape --- PASS: TestExtractClaimPoliciesUnsupportedShape (0.00s) === RUN TestExtractPrincipalTagsObjectShape --- PASS: TestExtractPrincipalTagsObjectShape (0.00s) === RUN TestExtractPrincipalTagsArrayValueTakesFirst --- PASS: TestExtractPrincipalTagsArrayValueTakesFirst (0.00s) === RUN TestExtractPrincipalTagsAbsent --- PASS: TestExtractPrincipalTagsAbsent (0.00s) === RUN TestExtractPrincipalTagsWrongShape --- PASS: TestExtractPrincipalTagsWrongShape (0.00s) === RUN TestFilterPrincipalTagsAllowlist === RUN TestFilterPrincipalTagsAllowlist/empty_allowlist_denies_all === RUN TestFilterPrincipalTagsAllowlist/partial_allowlist === RUN TestFilterPrincipalTagsAllowlist/allowlist_with_no_matches === RUN TestFilterPrincipalTagsAllowlist/case-insensitive_on_key --- PASS: TestFilterPrincipalTagsAllowlist (0.00s) --- PASS: TestFilterPrincipalTagsAllowlist/empty_allowlist_denies_all (0.00s) --- PASS: TestFilterPrincipalTagsAllowlist/partial_allowlist (0.00s) --- PASS: TestFilterPrincipalTagsAllowlist/allowlist_with_no_matches (0.00s) --- PASS: TestFilterPrincipalTagsAllowlist/case-insensitive_on_key (0.00s) === RUN TestThumbprintMatchAccepted W0824 22:18:23.569991 oidc_provider.go:370 OIDC provider "thumbprint-test" is configured to skip TLS verification. This is insecure and should not be used in production. --- PASS: TestThumbprintMatchAccepted (0.15s) === RUN TestThumbprintMismatchRejected W0824 22:18:23.707226 oidc_provider.go:370 OIDC provider "thumbprint-test" is configured to skip TLS verification. This is insecure and should not be used in production. 2026/08/24 22:18:23 http: TLS handshake error from 127.0.0.1:37324: remote error: tls: bad certificate 2026/08/24 22:18:23 http: TLS handshake error from 127.0.0.1:37328: remote error: tls: bad certificate --- PASS: TestThumbprintMismatchRejected (0.13s) === RUN TestThumbprintAllowlistAcceptsMixedCase W0824 22:18:23.824431 oidc_provider.go:370 OIDC provider "thumbprint-test" is configured to skip TLS verification. This is insecure and should not be used in production. --- PASS: TestThumbprintAllowlistAcceptsMixedCase (0.14s) === RUN TestThumbprintEmptyAllowlistSkipsCheck W0824 22:18:23.965902 oidc_provider.go:370 OIDC provider "thumbprint-test" is configured to skip TLS verification. This is insecure and should not be used in production. --- PASS: TestThumbprintEmptyAllowlistSkipsCheck (0.03s) === RUN TestThumbprintSkipVerifyHonoured W0824 22:18:24.000161 oidc_provider.go:370 OIDC provider "skip" is configured to skip TLS verification. This is insecure and should not be used in production. --- PASS: TestThumbprintSkipVerifyHonoured (0.01s) PASS ok github.com/seaweedfs/seaweedfs/weed/iam/oidc 15.275s === RUN TestAWSIAMMatch === RUN TestAWSIAMMatch/case_insensitive_exact_match === RUN TestAWSIAMMatch/case_insensitive_wildcard_match === RUN TestAWSIAMMatch/AWS_username_variable_expansion === RUN TestAWSIAMMatch/SAML_username_variable_expansion === RUN TestAWSIAMMatch/OIDC_subject_variable_expansion === RUN TestAWSIAMMatch/case_insensitive_with_variable === RUN TestAWSIAMMatch/universal_wildcard === RUN TestAWSIAMMatch/question_mark_wildcard === RUN TestAWSIAMMatch/no_match_different_pattern === RUN TestAWSIAMMatch/variable_not_expanded_due_to_missing_context --- PASS: TestAWSIAMMatch (0.10s) --- PASS: TestAWSIAMMatch/case_insensitive_exact_match (0.07s) --- PASS: TestAWSIAMMatch/case_insensitive_wildcard_match (0.00s) --- PASS: TestAWSIAMMatch/AWS_username_variable_expansion (0.00s) --- PASS: TestAWSIAMMatch/SAML_username_variable_expansion (0.00s) --- PASS: TestAWSIAMMatch/OIDC_subject_variable_expansion (0.00s) --- PASS: TestAWSIAMMatch/case_insensitive_with_variable (0.00s) --- PASS: TestAWSIAMMatch/universal_wildcard (0.00s) --- PASS: TestAWSIAMMatch/question_mark_wildcard (0.00s) --- PASS: TestAWSIAMMatch/no_match_different_pattern (0.00s) --- PASS: TestAWSIAMMatch/variable_not_expanded_due_to_missing_context (0.00s) === RUN TestMatchesActionsMultipartExpansion === RUN TestMatchesActionsMultipartExpansion/PutObject_directly_matches_PutObject === RUN TestMatchesActionsMultipartExpansion/PutObject_implicitly_allows_CreateMultipartUpload === RUN TestMatchesActionsMultipartExpansion/PutObject_implicitly_allows_UploadPart === RUN TestMatchesActionsMultipartExpansion/PutObject_implicitly_allows_CompleteMultipartUpload === RUN TestMatchesActionsMultipartExpansion/PutObject_implicitly_allows_AbortMultipartUpload === RUN TestMatchesActionsMultipartExpansion/PutObject_implicitly_allows_ListMultipartUploadParts === RUN TestMatchesActionsMultipartExpansion/PutObject_implicitly_allows_ListBucketMultipartUploads === RUN TestMatchesActionsMultipartExpansion/PutObject_does_not_allow_GetObject === RUN TestMatchesActionsMultipartExpansion/GetObject_does_not_allow_CreateMultipartUpload === RUN TestMatchesActionsMultipartExpansion/wildcard_s3:Put*_implicitly_allows_multipart_via_PutObject_match === RUN TestMatchesActionsMultipartExpansion/case-insensitive_multipart_action_lookup --- PASS: TestMatchesActionsMultipartExpansion (0.00s) --- PASS: TestMatchesActionsMultipartExpansion/PutObject_directly_matches_PutObject (0.00s) --- PASS: TestMatchesActionsMultipartExpansion/PutObject_implicitly_allows_CreateMultipartUpload (0.00s) --- PASS: TestMatchesActionsMultipartExpansion/PutObject_implicitly_allows_UploadPart (0.00s) --- PASS: TestMatchesActionsMultipartExpansion/PutObject_implicitly_allows_CompleteMultipartUpload (0.00s) --- PASS: TestMatchesActionsMultipartExpansion/PutObject_implicitly_allows_AbortMultipartUpload (0.00s) --- PASS: TestMatchesActionsMultipartExpansion/PutObject_implicitly_allows_ListMultipartUploadParts (0.00s) --- PASS: TestMatchesActionsMultipartExpansion/PutObject_implicitly_allows_ListBucketMultipartUploads (0.00s) --- PASS: TestMatchesActionsMultipartExpansion/PutObject_does_not_allow_GetObject (0.00s) --- PASS: TestMatchesActionsMultipartExpansion/GetObject_does_not_allow_CreateMultipartUpload (0.00s) --- PASS: TestMatchesActionsMultipartExpansion/wildcard_s3:Put*_implicitly_allows_multipart_via_PutObject_match (0.00s) --- PASS: TestMatchesActionsMultipartExpansion/case-insensitive_multipart_action_lookup (0.00s) === RUN TestExpandPolicyVariables === RUN TestExpandPolicyVariables/expand_aws_username === RUN TestExpandPolicyVariables/expand_multiple_variables === RUN TestExpandPolicyVariables/no_variables_to_expand === RUN TestExpandPolicyVariables/nil_context === RUN TestExpandPolicyVariables/missing_variable_in_context --- PASS: TestExpandPolicyVariables (0.00s) --- PASS: TestExpandPolicyVariables/expand_aws_username (0.00s) --- PASS: TestExpandPolicyVariables/expand_multiple_variables (0.00s) --- PASS: TestExpandPolicyVariables/no_variables_to_expand (0.00s) --- PASS: TestExpandPolicyVariables/nil_context (0.00s) --- PASS: TestExpandPolicyVariables/missing_variable_in_context (0.00s) === RUN TestAWSWildcardMatch === RUN TestAWSWildcardMatch/case_insensitive_asterisk === RUN TestAWSWildcardMatch/case_insensitive_question_mark === RUN TestAWSWildcardMatch/mixed_wildcards === RUN TestAWSWildcardMatch/no_match --- PASS: TestAWSWildcardMatch (0.00s) --- PASS: TestAWSWildcardMatch/case_insensitive_asterisk (0.00s) --- PASS: TestAWSWildcardMatch/case_insensitive_question_mark (0.00s) --- PASS: TestAWSWildcardMatch/mixed_wildcards (0.00s) --- PASS: TestAWSWildcardMatch/no_match (0.00s) === RUN TestStringifyClaimValue === RUN TestStringifyClaimValue/string === RUN TestStringifyClaimValue/bool_true === RUN TestStringifyClaimValue/bool_false === RUN TestStringifyClaimValue/float64_integer === RUN TestStringifyClaimValue/float64_fraction === RUN TestStringifyClaimValue/float32 === RUN TestStringifyClaimValue/int === RUN TestStringifyClaimValue/int8 === RUN TestStringifyClaimValue/int16 === RUN TestStringifyClaimValue/int32 === RUN TestStringifyClaimValue/int64 === RUN TestStringifyClaimValue/uint === RUN TestStringifyClaimValue/uint8 === RUN TestStringifyClaimValue/uint16 === RUN TestStringifyClaimValue/uint32 === RUN TestStringifyClaimValue/uint64 === RUN TestStringifyClaimValue/json.Number_int === RUN TestStringifyClaimValue/json.Number_float === RUN TestStringifyClaimValue/nil === RUN TestStringifyClaimValue/slice_unsupported === RUN TestStringifyClaimValue/map_unsupported --- PASS: TestStringifyClaimValue (0.00s) --- PASS: TestStringifyClaimValue/string (0.00s) --- PASS: TestStringifyClaimValue/bool_true (0.00s) --- PASS: TestStringifyClaimValue/bool_false (0.00s) --- PASS: TestStringifyClaimValue/float64_integer (0.00s) --- PASS: TestStringifyClaimValue/float64_fraction (0.00s) --- PASS: TestStringifyClaimValue/float32 (0.00s) --- PASS: TestStringifyClaimValue/int (0.00s) --- PASS: TestStringifyClaimValue/int8 (0.00s) --- PASS: TestStringifyClaimValue/int16 (0.00s) --- PASS: TestStringifyClaimValue/int32 (0.00s) --- PASS: TestStringifyClaimValue/int64 (0.00s) --- PASS: TestStringifyClaimValue/uint (0.00s) --- PASS: TestStringifyClaimValue/uint8 (0.00s) --- PASS: TestStringifyClaimValue/uint16 (0.00s) --- PASS: TestStringifyClaimValue/uint32 (0.00s) --- PASS: TestStringifyClaimValue/uint64 (0.00s) --- PASS: TestStringifyClaimValue/json.Number_int (0.00s) --- PASS: TestStringifyClaimValue/json.Number_float (0.00s) --- PASS: TestStringifyClaimValue/nil (0.00s) --- PASS: TestStringifyClaimValue/slice_unsupported (0.00s) --- PASS: TestStringifyClaimValue/map_unsupported (0.00s) === RUN TestExpandPolicyVariablesDynamicJWTClaims === RUN TestExpandPolicyVariablesDynamicJWTClaims/dynamic_jwt_claim_substituted_in_resource_arn === RUN TestExpandPolicyVariablesDynamicJWTClaims/multiple_dynamic_jwt_claims_in_one_pattern === RUN TestExpandPolicyVariablesDynamicJWTClaims/dynamic_numeric_jwt_claim_is_stringified === RUN TestExpandPolicyVariablesDynamicJWTClaims/dynamic_saml_claim_substituted === RUN TestExpandPolicyVariablesDynamicJWTClaims/dynamic_oidc_claim_substituted === RUN TestExpandPolicyVariablesDynamicJWTClaims/missing_jwt_claim_leaves_placeholder_intact === RUN TestExpandPolicyVariablesDynamicJWTClaims/non-identity_prefix_is_not_substituted --- PASS: TestExpandPolicyVariablesDynamicJWTClaims (0.04s) --- PASS: TestExpandPolicyVariablesDynamicJWTClaims/dynamic_jwt_claim_substituted_in_resource_arn (0.03s) --- PASS: TestExpandPolicyVariablesDynamicJWTClaims/multiple_dynamic_jwt_claims_in_one_pattern (0.00s) --- PASS: TestExpandPolicyVariablesDynamicJWTClaims/dynamic_numeric_jwt_claim_is_stringified (0.00s) --- PASS: TestExpandPolicyVariablesDynamicJWTClaims/dynamic_saml_claim_substituted (0.00s) --- PASS: TestExpandPolicyVariablesDynamicJWTClaims/dynamic_oidc_claim_substituted (0.00s) --- PASS: TestExpandPolicyVariablesDynamicJWTClaims/missing_jwt_claim_leaves_placeholder_intact (0.00s) --- PASS: TestExpandPolicyVariablesDynamicJWTClaims/non-identity_prefix_is_not_substituted (0.00s) === RUN TestGitLabProjectUploadPolicy === RUN TestGitLabProjectUploadPolicy/user_writes_into_own_project_folder_-_allowed === RUN TestGitLabProjectUploadPolicy/user_tries_to_write_into_another_project_-_falls_through_to_default_Deny === RUN TestGitLabProjectUploadPolicy/different_user_writes_into_their_own_project_-_allowed === RUN TestGitLabProjectUploadPolicy/user_tries_to_GetObject_-_only_PutObject_allowed,_default_Deny === RUN TestGitLabProjectUploadPolicy/missing_project_path_claim_-_placeholder_prevents_match,_default_Deny --- PASS: TestGitLabProjectUploadPolicy (0.01s) --- PASS: TestGitLabProjectUploadPolicy/user_writes_into_own_project_folder_-_allowed (0.00s) --- PASS: TestGitLabProjectUploadPolicy/user_tries_to_write_into_another_project_-_falls_through_to_default_Deny (0.00s) --- PASS: TestGitLabProjectUploadPolicy/different_user_writes_into_their_own_project_-_allowed (0.00s) --- PASS: TestGitLabProjectUploadPolicy/user_tries_to_GetObject_-_only_PutObject_allowed,_default_Deny (0.00s) --- PASS: TestGitLabProjectUploadPolicy/missing_project_path_claim_-_placeholder_prevents_match,_default_Deny (0.00s) === RUN TestDistributedPolicyEngine === RUN TestDistributedPolicyEngine/policy_storage_consistency === RUN TestDistributedPolicyEngine/evaluation_consistency === RUN TestDistributedPolicyEngine/deny_precedence_consistency === RUN TestDistributedPolicyEngine/default_effect_consistency --- PASS: TestDistributedPolicyEngine (0.00s) --- PASS: TestDistributedPolicyEngine/policy_storage_consistency (0.00s) --- PASS: TestDistributedPolicyEngine/evaluation_consistency (0.00s) --- PASS: TestDistributedPolicyEngine/deny_precedence_consistency (0.00s) --- PASS: TestDistributedPolicyEngine/default_effect_consistency (0.00s) === RUN TestPolicyEngineConfigurationConsistency === RUN TestPolicyEngineConfigurationConsistency/consistent_default_effects_required === RUN TestPolicyEngineConfigurationConsistency/invalid_configuration_handling === RUN TestPolicyEngineConfigurationConsistency/invalid_configuration_handling/invalid_config_0 === RUN TestPolicyEngineConfigurationConsistency/invalid_configuration_handling/invalid_config_1 --- PASS: TestPolicyEngineConfigurationConsistency (0.00s) --- PASS: TestPolicyEngineConfigurationConsistency/consistent_default_effects_required (0.00s) --- PASS: TestPolicyEngineConfigurationConsistency/invalid_configuration_handling (0.00s) --- PASS: TestPolicyEngineConfigurationConsistency/invalid_configuration_handling/invalid_config_0 (0.00s) --- PASS: TestPolicyEngineConfigurationConsistency/invalid_configuration_handling/invalid_config_1 (0.00s) === RUN TestPolicyStoreDistributed === RUN TestPolicyStoreDistributed/memory_store_isolation === RUN TestPolicyStoreDistributed/policy_loading_error_handling --- PASS: TestPolicyStoreDistributed (0.00s) --- PASS: TestPolicyStoreDistributed/memory_store_isolation (0.00s) --- PASS: TestPolicyStoreDistributed/policy_loading_error_handling (0.00s) === RUN TestFilerPolicyStoreConfiguration === RUN TestFilerPolicyStoreConfiguration/filer_store_creation === RUN TestFilerPolicyStoreConfiguration/filer_store_custom_path === RUN TestFilerPolicyStoreConfiguration/filer_store_missing_address --- PASS: TestFilerPolicyStoreConfiguration (0.06s) --- PASS: TestFilerPolicyStoreConfiguration/filer_store_creation (0.06s) --- PASS: TestFilerPolicyStoreConfiguration/filer_store_custom_path (0.00s) --- PASS: TestFilerPolicyStoreConfiguration/filer_store_missing_address (0.00s) === RUN TestPolicyEvaluationPerformance policy_engine_distributed_test.go:384: Average policy evaluation time: 87.374µs --- PASS: TestPolicyEvaluationPerformance (0.01s) === RUN TestFilerPolicyStoreGetPolicyPrefersCanonicalFiles --- PASS: TestFilerPolicyStoreGetPolicyPrefersCanonicalFiles (0.24s) === RUN TestFilerPolicyStoreListPoliciesIncludesCanonicalAndLegacyFiles --- PASS: TestFilerPolicyStoreListPoliciesIncludesCanonicalAndLegacyFiles (0.08s) === RUN TestFilerPolicyStoreDeletePolicyRemovesCanonicalAndLegacyFiles --- PASS: TestFilerPolicyStoreDeletePolicyRemovesCanonicalAndLegacyFiles (0.01s) === RUN TestFilerPolicyStoreStorePolicyWritesCanonicalFileAndRemovesLegacyTwin --- PASS: TestFilerPolicyStoreStorePolicyWritesCanonicalFileAndRemovesLegacyTwin (0.05s) === RUN TestFilerPolicyStoreStorePolicyUpdatesExistingCanonicalFile --- PASS: TestFilerPolicyStoreStorePolicyUpdatesExistingCanonicalFile (0.02s) === RUN TestCopyPolicyDocumentClonesConditionState --- PASS: TestCopyPolicyDocumentClonesConditionState (0.00s) === RUN TestIsAlreadyExistsPolicyStoreErrorUsesStatusCode --- PASS: TestIsAlreadyExistsPolicyStoreErrorUsesStatusCode (0.00s) === RUN TestPolicyVariableMatchingInActionsAndResources === RUN TestPolicyVariableMatchingInActionsAndResources/policy_variable_in_action_matches === RUN TestPolicyVariableMatchingInActionsAndResources/policy_variable_in_resource_matches === RUN TestPolicyVariableMatchingInActionsAndResources/saml_username_variable_in_resource === RUN TestPolicyVariableMatchingInActionsAndResources/policy_variable_no_match_wrong_user === RUN TestPolicyVariableMatchingInActionsAndResources/missing_policy_variable_context --- PASS: TestPolicyVariableMatchingInActionsAndResources (0.00s) --- PASS: TestPolicyVariableMatchingInActionsAndResources/policy_variable_in_action_matches (0.00s) --- PASS: TestPolicyVariableMatchingInActionsAndResources/policy_variable_in_resource_matches (0.00s) --- PASS: TestPolicyVariableMatchingInActionsAndResources/saml_username_variable_in_resource (0.00s) --- PASS: TestPolicyVariableMatchingInActionsAndResources/policy_variable_no_match_wrong_user (0.00s) --- PASS: TestPolicyVariableMatchingInActionsAndResources/missing_policy_variable_context (0.00s) === RUN TestActionResourceConsistencyWithStringConditions --- PASS: TestActionResourceConsistencyWithStringConditions (0.00s) PASS ok github.com/seaweedfs/seaweedfs/weed/iam/policy 15.673s ? github.com/seaweedfs/seaweedfs/weed/iam/providers [no test files] === RUN TestIsClaimBasedPolicyRoleArn === RUN TestIsClaimBasedPolicyRoleArn/empty_matches === RUN TestIsClaimBasedPolicyRoleArn/sentinel_matches === RUN TestIsClaimBasedPolicyRoleArn/concrete_role_does_not === RUN TestIsClaimBasedPolicyRoleArn/random_ARN_does_not --- PASS: TestIsClaimBasedPolicyRoleArn (0.03s) --- PASS: TestIsClaimBasedPolicyRoleArn/empty_matches (0.00s) --- PASS: TestIsClaimBasedPolicyRoleArn/sentinel_matches (0.00s) --- PASS: TestIsClaimBasedPolicyRoleArn/concrete_role_does_not (0.00s) --- PASS: TestIsClaimBasedPolicyRoleArn/random_ARN_does_not (0.00s) === RUN TestTemporaryCredentialPrefix --- PASS: TestTemporaryCredentialPrefix (0.00s) === RUN TestTemporaryCredentialFormat --- PASS: TestTemporaryCredentialFormat (0.00s) === RUN TestAssumeRoleForPrincipalValidatesDuration --- PASS: TestAssumeRoleForPrincipalValidatesDuration (0.00s) === RUN TestLookupOIDCProviderForAccountPrefersAccountMatch --- PASS: TestLookupOIDCProviderForAccountPrefersAccountMatch (0.00s) === RUN TestLookupOIDCProviderForAccountFallsBackToGlobal --- PASS: TestLookupOIDCProviderForAccountFallsBackToGlobal (0.00s) === RUN TestLookupOIDCProviderForAccountSkipsAccountSpecificWhenAccountUnknown --- PASS: TestLookupOIDCProviderForAccountSkipsAccountSpecificWhenAccountUnknown (0.03s) === RUN TestLookupOIDCProviderForAccountFallsBackToStatic --- PASS: TestLookupOIDCProviderForAccountFallsBackToStatic (0.00s) === RUN TestComputeParentUserStability --- PASS: TestComputeParentUserStability (0.00s) === RUN TestComputeParentUserDistinguishesIssuer --- PASS: TestComputeParentUserDistinguishesIssuer (0.00s) === RUN TestComputeParentUserDistinguishesSubject --- PASS: TestComputeParentUserDistinguishesSubject (0.00s) === RUN TestComputeParentUserEmptySub --- PASS: TestComputeParentUserEmptySub (0.00s) === RUN TestComputeParentUserEmptyIss --- PASS: TestComputeParentUserEmptyIss (0.00s) === RUN TestComputeParentUserEncoding --- PASS: TestComputeParentUserEncoding (0.00s) === RUN TestSessionClaimsRoundTripParentUser --- PASS: TestSessionClaimsRoundTripParentUser (0.00s) === RUN TestSTSSessionClaimsToSessionInfo --- PASS: TestSTSSessionClaimsToSessionInfo (0.00s) === RUN TestSTSSessionClaimsToSessionInfoCredentialGeneration --- PASS: TestSTSSessionClaimsToSessionInfoCredentialGeneration (0.00s) === RUN TestSTSSessionClaimsToSessionInfoPreservesAllFields --- PASS: TestSTSSessionClaimsToSessionInfoPreservesAllFields (0.00s) === RUN TestSTSSessionClaimsToSessionInfoEmptyFields --- PASS: TestSTSSessionClaimsToSessionInfoEmptyFields (0.00s) === RUN TestSTSSessionClaimsToSessionInfoCredentialExpiration === RUN TestSTSSessionClaimsToSessionInfoCredentialExpiration/future_expiration === RUN TestSTSSessionClaimsToSessionInfoCredentialExpiration/past_expiration === RUN TestSTSSessionClaimsToSessionInfoCredentialExpiration/near_future_expiration --- PASS: TestSTSSessionClaimsToSessionInfoCredentialExpiration (0.00s) --- PASS: TestSTSSessionClaimsToSessionInfoCredentialExpiration/future_expiration (0.00s) --- PASS: TestSTSSessionClaimsToSessionInfoCredentialExpiration/past_expiration (0.00s) --- PASS: TestSTSSessionClaimsToSessionInfoCredentialExpiration/near_future_expiration (0.00s) === RUN TestSessionInfoIntegration --- PASS: TestSessionInfoIntegration (0.00s) === RUN TestSecretAccessKeyDeterminism --- PASS: TestSecretAccessKeyDeterminism (0.00s) PASS ok github.com/seaweedfs/seaweedfs/weed/iam/sts 16.120s ? github.com/seaweedfs/seaweedfs/weed/iam/util [no test files] === RUN TestExtractRoleNameFromArn === RUN TestExtractRoleNameFromArn/legacy_format_simple_role_name === RUN TestExtractRoleNameFromArn/legacy_format_custom_role_name === RUN TestExtractRoleNameFromArn/legacy_format_with_path === RUN TestExtractRoleNameFromArn/legacy_format_with_nested_path === RUN TestExtractRoleNameFromArn/standard_format_simple_role_name === RUN TestExtractRoleNameFromArn/standard_format_custom_role_name === RUN TestExtractRoleNameFromArn/standard_format_with_path === RUN TestExtractRoleNameFromArn/standard_format_with_nested_path === RUN TestExtractRoleNameFromArn/invalid_arn_missing_prefix === RUN TestExtractRoleNameFromArn/invalid_arn_incomplete === RUN TestExtractRoleNameFromArn/invalid_arn_no_role_marker === RUN TestExtractRoleNameFromArn/invalid_arn_wrong_service === RUN TestExtractRoleNameFromArn/empty_string === RUN TestExtractRoleNameFromArn/role_marker_no_name === RUN TestExtractRoleNameFromArn/standard_format_role_marker_no_name --- PASS: TestExtractRoleNameFromArn (0.01s) --- PASS: TestExtractRoleNameFromArn/legacy_format_simple_role_name (0.00s) --- PASS: TestExtractRoleNameFromArn/legacy_format_custom_role_name (0.00s) --- PASS: TestExtractRoleNameFromArn/legacy_format_with_path (0.00s) --- PASS: TestExtractRoleNameFromArn/legacy_format_with_nested_path (0.00s) --- PASS: TestExtractRoleNameFromArn/standard_format_simple_role_name (0.00s) --- PASS: TestExtractRoleNameFromArn/standard_format_custom_role_name (0.00s) --- PASS: TestExtractRoleNameFromArn/standard_format_with_path (0.00s) --- PASS: TestExtractRoleNameFromArn/standard_format_with_nested_path (0.00s) --- PASS: TestExtractRoleNameFromArn/invalid_arn_missing_prefix (0.00s) --- PASS: TestExtractRoleNameFromArn/invalid_arn_incomplete (0.00s) --- PASS: TestExtractRoleNameFromArn/invalid_arn_no_role_marker (0.00s) --- PASS: TestExtractRoleNameFromArn/invalid_arn_wrong_service (0.00s) --- PASS: TestExtractRoleNameFromArn/empty_string (0.00s) --- PASS: TestExtractRoleNameFromArn/role_marker_no_name (0.00s) --- PASS: TestExtractRoleNameFromArn/standard_format_role_marker_no_name (0.00s) === RUN TestExtractRoleNameFromPrincipal === RUN TestExtractRoleNameFromPrincipal/sts_assumed_role_legacy === RUN TestExtractRoleNameFromPrincipal/sts_assumed_role_legacy_no_session === RUN TestExtractRoleNameFromPrincipal/sts_assumed_role_standard === RUN TestExtractRoleNameFromPrincipal/sts_assumed_role_standard_no_session === RUN TestExtractRoleNameFromPrincipal/iam_role_legacy === RUN TestExtractRoleNameFromPrincipal/iam_role_legacy_with_path === RUN TestExtractRoleNameFromPrincipal/iam_role_standard === RUN TestExtractRoleNameFromPrincipal/iam_role_standard_with_path === RUN TestExtractRoleNameFromPrincipal/invalid_principal === RUN TestExtractRoleNameFromPrincipal/empty_string --- PASS: TestExtractRoleNameFromPrincipal (0.00s) --- PASS: TestExtractRoleNameFromPrincipal/sts_assumed_role_legacy (0.00s) --- PASS: TestExtractRoleNameFromPrincipal/sts_assumed_role_legacy_no_session (0.00s) --- PASS: TestExtractRoleNameFromPrincipal/sts_assumed_role_standard (0.00s) --- PASS: TestExtractRoleNameFromPrincipal/sts_assumed_role_standard_no_session (0.00s) --- PASS: TestExtractRoleNameFromPrincipal/iam_role_legacy (0.00s) --- PASS: TestExtractRoleNameFromPrincipal/iam_role_legacy_with_path (0.00s) --- PASS: TestExtractRoleNameFromPrincipal/iam_role_standard (0.00s) --- PASS: TestExtractRoleNameFromPrincipal/iam_role_standard_with_path (0.00s) --- PASS: TestExtractRoleNameFromPrincipal/invalid_principal (0.00s) --- PASS: TestExtractRoleNameFromPrincipal/empty_string (0.00s) === RUN TestParseRoleARN === RUN TestParseRoleARN/legacy_format_simple_role === RUN TestParseRoleARN/legacy_format_with_path === RUN TestParseRoleARN/standard_format_simple_role === RUN TestParseRoleARN/standard_format_with_path === RUN TestParseRoleARN/invalid_arn_missing_prefix === RUN TestParseRoleARN/invalid_arn_no_role_marker === RUN TestParseRoleARN/invalid_arn_empty_role_name === RUN TestParseRoleARN/invalid_arn_empty_role_name_standard_format --- PASS: TestParseRoleARN (0.01s) --- PASS: TestParseRoleARN/legacy_format_simple_role (0.00s) --- PASS: TestParseRoleARN/legacy_format_with_path (0.00s) --- PASS: TestParseRoleARN/standard_format_simple_role (0.00s) --- PASS: TestParseRoleARN/standard_format_with_path (0.00s) --- PASS: TestParseRoleARN/invalid_arn_missing_prefix (0.00s) --- PASS: TestParseRoleARN/invalid_arn_no_role_marker (0.00s) --- PASS: TestParseRoleARN/invalid_arn_empty_role_name (0.00s) --- PASS: TestParseRoleARN/invalid_arn_empty_role_name_standard_format (0.00s) === RUN TestParsePrincipalARN === RUN TestParsePrincipalARN/sts_assumed_role_legacy === RUN TestParsePrincipalARN/sts_assumed_role_standard === RUN TestParsePrincipalARN/sts_assumed_role_no_session === RUN TestParsePrincipalARN/iam_role_legacy === RUN TestParsePrincipalARN/iam_role_standard === RUN TestParsePrincipalARN/iam_role_with_path === RUN TestParsePrincipalARN/invalid_principal === RUN TestParsePrincipalARN/invalid_sts_empty_role_name === RUN TestParsePrincipalARN/invalid_sts_empty_role_name_standard_format --- PASS: TestParsePrincipalARN (0.00s) --- PASS: TestParsePrincipalARN/sts_assumed_role_legacy (0.00s) --- PASS: TestParsePrincipalARN/sts_assumed_role_standard (0.00s) --- PASS: TestParsePrincipalARN/sts_assumed_role_no_session (0.00s) --- PASS: TestParsePrincipalARN/iam_role_legacy (0.00s) --- PASS: TestParsePrincipalARN/iam_role_standard (0.00s) --- PASS: TestParsePrincipalARN/iam_role_with_path (0.00s) --- PASS: TestParsePrincipalARN/invalid_principal (0.00s) --- PASS: TestParsePrincipalARN/invalid_sts_empty_role_name (0.00s) --- PASS: TestParsePrincipalARN/invalid_sts_empty_role_name_standard_format (0.00s) === RUN TestSecurityMaliciousUserARNs === RUN TestSecurityMaliciousUserARNs/user_ARN_with_role/_in_path === RUN TestSecurityMaliciousUserARNs/policy_ARN_with_role/_in_name === RUN TestSecurityMaliciousUserARNs/group_ARN_with_role/_in_name === RUN TestSecurityMaliciousUserARNs/legacy_user_ARN_with_role/_in_path --- PASS: TestSecurityMaliciousUserARNs (0.00s) --- PASS: TestSecurityMaliciousUserARNs/user_ARN_with_role/_in_path (0.00s) --- PASS: TestSecurityMaliciousUserARNs/policy_ARN_with_role/_in_name (0.00s) --- PASS: TestSecurityMaliciousUserARNs/group_ARN_with_role/_in_name (0.00s) --- PASS: TestSecurityMaliciousUserARNs/legacy_user_ARN_with_role/_in_path (0.00s) === RUN TestSecurityMaliciousSTSUserARNs === RUN TestSecurityMaliciousSTSUserARNs/STS_user_with_assumed-role_in_path === RUN TestSecurityMaliciousSTSUserARNs/legacy_STS_user_with_assumed-role_in_path --- PASS: TestSecurityMaliciousSTSUserARNs (0.00s) --- PASS: TestSecurityMaliciousSTSUserARNs/STS_user_with_assumed-role_in_path (0.00s) --- PASS: TestSecurityMaliciousSTSUserARNs/legacy_STS_user_with_assumed-role_in_path (0.00s) === RUN TestEdgeCaseMultipleRoleMarkers === RUN TestEdgeCaseMultipleRoleMarkers/legacy_format_role_in_path === RUN TestEdgeCaseMultipleRoleMarkers/standard_format_role_in_path === RUN TestEdgeCaseMultipleRoleMarkers/multiple_role_markers_in_path === RUN TestEdgeCaseMultipleRoleMarkers/sts_assumed_role_with_role_in_path --- PASS: TestEdgeCaseMultipleRoleMarkers (0.01s) --- PASS: TestEdgeCaseMultipleRoleMarkers/legacy_format_role_in_path (0.00s) --- PASS: TestEdgeCaseMultipleRoleMarkers/standard_format_role_in_path (0.00s) --- PASS: TestEdgeCaseMultipleRoleMarkers/multiple_role_markers_in_path (0.00s) --- PASS: TestEdgeCaseMultipleRoleMarkers/sts_assumed_role_with_role_in_path (0.00s) === RUN TestEdgeCaseConsecutiveSlashes === RUN TestEdgeCaseConsecutiveSlashes/consecutive_slashes_immediately_after_role === RUN TestEdgeCaseConsecutiveSlashes/consecutive_slashes_in_path === RUN TestEdgeCaseConsecutiveSlashes/multiple_consecutive_slashes --- PASS: TestEdgeCaseConsecutiveSlashes (0.00s) --- PASS: TestEdgeCaseConsecutiveSlashes/consecutive_slashes_immediately_after_role (0.00s) --- PASS: TestEdgeCaseConsecutiveSlashes/consecutive_slashes_in_path (0.00s) --- PASS: TestEdgeCaseConsecutiveSlashes/multiple_consecutive_slashes (0.00s) === RUN TestEdgeCaseSpecialCharactersInRoleName === RUN TestEdgeCaseSpecialCharactersInRoleName/role_name_with_hyphens === RUN TestEdgeCaseSpecialCharactersInRoleName/role_name_with_underscores === RUN TestEdgeCaseSpecialCharactersInRoleName/role_name_with_dots === RUN TestEdgeCaseSpecialCharactersInRoleName/role_name_with_at_sign === RUN TestEdgeCaseSpecialCharactersInRoleName/role_name_with_plus_and_equals === RUN TestEdgeCaseSpecialCharactersInRoleName/role_name_with_commas === RUN TestEdgeCaseSpecialCharactersInRoleName/role_name_with_mixed_special_chars === RUN TestEdgeCaseSpecialCharactersInRoleName/path_with_special_characters --- PASS: TestEdgeCaseSpecialCharactersInRoleName (0.01s) --- PASS: TestEdgeCaseSpecialCharactersInRoleName/role_name_with_hyphens (0.00s) --- PASS: TestEdgeCaseSpecialCharactersInRoleName/role_name_with_underscores (0.00s) --- PASS: TestEdgeCaseSpecialCharactersInRoleName/role_name_with_dots (0.00s) --- PASS: TestEdgeCaseSpecialCharactersInRoleName/role_name_with_at_sign (0.00s) --- PASS: TestEdgeCaseSpecialCharactersInRoleName/role_name_with_plus_and_equals (0.00s) --- PASS: TestEdgeCaseSpecialCharactersInRoleName/role_name_with_commas (0.00s) --- PASS: TestEdgeCaseSpecialCharactersInRoleName/role_name_with_mixed_special_chars (0.00s) --- PASS: TestEdgeCaseSpecialCharactersInRoleName/path_with_special_characters (0.00s) === RUN TestEdgeCaseExtremelyLongRoleName === RUN TestEdgeCaseExtremelyLongRoleName/role_name_at_max_length === RUN TestEdgeCaseExtremelyLongRoleName/role_with_long_path_components --- PASS: TestEdgeCaseExtremelyLongRoleName (0.00s) --- PASS: TestEdgeCaseExtremelyLongRoleName/role_name_at_max_length (0.00s) --- PASS: TestEdgeCaseExtremelyLongRoleName/role_with_long_path_components (0.00s) PASS ok github.com/seaweedfs/seaweedfs/weed/iam/utils 0.658s === RUN TestIamManagementDeniedForDataPlanePolicy === RUN TestIamManagementDeniedForDataPlanePolicy/CreateUser === RUN TestIamManagementDeniedForDataPlanePolicy/CreateAccessKey === RUN TestIamManagementDeniedForDataPlanePolicy/PutUserPolicy === RUN TestIamManagementDeniedForDataPlanePolicy/AttachUserPolicy --- PASS: TestIamManagementDeniedForDataPlanePolicy (0.26s) --- PASS: TestIamManagementDeniedForDataPlanePolicy/CreateUser (0.13s) --- PASS: TestIamManagementDeniedForDataPlanePolicy/CreateAccessKey (0.00s) --- PASS: TestIamManagementDeniedForDataPlanePolicy/PutUserPolicy (0.00s) --- PASS: TestIamManagementDeniedForDataPlanePolicy/AttachUserPolicy (0.00s) === RUN TestIamManagementDeniedWithS3QueryParameter === RUN TestIamManagementDeniedWithS3QueryParameter/delete === RUN TestIamManagementDeniedWithS3QueryParameter/acl === RUN TestIamManagementDeniedWithS3QueryParameter/tagging === RUN TestIamManagementDeniedWithS3QueryParameter/policy --- PASS: TestIamManagementDeniedWithS3QueryParameter (0.03s) --- PASS: TestIamManagementDeniedWithS3QueryParameter/delete (0.02s) --- PASS: TestIamManagementDeniedWithS3QueryParameter/acl (0.00s) --- PASS: TestIamManagementDeniedWithS3QueryParameter/tagging (0.00s) --- PASS: TestIamManagementDeniedWithS3QueryParameter/policy (0.00s) === RUN TestIamSelfServiceAllowedForDataPlanePolicy --- PASS: TestIamSelfServiceAllowedForDataPlanePolicy (0.02s) === RUN TestIamManagementAllowedForAdmin --- PASS: TestIamManagementAllowedForAdmin (0.01s) === RUN TestGetActionsUserPath --- PASS: TestGetActionsUserPath (0.00s) === RUN TestGetActionsWildcardPath --- PASS: TestGetActionsWildcardPath (0.00s) === RUN TestGetActionsInvalidAction --- PASS: TestGetActionsInvalidAction (0.00s) === RUN TestPutGetUserPolicyPreservesStatements --- PASS: TestPutGetUserPolicyPreservesStatements (0.01s) === RUN TestPutGetUserPolicyIssue9008 --- PASS: TestPutGetUserPolicyIssue9008 (0.00s) === RUN TestMultipleInlinePoliciesAggregateActions --- PASS: TestMultipleInlinePoliciesAggregateActions (0.00s) === RUN TestGetPolicy --- PASS: TestGetPolicy (0.00s) === RUN TestDeletePolicy --- PASS: TestDeletePolicy (0.00s) === RUN TestListPolicies --- PASS: TestListPolicies (0.01s) === RUN TestAttachUserPolicy --- PASS: TestAttachUserPolicy (0.00s) === RUN TestManagedPolicyActionsPreservedAcrossInlineMutations --- PASS: TestManagedPolicyActionsPreservedAcrossInlineMutations (0.00s) === RUN TestDetachUserPolicy --- PASS: TestDetachUserPolicy (0.00s) === RUN TestListAttachedUserPolicies --- PASS: TestListAttachedUserPolicies (0.00s) === RUN TestUserArnIsValid --- PASS: TestUserArnIsValid (0.00s) === RUN TestCreateAccessKeyWithCallerSuppliedKeys --- PASS: TestCreateAccessKeyWithCallerSuppliedKeys (0.00s) === RUN TestCreateAccessKeyRandomGeneration --- PASS: TestCreateAccessKeyRandomGeneration (0.00s) === RUN TestCreateAccessKeyRejectsWeakKeys --- PASS: TestCreateAccessKeyRejectsWeakKeys (0.00s) === RUN TestCreateAccessKeyRejectsCollision === RUN TestCreateAccessKeyRejectsCollision/identity_credential === RUN TestCreateAccessKeyRejectsCollision/service_account_credential --- PASS: TestCreateAccessKeyRejectsCollision (0.00s) --- PASS: TestCreateAccessKeyRejectsCollision/identity_credential (0.00s) --- PASS: TestCreateAccessKeyRejectsCollision/service_account_credential (0.00s) === RUN TestCreateAccessKeyBoundary --- PASS: TestCreateAccessKeyBoundary (0.01s) === RUN TestCreateAccessKeyRejectsPartialSupply --- PASS: TestCreateAccessKeyRejectsPartialSupply (0.00s) === RUN TestPutPoliciesCreatesInCredentialStore --- PASS: TestPutPoliciesCreatesInCredentialStore (0.00s) === RUN TestGetPoliciesReadsFromCredentialStore --- PASS: TestGetPoliciesReadsFromCredentialStore (0.00s) === RUN TestPutPoliciesDeletesRemovedFromCredentialStore --- PASS: TestPutPoliciesDeletesRemovedFromCredentialStore (0.00s) === RUN TestPutPoliciesUpdatesChangedInCredentialStore --- PASS: TestPutPoliciesUpdatesChangedInCredentialStore (0.00s) === RUN TestPutPoliciesRoutesUserInlineThroughCredentialStore --- PASS: TestPutPoliciesRoutesUserInlineThroughCredentialStore (0.00s) === RUN TestPutPoliciesDeletesRemovedUserInline --- PASS: TestPutPoliciesDeletesRemovedUserInline (0.00s) === RUN TestPutPoliciesRoutesGroupInlineThroughCredentialStore --- PASS: TestPutPoliciesRoutesGroupInlineThroughCredentialStore (0.00s) === RUN TestPutPoliciesDeletesRemovedGroupInline --- PASS: TestPutPoliciesDeletesRemovedGroupInline (0.00s) === RUN TestPutPoliciesIsAuthoritativeWhenStoreIsNotFilerEtc --- PASS: TestPutPoliciesIsAuthoritativeWhenStoreIsNotFilerEtc (0.00s) === RUN TestGetPoliciesReadsInlineAndGroupInlineFromCredentialStore --- PASS: TestGetPoliciesReadsInlineAndGroupInlineFromCredentialStore (0.00s) === RUN TestCreateUser --- PASS: TestCreateUser (0.04s) === RUN TestListUsers --- PASS: TestListUsers (0.00s) === RUN TestListUsersRequestIdMatchesResponseHeader --- PASS: TestListUsersRequestIdMatchesResponseHeader (0.00s) === RUN TestListAccessKeys --- PASS: TestListAccessKeys (0.00s) === RUN TestUpdateAccessKey --- PASS: TestUpdateAccessKey (0.00s) === RUN TestGetUser --- PASS: TestGetUser (0.00s) === RUN TestCreatePolicy --- PASS: TestCreatePolicy (0.00s) === RUN TestCreatePolicyVersion --- PASS: TestCreatePolicyVersion (0.01s) === RUN TestCreatePolicyVersionMissingPolicy E0824 22:18:34.894636 iamapi_handlers.go:30 Response policy does-not-exist not found --- PASS: TestCreatePolicyVersionMissingPolicy (0.01s) === RUN TestCreatePolicyVersionRequiresSetAsDefault E0824 22:18:34.904435 iamapi_handlers.go:30 Response SetAsDefault must be true: SeaweedFS stores a single managed policy version --- PASS: TestCreatePolicyVersionRequiresSetAsDefault (0.00s) === RUN TestPutUserPolicy --- PASS: TestPutUserPolicy (0.00s) === RUN TestPutUserPolicyError E0824 22:18:34.907771 iamapi_management_handlers.go:1403 PutUserPolicy: the user with name InvalidUser cannot be found E0824 22:18:34.907876 iamapi_handlers.go:30 Response the user with name InvalidUser cannot be found --- PASS: TestPutUserPolicyError (0.00s) === RUN TestGetUserPolicy --- PASS: TestGetUserPolicy (0.00s) === RUN TestUpdateUser --- PASS: TestUpdateUser (0.00s) === RUN TestDeleteUser --- PASS: TestDeleteUser (0.00s) === RUN TestHandleImplicitUsername --- PASS: TestHandleImplicitUsername (0.01s) === RUN TestCreateUserDoesNotSaveAllUsers --- PASS: TestCreateUserDoesNotSaveAllUsers (0.00s) PASS ok github.com/seaweedfs/seaweedfs/weed/iamapi 26.143s === RUN TestCropping --- PASS: TestCropping (1.10s) === RUN TestXYZ --- PASS: TestXYZ (14.88s) === RUN TestResizing --- PASS: TestResizing (0.42s) PASS ok github.com/seaweedfs/seaweedfs/weed/images 24.637s === RUN TestCachedProviderDecryptHitsProviderOnce --- PASS: TestCachedProviderDecryptHitsProviderOnce (0.01s) === RUN TestCachedProviderContextSensitivity --- PASS: TestCachedProviderContextSensitivity (0.00s) === RUN TestCachedProviderTTLExpiry --- PASS: TestCachedProviderTTLExpiry (0.08s) === RUN TestCachedProviderMaxEntries --- PASS: TestCachedProviderMaxEntries (0.00s) === RUN TestCachedProviderClearsOverwrittenPlaintext --- PASS: TestCachedProviderClearsOverwrittenPlaintext (0.00s) === RUN TestCachedProviderNoWriteAfterClose --- PASS: TestCachedProviderNoWriteAfterClose (0.00s) === RUN TestAddKMSProviderWiresCache --- PASS: TestAddKMSProviderWiresCache (0.00s) === RUN TestCiphertextEnvelope_CreateAndParse --- PASS: TestCiphertextEnvelope_CreateAndParse (0.05s) === RUN TestCiphertextEnvelope_InvalidFormat --- PASS: TestCiphertextEnvelope_InvalidFormat (0.01s) === RUN TestCiphertextEnvelope_ValidationErrors === RUN TestCiphertextEnvelope_ValidationErrors/Valid === RUN TestCiphertextEnvelope_ValidationErrors/Empty_provider === RUN TestCiphertextEnvelope_ValidationErrors/Empty_keyID === RUN TestCiphertextEnvelope_ValidationErrors/Empty_ciphertext --- PASS: TestCiphertextEnvelope_ValidationErrors (0.01s) --- PASS: TestCiphertextEnvelope_ValidationErrors/Valid (0.00s) --- PASS: TestCiphertextEnvelope_ValidationErrors/Empty_provider (0.00s) --- PASS: TestCiphertextEnvelope_ValidationErrors/Empty_keyID (0.00s) --- PASS: TestCiphertextEnvelope_ValidationErrors/Empty_ciphertext (0.00s) === RUN TestCiphertextEnvelope_MultipleProviders === RUN TestCiphertextEnvelope_MultipleProviders/openbao === RUN TestCiphertextEnvelope_MultipleProviders/gcp === RUN TestCiphertextEnvelope_MultipleProviders/azure === RUN TestCiphertextEnvelope_MultipleProviders/aws --- PASS: TestCiphertextEnvelope_MultipleProviders (0.01s) --- PASS: TestCiphertextEnvelope_MultipleProviders/openbao (0.00s) --- PASS: TestCiphertextEnvelope_MultipleProviders/gcp (0.01s) --- PASS: TestCiphertextEnvelope_MultipleProviders/azure (0.00s) --- PASS: TestCiphertextEnvelope_MultipleProviders/aws (0.00s) PASS ok github.com/seaweedfs/seaweedfs/weed/kms 10.616s ? github.com/seaweedfs/seaweedfs/weed/kms/aws [no test files] ? github.com/seaweedfs/seaweedfs/weed/kms/gcp [no test files] ? github.com/seaweedfs/seaweedfs/weed/kms/local [no test files] ? github.com/seaweedfs/seaweedfs/weed/kms/openbao [no test files] === RUN TestChunkedDirtyPagesFailWritesAfterUploadError --- PASS: TestChunkedDirtyPagesFailWritesAfterUploadError (0.08s) === RUN TestGrpcErrorToFuseStatusUnwrapsCanceledThroughFmtErrorf --- PASS: TestGrpcErrorToFuseStatusUnwrapsCanceledThroughFmtErrorf (0.13s) === RUN TestGrpcErrorToFuseStatusDropsCanceledThroughPercentV --- PASS: TestGrpcErrorToFuseStatusDropsCanceledThroughPercentV (0.00s) === RUN TestWriteErrorToFuseStatus === RUN TestWriteErrorToFuseStatus/assign_shed_during_growth === RUN TestWriteErrorToFuseStatus/no_free_volumes_left === RUN TestWriteErrorToFuseStatus/unrelated_upload_failure --- PASS: TestWriteErrorToFuseStatus (0.00s) --- PASS: TestWriteErrorToFuseStatus/assign_shed_during_growth (0.00s) --- PASS: TestWriteErrorToFuseStatus/no_free_volumes_left (0.00s) --- PASS: TestWriteErrorToFuseStatus/unrelated_upload_failure (0.00s) === RUN TestIsRetryableFilerError === RUN TestIsRetryableFilerError/nil === RUN TestIsRetryableFilerError/canceled === RUN TestIsRetryableFilerError/unavailable === RUN TestIsRetryableFilerError/deadline_exceeded === RUN TestIsRetryableFilerError/resource_exhausted === RUN TestIsRetryableFilerError/internal === RUN TestIsRetryableFilerError/not_found === RUN TestIsRetryableFilerError/already_exists === RUN TestIsRetryableFilerError/invalid_argument === RUN TestIsRetryableFilerError/permission_denied === RUN TestIsRetryableFilerError/unauthenticated === RUN TestIsRetryableFilerError/failed_precondition === RUN TestIsRetryableFilerError/plain_error_retries === RUN TestIsRetryableFilerError/wrapped_canceled_still_retries === RUN TestIsRetryableFilerError/wrapped_not_found_still_skipped --- PASS: TestIsRetryableFilerError (0.06s) --- PASS: TestIsRetryableFilerError/nil (0.00s) --- PASS: TestIsRetryableFilerError/canceled (0.00s) --- PASS: TestIsRetryableFilerError/unavailable (0.00s) --- PASS: TestIsRetryableFilerError/deadline_exceeded (0.00s) --- PASS: TestIsRetryableFilerError/resource_exhausted (0.00s) --- PASS: TestIsRetryableFilerError/internal (0.00s) --- PASS: TestIsRetryableFilerError/not_found (0.00s) --- PASS: TestIsRetryableFilerError/already_exists (0.00s) --- PASS: TestIsRetryableFilerError/invalid_argument (0.00s) --- PASS: TestIsRetryableFilerError/permission_denied (0.00s) --- PASS: TestIsRetryableFilerError/unauthenticated (0.00s) --- PASS: TestIsRetryableFilerError/failed_precondition (0.00s) --- PASS: TestIsRetryableFilerError/plain_error_retries (0.00s) --- PASS: TestIsRetryableFilerError/wrapped_canceled_still_retries (0.03s) --- PASS: TestIsRetryableFilerError/wrapped_not_found_still_skipped (0.00s) === RUN TestRetryMetadataFlushIfShortCircuitsOnPermanentError --- PASS: TestRetryMetadataFlushIfShortCircuitsOnPermanentError (0.00s) === RUN TestRetryMetadataFlushIfRetriesTransientErrors --- PASS: TestRetryMetadataFlushIfRetriesTransientErrors (0.00s) === RUN TestReadUncachedRemoteEntryDoesNotDeadlock I0824 22:18:26.259395 leveldb_store.go:48 filer store dir: /tmp/TestReadUncachedRemoteEntryDoesNotDeadlock4027119608/001/meta I0824 22:18:26.268218 file_util.go:29 Folder /tmp/TestReadUncachedRemoteEntryDoesNotDeadlock4027119608/001/meta Permission: -rwxr-xr-x --- PASS: TestReadUncachedRemoteEntryDoesNotDeadlock (0.52s) === RUN TestFileHandleFullPathFallsBackAfterForget --- PASS: TestFileHandleFullPathFallsBackAfterForget (0.00s) === RUN TestFileHandleFullPathUsesSavedRenamePathAfterForget --- PASS: TestFileHandleFullPathUsesSavedRenamePathAfterForget (0.00s) === RUN TestIsFilerConfUpdateEventMatchesRenameTarget --- PASS: TestIsFilerConfUpdateEventMatchesRenameTarget (0.00s) === RUN TestDirStateIndexesStayInStep --- PASS: TestDirStateIndexesStayInStep (0.00s) === RUN TestForgetDoesNotDropAReusedDirPath --- PASS: TestForgetDoesNotDropAReusedDirPath (0.00s) === RUN TestInodeEntryStaysInSizeClass --- PASS: TestInodeEntryStaysInSizeClass (0.00s) === RUN TestInodeEntry_removeOnePath === RUN TestInodeEntry_removeOnePath/actual_case === RUN TestInodeEntry_removeOnePath/empty === RUN TestInodeEntry_removeOnePath/single === RUN TestInodeEntry_removeOnePath/first === RUN TestInodeEntry_removeOnePath/middle === RUN TestInodeEntry_removeOnePath/last === RUN TestInodeEntry_removeOnePath/not_found --- PASS: TestInodeEntry_removeOnePath (0.00s) --- PASS: TestInodeEntry_removeOnePath/actual_case (0.00s) --- PASS: TestInodeEntry_removeOnePath/empty (0.00s) --- PASS: TestInodeEntry_removeOnePath/single (0.00s) --- PASS: TestInodeEntry_removeOnePath/first (0.00s) --- PASS: TestInodeEntry_removeOnePath/middle (0.00s) --- PASS: TestInodeEntry_removeOnePath/last (0.00s) --- PASS: TestInodeEntry_removeOnePath/not_found (0.00s) === RUN TestOnlyDirectoriesGetDirState W0824 22:18:26.661970 inode_to_path.go:310 MarkChildrenCached not a tracked directory: /data/file.txt --- PASS: TestOnlyDirectoriesGetDirState (0.01s) === RUN TestMarkChildrenCachedClearsReadThroughMode --- PASS: TestMarkChildrenCachedClearsReadThroughMode (0.00s) === RUN TestForgetOnReleaseRunsUnderLock --- PASS: TestForgetOnReleaseRunsUnderLock (0.00s) === RUN TestMovePathWithNoSourceLeavesTheTarget --- PASS: TestMovePathWithNoSourceLeavesTheTarget (0.00s) === RUN TestRetryMetadataFlushEventuallySucceeds --- PASS: TestRetryMetadataFlushEventuallySucceeds (0.00s) === RUN TestRetryMetadataFlushReturnsLastError --- PASS: TestRetryMetadataFlushReturnsLastError (0.00s) === RUN TestRetryMetadataFlushStopsOnCancel --- PASS: TestRetryMetadataFlushStopsOnCancel (0.00s) === RUN TestWriterPatternSequentialAndHysteresis --- PASS: TestWriterPatternSequentialAndHysteresis (0.00s) === RUN TestWriterPatternFarFirstWriteIsRandom --- PASS: TestWriterPatternFarFirstWriteIsRandom (0.00s) === RUN TestWriterPatternToleranceAbsorbsReorder --- PASS: TestWriterPatternToleranceAbsorbsReorder (0.00s) === RUN TestWriterPatternRandomRecoveryNeedsHysteresis --- PASS: TestWriterPatternRandomRecoveryNeedsHysteresis (0.00s) === RUN TestWriterPatternFrontierNeverRegresses --- PASS: TestWriterPatternFrontierNeverRegresses (0.00s) === RUN TestWriterPatternToleranceBoundary --- PASS: TestWriterPatternToleranceBoundary (0.00s) === RUN TestWriterPatternRecoversToSequential --- PASS: TestWriterPatternRecoversToSequential (0.00s) === RUN TestResolvePeerAdvertiseAddr === RUN TestResolvePeerAdvertiseAddr/explicit_advertise_wins === RUN TestResolvePeerAdvertiseAddr/bind_host_used_verbatim === RUN TestResolvePeerAdvertiseAddr/wildcard_ipv4_bind === RUN TestResolvePeerAdvertiseAddr/empty_host_bind === RUN TestResolvePeerAdvertiseAddr/ipv6_wildcard === RUN TestResolvePeerAdvertiseAddr/unparseable_listen_errors --- PASS: TestResolvePeerAdvertiseAddr (0.07s) --- PASS: TestResolvePeerAdvertiseAddr/explicit_advertise_wins (0.00s) --- PASS: TestResolvePeerAdvertiseAddr/bind_host_used_verbatim (0.00s) --- PASS: TestResolvePeerAdvertiseAddr/wildcard_ipv4_bind (0.02s) --- PASS: TestResolvePeerAdvertiseAddr/empty_host_bind (0.02s) --- PASS: TestResolvePeerAdvertiseAddr/ipv6_wildcard (0.03s) --- PASS: TestResolvePeerAdvertiseAddr/unparseable_listen_errors (0.00s) === RUN TestIsWildcardHost --- PASS: TestIsWildcardHost (0.00s) === RUN TestPeerAnnouncer_FlushGroupsByOwner --- PASS: TestPeerAnnouncer_FlushGroupsByOwner (0.00s) === RUN TestPeerAnnouncer_SkipOwnerEqualsSelf --- PASS: TestPeerAnnouncer_SkipOwnerEqualsSelf (0.00s) === RUN TestPeerAnnouncer_RequeueOnRejection --- PASS: TestPeerAnnouncer_RequeueOnRejection (0.00s) === RUN TestPeerAnnouncer_TTLRenewal --- PASS: TestPeerAnnouncer_TTLRenewal (0.00s) === RUN TestPeerAnnouncer_ReAnnouncesOnOwnerChange --- PASS: TestPeerAnnouncer_ReAnnouncesOnOwnerChange (0.00s) === RUN TestPeerAnnouncer_DropsEvictedFids --- PASS: TestPeerAnnouncer_DropsEvictedFids (0.00s) === RUN TestPeerAnnouncer_StopWaitsForFlush --- PASS: TestPeerAnnouncer_StopWaitsForFlush (0.00s) === RUN TestPeerAnnouncer_SelfOwnedWritesToLocalDir --- PASS: TestPeerAnnouncer_SelfOwnedWritesToLocalDir (0.00s) === RUN TestPeerAnnouncer_DialerErrorRequeues --- PASS: TestPeerAnnouncer_DialerErrorRequeues (0.00s) === RUN TestPeerDirectory_AnnounceAndLookup --- PASS: TestPeerDirectory_AnnounceAndLookup (0.00s) === RUN TestPeerDirectory_OwnerRejection --- PASS: TestPeerDirectory_OwnerRejection (0.00s) === RUN TestPeerDirectory_MultipleHolders --- PASS: TestPeerDirectory_MultipleHolders (0.00s) === RUN TestPeerDirectory_RenewExtendsExpiry --- PASS: TestPeerDirectory_RenewExtendsExpiry (0.00s) === RUN TestPeerDirectory_TTLExpiry --- PASS: TestPeerDirectory_TTLExpiry (0.00s) === RUN TestPeerDirectory_Sweep --- PASS: TestPeerDirectory_Sweep (0.00s) === RUN TestPeerDirectory_LookupOrdersByRecency --- PASS: TestPeerDirectory_LookupOrdersByRecency (0.00s) === RUN TestPeerDirectory_LookupOrderAfterRenewal --- PASS: TestPeerDirectory_LookupOrderAfterRenewal (0.00s) === RUN TestPeerDirectory_StatsCountersIncrement --- PASS: TestPeerDirectory_StatsCountersIncrement (0.00s) === RUN TestPeerDirectory_LookupCapsHolderList --- PASS: TestPeerDirectory_LookupCapsHolderList (0.00s) === RUN TestFetchChunkFromPeer_Hit I0824 22:18:26.761379 peer_grpc.go:88 peer-grpc listening on 127.0.0.1:42257 --- PASS: TestFetchChunkFromPeer_Hit (0.03s) === RUN TestFetchChunkFromPeer_Base64Etag I0824 22:18:26.788852 peer_grpc.go:88 peer-grpc listening on 127.0.0.1:43471 --- PASS: TestFetchChunkFromPeer_Base64Etag (0.02s) === RUN TestFetchChunkFromPeer_EtagMismatch I0824 22:18:26.806934 peer_grpc.go:88 peer-grpc listening on 127.0.0.1:45365 --- PASS: TestFetchChunkFromPeer_EtagMismatch (0.03s) === RUN TestFetchChunkFromPeer_NotFound I0824 22:18:26.835243 peer_grpc.go:88 peer-grpc listening on 127.0.0.1:33067 --- PASS: TestFetchChunkFromPeer_NotFound (0.02s) === RUN TestSortHoldersByLocality --- PASS: TestSortHoldersByLocality (0.00s) === RUN TestSortHoldersByLocality_NoSelfLabels --- PASS: TestSortHoldersByLocality_NoSelfLabels (0.00s) === RUN TestFetchChunkFromPeer_MultiFrameChunkAssembledCorrectly I0824 22:18:26.890981 peer_grpc.go:88 peer-grpc listening on 127.0.0.1:43607 --- PASS: TestFetchChunkFromPeer_MultiFrameChunkAssembledCorrectly (0.22s) === RUN TestPeerGrpcServer_AnnounceAndLookup --- PASS: TestPeerGrpcServer_AnnounceAndLookup (0.07s) === RUN TestPeerGrpcServer_OwnerMismatch --- PASS: TestPeerGrpcServer_OwnerMismatch (0.04s) === RUN TestPeerGrpcServer_FetchChunk_StreamsHit --- PASS: TestPeerGrpcServer_FetchChunk_StreamsHit (0.22s) === RUN TestPeerGrpcServer_FetchChunk_NotFound --- PASS: TestPeerGrpcServer_FetchChunk_NotFound (0.05s) === RUN TestOwnerFor_EmptySeeds --- PASS: TestOwnerFor_EmptySeeds (0.00s) === RUN TestOwnerFor_Deterministic --- PASS: TestOwnerFor_Deterministic (0.00s) === RUN TestOwnerFor_DistributesEvenly --- PASS: TestOwnerFor_DistributesEvenly (0.03s) === RUN TestOwnerFor_MinimalShuffleOnSeedChange --- PASS: TestOwnerFor_MinimalShuffleOnSeedChange (0.02s) === RUN TestOwnerFor_TieBreakerDeterministic --- PASS: TestOwnerFor_TieBreakerDeterministic (0.00s) === RUN TestPeerRegistrar_StartPopulatesSeedsFromFiler --- PASS: TestPeerRegistrar_StartPopulatesSeedsFromFiler (0.00s) === RUN TestPeerRegistrar_HeartbeatTTLMatchesConfig --- PASS: TestPeerRegistrar_HeartbeatTTLMatchesConfig (0.00s) === RUN TestPeerRegistrar_StopIsIdempotent --- PASS: TestPeerRegistrar_StopIsIdempotent (0.00s) === RUN TestPeerRegistrar_RegisterBroadcastsToAllFilers --- PASS: TestPeerRegistrar_RegisterBroadcastsToAllFilers (0.00s) === RUN TestPeerRegistrar_ListMergesAcrossFilers --- PASS: TestPeerRegistrar_ListMergesAcrossFilers (0.00s) === RUN TestPeerRegistrar_ListMergeKeepsNewestLastSeen --- PASS: TestPeerRegistrar_ListMergeKeepsNewestLastSeen (0.00s) === RUN TestNonOverlappingLocksFromDifferentOwners --- PASS: TestNonOverlappingLocksFromDifferentOwners (0.00s) === RUN TestOverlappingReadLocksFromDifferentOwners --- PASS: TestOverlappingReadLocksFromDifferentOwners (0.00s) === RUN TestOverlappingWriteReadConflict --- PASS: TestOverlappingWriteReadConflict (0.00s) === RUN TestOverlappingWriteWriteConflict --- PASS: TestOverlappingWriteWriteConflict (0.00s) === RUN TestSameOwnerUpgradeReadToWrite --- PASS: TestSameOwnerUpgradeReadToWrite (0.00s) === RUN TestSameOwnerDowngradeWriteToRead --- PASS: TestSameOwnerDowngradeWriteToRead (0.00s) === RUN TestLockCoalescing --- PASS: TestLockCoalescing (0.00s) === RUN TestLockSplitting --- PASS: TestLockSplitting (0.00s) === RUN TestGetLkConflict --- PASS: TestGetLkConflict (0.00s) === RUN TestGetLkNoConflict --- PASS: TestGetLkNoConflict (0.00s) === RUN TestGetLkSameOwnerNoConflict --- PASS: TestGetLkSameOwnerNoConflict (0.00s) === RUN TestReleaseOwner --- PASS: TestReleaseOwner (0.00s) === RUN TestDifferentLockKindsDoNotConflict --- PASS: TestDifferentLockKindsDoNotConflict (0.00s) === RUN TestReleasePosixOwnerReleasesPosixLocksAndWakesWaiters --- PASS: TestReleasePosixOwnerReleasesPosixLocksAndWakesWaiters (0.05s) === RUN TestReleasePosixOwnerDoesNotReleaseFlockLocks --- PASS: TestReleasePosixOwnerDoesNotReleaseFlockLocks (0.00s) === RUN TestHasPosixOwnerIgnoresMissingOwnerAndFlock --- PASS: TestHasPosixOwnerIgnoresMissingOwnerAndFlock (0.00s) === RUN TestWakeEligibleWaitersKeepsInodeUntilWakeRefReleased --- PASS: TestWakeEligibleWaitersKeepsInodeUntilWakeRefReleased (0.00s) === RUN TestReleaseFlockOwnerDoesNotReleasePosixLocks --- PASS: TestReleaseFlockOwnerDoesNotReleasePosixLocks (0.00s) === RUN TestReleaseOwnerWakesWaiters --- PASS: TestReleaseOwnerWakesWaiters (0.06s) === RUN TestSetLkwBlocksAndSucceeds --- PASS: TestSetLkwBlocksAndSucceeds (0.06s) === RUN TestSetLkwCancellation --- PASS: TestSetLkwCancellation (0.06s) === RUN TestWholeFileLock --- PASS: TestWholeFileLock (0.00s) === RUN TestUnlockNoExistingLocks --- PASS: TestUnlockNoExistingLocks (0.00s) === RUN TestMultipleInodesIndependent --- PASS: TestMultipleInodesIndependent (0.00s) === RUN TestMemoryCleanup --- PASS: TestMemoryCleanup (0.00s) === RUN TestSelectiveWaking --- PASS: TestSelectiveWaking (0.17s) === RUN TestSameOwnerReplaceDifferentType --- PASS: TestSameOwnerReplaceDifferentType (0.00s) === RUN TestNonAdjacentRangesNotCoalesced --- PASS: TestNonAdjacentRangesNotCoalesced (0.00s) === RUN TestAdjacencyNoOverflowAtMaxUint64 --- PASS: TestAdjacencyNoOverflowAtMaxUint64 (0.00s) === RUN TestSetLkRetriesPastDeadInodeLocks --- PASS: TestSetLkRetriesPastDeadInodeLocks (0.00s) === RUN TestGetInodeLocksEvictsDeadEntry --- PASS: TestGetInodeLocksEvictsDeadEntry (0.00s) === RUN TestConcurrentFlockChurnPreservesMutualExclusion --- PASS: TestConcurrentFlockChurnPreservesMutualExclusion (1.53s) === RUN TestPermissionCheckPerformance === Permission Check Performance Test (WITH CACHE) === Files simulated: 10000 Checks per file: 5 Total checks: 50000 System lookups: 1 (would be 50000 without cache) Lookups eliminated: 49999 (100.0% reduction) Time elapsed: 174.693659ms Throughput: 286215 checks/sec --- PASS: TestPermissionCheckPerformance (0.17s) === RUN TestAttrChunkRace --- PASS: TestAttrChunkRace (0.14s) === RUN TestReadFromChunksRace --- PASS: TestReadFromChunksRace (0.05s) === RUN TestOpenDirEnablesKernelListingCache I0824 22:18:29.832807 leveldb_store.go:48 filer store dir: /tmp/TestOpenDirEnablesKernelListingCache2202849870/001/meta I0824 22:18:29.833070 file_util.go:29 Folder /tmp/TestOpenDirEnablesKernelListingCache2202849870/001/meta Permission: -rwxr-xr-x --- PASS: TestOpenDirEnablesKernelListingCache (0.02s) === RUN TestListDirectoryEntriesOmitsChunks I0824 22:18:29.852966 leveldb_store.go:48 filer store dir: /tmp/TestListDirectoryEntriesOmitsChunks3284156839/001/meta I0824 22:18:29.853153 file_util.go:29 Folder /tmp/TestListDirectoryEntriesOmitsChunks3284156839/001/meta Permission: -rwxr-xr-x === RUN TestListDirectoryEntriesOmitsChunks/plain_listing_keeps_chunks === RUN TestListDirectoryEntriesOmitsChunks/marked_listing_drops_chunks --- PASS: TestListDirectoryEntriesOmitsChunks (0.18s) --- PASS: TestListDirectoryEntriesOmitsChunks/plain_listing_keeps_chunks (0.00s) --- PASS: TestListDirectoryEntriesOmitsChunks/marked_listing_drops_chunks (0.00s) === RUN TestReadDirResumePastShrunkDirectory I0824 22:18:30.032690 leveldb_store.go:48 filer store dir: /tmp/TestReadDirResumePastShrunkDirectory202341581/001/meta I0824 22:18:30.032971 file_util.go:29 Folder /tmp/TestReadDirResumePastShrunkDirectory202341581/001/meta Permission: -rwxr-xr-x --- PASS: TestReadDirResumePastShrunkDirectory (0.39s) === RUN TestReadDirWithExpiredEntries I0824 22:18:30.432348 leveldb_store.go:48 filer store dir: /tmp/TestReadDirWithExpiredEntries388535543/001/meta I0824 22:18:30.432948 file_util.go:29 Folder /tmp/TestReadDirWithExpiredEntries388535543/001/meta Permission: -rwxr-xr-x --- PASS: TestReadDirWithExpiredEntries (0.29s) === RUN TestReadDirTrimsConsumedEntries I0824 22:18:30.719166 leveldb_store.go:48 filer store dir: /tmp/TestReadDirTrimsConsumedEntries2398245339/001/meta I0824 22:18:30.719449 file_util.go:29 Folder /tmp/TestReadDirTrimsConsumedEntries2398245339/001/meta Permission: -rwxr-xr-x --- PASS: TestReadDirTrimsConsumedEntries (1.08s) === RUN TestReadDirDirectTrimsConsumedEntries I0824 22:18:31.798778 leveldb_store.go:48 filer store dir: /tmp/TestReadDirDirectTrimsConsumedEntries1703723842/001/meta I0824 22:18:31.799049 file_util.go:29 Folder /tmp/TestReadDirDirectTrimsConsumedEntries1703723842/001/meta Permission: -rwxr-xr-x --- PASS: TestReadDirDirectTrimsConsumedEntries (5.22s) === RUN TestReadDirPlusSanitizedName I0824 22:18:37.014884 leveldb_store.go:48 filer store dir: /tmp/TestReadDirPlusSanitizedName3196742758/001/meta I0824 22:18:37.016216 file_util.go:29 Folder /tmp/TestReadDirPlusSanitizedName3196742758/001/meta Permission: -rwxr-xr-x --- PASS: TestReadDirPlusSanitizedName (0.04s) === RUN TestLoadDirectoryEntriesDirectFiltersHiddenEntriesAndMapsIds --- PASS: TestLoadDirectoryEntriesDirectFiltersHiddenEntriesAndMapsIds (0.00s) === RUN TestLoadDirectoryEntriesDirectShowsSystemEntriesWhenEnabled --- PASS: TestLoadDirectoryEntriesDirectShowsSystemEntriesWhenEnabled (0.00s) === RUN TestWholeFileServerCopyCandidate --- PASS: TestWholeFileServerCopyCandidate (0.00s) === RUN TestCopyFileRangeUsesServerSideWholeFileCopy --- PASS: TestCopyFileRangeUsesServerSideWholeFileCopy (0.00s) === RUN TestCopyFileRangeDoesNotFallbackAfterCommittedServerCopyRefreshFailure I0824 22:18:37.088966 leveldb_store.go:48 filer store dir: /tmp/TestCopyFileRangeDoesNotFallbackAfterCommittedServerCopyRefreshF1677973738/001/meta I0824 22:18:37.089631 file_util.go:29 Folder /tmp/TestCopyFileRangeDoesNotFallbackAfterCommittedServerCopyRefreshF1677973738/001/meta Permission: -rwxr-xr-x W0824 22:18:37.112096 weedfs_file_copy_range.go:223 CopyFileRange server-side copy /src.txt => /dst.txt committed but local refresh failed: reload copied entry: transient filer read failure --- PASS: TestCopyFileRangeDoesNotFallbackAfterCommittedServerCopyRefreshFailure (0.03s) === RUN TestCopyFileRangeReturnsEIOForAmbiguousServerSideCopy W0824 22:18:37.120817 weedfs_file_copy_range.go:230 CopyFileRange server-side copy /src.txt => /dst.txt outcome ambiguous: transport timeout after request dispatch --- PASS: TestCopyFileRangeReturnsEIOForAmbiguousServerSideCopy (0.00s) === RUN TestOpenKeepCache_FirstOpen --- PASS: TestOpenKeepCache_FirstOpen (0.00s) === RUN TestOpenKeepCache_SecondOpenSameMtime --- PASS: TestOpenKeepCache_SecondOpenSameMtime (0.00s) === RUN TestOpenKeepCache_MtimeChanged --- PASS: TestOpenKeepCache_MtimeChanged (0.00s) === RUN TestOpenKeepCache_NanosecondPrecision --- PASS: TestOpenKeepCache_NanosecondPrecision (0.00s) === RUN TestOpenKeepCache_WriteInvalidation --- PASS: TestOpenKeepCache_WriteInvalidation (0.00s) === RUN TestOpenKeepCache_WriteOpenSkipped --- PASS: TestOpenKeepCache_WriteOpenSkipped (0.00s) === RUN TestOpenKeepCache_BoundedEviction --- PASS: TestOpenKeepCache_BoundedEviction (0.00s) === RUN TestCreateCreatesAndOpensFile I0824 22:18:37.133269 leveldb_store.go:48 filer store dir: /tmp/TestCreateCreatesAndOpensFile4219404840/002/meta I0824 22:18:37.133528 file_util.go:29 Folder /tmp/TestCreateCreatesAndOpensFile4219404840/002/meta Permission: -rwxr-xr-x W0824 22:18:37.167444 weedfs_access.go:27 hasAccess: user.LookupId for uid 123 failed: user: unknown userid 123 --- PASS: TestCreateCreatesAndOpensFile (0.06s) === RUN TestReleaseFlushesDirtyCreateIfFlushWasSkipped I0824 22:18:37.198642 leveldb_store.go:48 filer store dir: /tmp/TestReleaseFlushesDirtyCreateIfFlushWasSkipped3447290485/002/meta I0824 22:18:37.198986 file_util.go:29 Folder /tmp/TestReleaseFlushesDirtyCreateIfFlushWasSkipped3447290485/002/meta Permission: -rwxr-xr-x W0824 22:18:37.214481 weedfs_access.go:27 hasAccess: user.LookupId for uid 123 failed: user: unknown userid 123 --- PASS: TestReleaseFlushesDirtyCreateIfFlushWasSkipped (0.05s) === RUN TestTruncateEntryClearsDirtyPagesForOpenHandle I0824 22:18:37.245161 leveldb_store.go:48 filer store dir: /tmp/TestTruncateEntryClearsDirtyPagesForOpenHandle745220365/002/meta I0824 22:18:37.245485 file_util.go:29 Folder /tmp/TestTruncateEntryClearsDirtyPagesForOpenHandle745220365/002/meta Permission: -rwxr-xr-x --- PASS: TestTruncateEntryClearsDirtyPagesForOpenHandle (0.10s) === RUN TestAccessChecksPermissions --- PASS: TestAccessChecksPermissions (0.00s) === RUN TestHasAccessUsesSupplementaryGroups --- PASS: TestHasAccessUsesSupplementaryGroups (0.00s) === RUN TestSupplementaryGroupCaching --- PASS: TestSupplementaryGroupCaching (0.00s) === RUN TestSupplementaryGroupCacheExpiry --- PASS: TestSupplementaryGroupCacheExpiry (0.00s) === RUN TestCreateExistingFileIgnoresQuotaPreflight I0824 22:18:37.347069 leveldb_store.go:48 filer store dir: /tmp/TestCreateExistingFileIgnoresQuotaPreflight2224389417/002/meta I0824 22:18:37.347527 file_util.go:29 Folder /tmp/TestCreateExistingFileIgnoresQuotaPreflight2224389417/002/meta Permission: -rwxr-xr-x I0824 22:18:37.400044 weedfs_file_mkrm.go:49 Create O_EXCL /existing.txt: already exists (uid=123 gid=456 mode=644) --- PASS: TestCreateExistingFileIgnoresQuotaPreflight (0.06s) === RUN TestAcquireHandleHonorsDefaultPermissions === RUN TestAcquireHandleHonorsDefaultPermissions/kernel_enforces I0824 22:18:37.407707 leveldb_store.go:48 filer store dir: /tmp/TestAcquireHandleHonorsDefaultPermissionskernel_enforces2148807028/002/meta I0824 22:18:37.407903 file_util.go:29 Folder /tmp/TestAcquireHandleHonorsDefaultPermissionskernel_enforces2148807028/002/meta Permission: -rwxr-xr-x === RUN TestAcquireHandleHonorsDefaultPermissions/mount_enforces I0824 22:18:37.440540 leveldb_store.go:48 filer store dir: /tmp/TestAcquireHandleHonorsDefaultPermissionsmount_enforces2421063349/002/meta I0824 22:18:37.440814 file_util.go:29 Folder /tmp/TestAcquireHandleHonorsDefaultPermissionsmount_enforces2421063349/002/meta Permission: -rwxr-xr-x --- PASS: TestAcquireHandleHonorsDefaultPermissions (0.05s) --- PASS: TestAcquireHandleHonorsDefaultPermissions/kernel_enforces (0.03s) --- PASS: TestAcquireHandleHonorsDefaultPermissions/mount_enforces (0.02s) === RUN TestCreateRegularFileValidatesParentUnderDefaultPermissions I0824 22:18:37.456773 leveldb_store.go:48 filer store dir: /tmp/TestCreateRegularFileValidatesParentUnderDefaultPermissions3164992135/002/meta I0824 22:18:37.457064 file_util.go:29 Folder /tmp/TestCreateRegularFileValidatesParentUnderDefaultPermissions3164992135/002/meta Permission: -rwxr-xr-x --- PASS: TestCreateRegularFileValidatesParentUnderDefaultPermissions (0.04s) === RUN TestShouldMergeChunks_NoChunks --- PASS: TestShouldMergeChunks_NoChunks (0.00s) === RUN TestShouldMergeChunks_SingleChunk --- PASS: TestShouldMergeChunks_SingleChunk (0.00s) === RUN TestShouldMergeChunks_NonOverlapping --- PASS: TestShouldMergeChunks_NonOverlapping (0.00s) === RUN TestShouldMergeChunks_ExactlyDouble --- PASS: TestShouldMergeChunks_ExactlyDouble (0.00s) === RUN TestShouldMergeChunks_JustOverDouble --- PASS: TestShouldMergeChunks_JustOverDouble (0.00s) === RUN TestShouldMergeChunks_ManifestExtendFileSize --- PASS: TestShouldMergeChunks_ManifestExtendFileSize (0.00s) === RUN TestShouldMergeChunks_ManifestSizesCounted --- PASS: TestShouldMergeChunks_ManifestSizesCounted (0.00s) === RUN TestShouldMergeChunks_AccumulatedManifests --- PASS: TestShouldMergeChunks_AccumulatedManifests (0.00s) === RUN TestShouldMergeChunks_DoesNotMutateInput --- PASS: TestShouldMergeChunks_DoesNotMutateInput (0.00s) === RUN TestCompactThenMergeCondition === RUN TestCompactThenMergeCondition/fully_superseded_chunks_removed,_no_bloat === RUN TestCompactThenMergeCondition/repeated_full_overwrites_all_compacted_away === RUN TestCompactThenMergeCondition/non-overlapping_small_chunks,_no_bloat === RUN TestCompactThenMergeCondition/staggered_partial_overlaps_create_bloat === RUN TestCompactThenMergeCondition/overlapping_writes_with_75pct_overlap === RUN TestCompactThenMergeCondition/many_4K_writes_at_1K_step_over_large_range --- PASS: TestCompactThenMergeCondition (0.01s) --- PASS: TestCompactThenMergeCondition/fully_superseded_chunks_removed,_no_bloat (0.00s) --- PASS: TestCompactThenMergeCondition/repeated_full_overwrites_all_compacted_away (0.00s) --- PASS: TestCompactThenMergeCondition/non-overlapping_small_chunks,_no_bloat (0.00s) --- PASS: TestCompactThenMergeCondition/staggered_partial_overlaps_create_bloat (0.00s) --- PASS: TestCompactThenMergeCondition/overlapping_writes_with_75pct_overlap (0.00s) --- PASS: TestCompactThenMergeCondition/many_4K_writes_at_1K_step_over_large_range (0.00s) === RUN TestRandomWritesBloatDetection --- PASS: TestRandomWritesBloatDetection (0.01s) === RUN TestFlushCycleManifestAccumulation weedfs_file_sync_test.go:344: cycle 2: merge correctly triggered (4 manifests, 10 regular chunks) --- PASS: TestFlushCycleManifestAccumulation (0.00s) === RUN TestVisibleContentPreservedAfterCompact --- PASS: TestVisibleContentPreservedAfterCompact (0.04s) === RUN TestUpdateEventRefreshesOpenFileHandle I0824 22:18:37.552114 leveldb_store.go:48 filer store dir: /tmp/TestUpdateEventRefreshesOpenFileHandle2860158661/001/meta I0824 22:18:37.552378 file_util.go:29 Folder /tmp/TestUpdateEventRefreshesOpenFileHandle2860158661/001/meta Permission: -rwxr-xr-x --- PASS: TestUpdateEventRefreshesOpenFileHandle (0.02s) === RUN TestQueuedEventDoesNotRollBackNewerLocalState I0824 22:18:37.571995 leveldb_store.go:48 filer store dir: /tmp/TestQueuedEventDoesNotRollBackNewerLocalState294336018/001/meta I0824 22:18:37.572321 file_util.go:29 Folder /tmp/TestQueuedEventDoesNotRollBackNewerLocalState294336018/001/meta Permission: -rwxr-xr-x --- PASS: TestQueuedEventDoesNotRollBackNewerLocalState (0.03s) === RUN TestBufferedBuildEventReinvalidatesOnCompletion I0824 22:18:37.604569 leveldb_store.go:48 filer store dir: /tmp/TestBufferedBuildEventReinvalidatesOnCompletion1198503813/001/meta I0824 22:18:37.604911 file_util.go:29 Folder /tmp/TestBufferedBuildEventReinvalidatesOnCompletion1198503813/001/meta Permission: -rwxr-xr-x --- PASS: TestBufferedBuildEventReinvalidatesOnCompletion (0.02s) === RUN TestUncachedDirStaleStoreEntryDoesNotMaskEvent I0824 22:18:37.626345 leveldb_store.go:48 filer store dir: /tmp/TestUncachedDirStaleStoreEntryDoesNotMaskEvent1503203563/001/meta I0824 22:18:37.626623 file_util.go:29 Folder /tmp/TestUncachedDirStaleStoreEntryDoesNotMaskEvent1503203563/001/meta Permission: -rwxr-xr-x --- PASS: TestUncachedDirStaleStoreEntryDoesNotMaskEvent (0.03s) === RUN TestQueuedEventOlderThanFlushedStateIsIgnored I0824 22:18:37.661700 leveldb_store.go:48 filer store dir: /tmp/TestQueuedEventOlderThanFlushedStateIsIgnored1451246101/001/meta I0824 22:18:37.662028 file_util.go:29 Folder /tmp/TestQueuedEventOlderThanFlushedStateIsIgnored1451246101/001/meta Permission: -rwxr-xr-x --- PASS: TestQueuedEventOlderThanFlushedStateIsIgnored (0.01s) === RUN TestSaveEntryKeepsOpenHandleAheadOfOlderEvents I0824 22:18:37.674106 leveldb_store.go:48 filer store dir: /tmp/TestSaveEntryKeepsOpenHandleAheadOfOlderEvents2397031426/001/meta I0824 22:18:37.674385 file_util.go:29 Folder /tmp/TestSaveEntryKeepsOpenHandleAheadOfOlderEvents2397031426/001/meta Permission: -rwxr-xr-x --- PASS: TestSaveEntryKeepsOpenHandleAheadOfOlderEvents (0.06s) === RUN TestQueuedEventDoesNotRollBackHandleOpenedAfterEnqueue I0824 22:18:37.733609 leveldb_store.go:48 filer store dir: /tmp/TestQueuedEventDoesNotRollBackHandleOpenedAfterEnqueue1018138680/001/meta I0824 22:18:37.734035 file_util.go:29 Folder /tmp/TestQueuedEventDoesNotRollBackHandleOpenedAfterEnqueue1018138680/001/meta Permission: -rwxr-xr-x --- PASS: TestQueuedEventDoesNotRollBackHandleOpenedAfterEnqueue (0.05s) === RUN TestAbortedBuildEventDoesNotRollBackLaterOpen I0824 22:18:37.778816 leveldb_store.go:48 filer store dir: /tmp/TestAbortedBuildEventDoesNotRollBackLaterOpen3884939136/001/meta I0824 22:18:37.779068 file_util.go:29 Folder /tmp/TestAbortedBuildEventDoesNotRollBackLaterOpen3884939136/001/meta Permission: -rwxr-xr-x --- PASS: TestAbortedBuildEventDoesNotRollBackLaterOpen (0.06s) === RUN TestEventDuringOpenLookupIsFenced I0824 22:18:37.841039 leveldb_store.go:48 filer store dir: /tmp/TestEventDuringOpenLookupIsFenced2144035598/001/meta I0824 22:18:37.841663 file_util.go:29 Folder /tmp/TestEventDuringOpenLookupIsFenced2144035598/001/meta Permission: -rwxr-xr-x --- PASS: TestEventDuringOpenLookupIsFenced (0.04s) === RUN TestRemoteCacheResponseVersionFencesUndeliveredEvents I0824 22:18:37.885765 leveldb_store.go:48 filer store dir: /tmp/TestRemoteCacheResponseVersionFencesUndeliveredEvents3661207883/001/meta I0824 22:18:37.885893 file_util.go:29 Folder /tmp/TestRemoteCacheResponseVersionFencesUndeliveredEvents3661207883/001/meta Permission: -rwxr-xr-x --- PASS: TestRemoteCacheResponseVersionFencesUndeliveredEvents (0.06s) === RUN TestEventlessSaveAckStillFencesOlderEvents I0824 22:18:37.946335 leveldb_store.go:48 filer store dir: /tmp/TestEventlessSaveAckStillFencesOlderEvents192375880/001/meta I0824 22:18:37.946577 file_util.go:29 Folder /tmp/TestEventlessSaveAckStillFencesOlderEvents192375880/001/meta Permission: -rwxr-xr-x --- PASS: TestEventlessSaveAckStillFencesOlderEvents (0.03s) === RUN TestLocalAckDoesNotFenceUnrelatedDelayedEvents I0824 22:18:37.976352 leveldb_store.go:48 filer store dir: /tmp/TestLocalAckDoesNotFenceUnrelatedDelayedEvents3152066634/001/meta I0824 22:18:37.976677 file_util.go:29 Folder /tmp/TestLocalAckDoesNotFenceUnrelatedDelayedEvents3152066634/001/meta Permission: -rwxr-xr-x --- PASS: TestLocalAckDoesNotFenceUnrelatedDelayedEvents (0.05s) === RUN TestSlowerConcurrentOpenDoesNotOverwriteNewerHandle I0824 22:18:38.026458 leveldb_store.go:48 filer store dir: /tmp/TestSlowerConcurrentOpenDoesNotOverwriteNewerHandle670029240/001/meta I0824 22:18:38.026850 file_util.go:29 Folder /tmp/TestSlowerConcurrentOpenDoesNotOverwriteNewerHandle670029240/001/meta Permission: -rwxr-xr-x --- PASS: TestSlowerConcurrentOpenDoesNotOverwriteNewerHandle (0.11s) === RUN TestFloorProtectsSnapshotStateFromDelayedEvents I0824 22:18:38.144709 leveldb_store.go:48 filer store dir: /tmp/TestFloorProtectsSnapshotStateFromDelayedEvents1569706421/001/meta I0824 22:18:38.145451 file_util.go:29 Folder /tmp/TestFloorProtectsSnapshotStateFromDelayedEvents1569706421/001/meta Permission: -rwxr-xr-x --- PASS: TestFloorProtectsSnapshotStateFromDelayedEvents (0.05s) === RUN TestAlreadyReflectedEventDoesNotDestroyDirtyPages I0824 22:18:38.189673 leveldb_store.go:48 filer store dir: /tmp/TestAlreadyReflectedEventDoesNotDestroyDirtyPages1600187892/001/meta I0824 22:18:38.190212 file_util.go:29 Folder /tmp/TestAlreadyReflectedEventDoesNotDestroyDirtyPages1600187892/001/meta Permission: -rwxr-xr-x --- PASS: TestAlreadyReflectedEventDoesNotDestroyDirtyPages (0.08s) === RUN TestSlowerOpenRejectedWhenHandleDirtyOrLookupUnversioned I0824 22:18:38.274136 leveldb_store.go:48 filer store dir: /tmp/TestSlowerOpenRejectedWhenHandleDirtyOrLookupUnversioned1737204651/001/meta I0824 22:18:38.274784 file_util.go:29 Folder /tmp/TestSlowerOpenRejectedWhenHandleDirtyOrLookupUnversioned1737204651/001/meta Permission: -rwxr-xr-x --- PASS: TestSlowerOpenRejectedWhenHandleDirtyOrLookupUnversioned (0.25s) === RUN TestUnversionedSlowerOpenDoesNotOverwrite I0824 22:18:38.528105 leveldb_store.go:48 filer store dir: /tmp/TestUnversionedSlowerOpenDoesNotOverwrite279364711/001/meta I0824 22:18:38.528443 file_util.go:29 Folder /tmp/TestUnversionedSlowerOpenDoesNotOverwrite279364711/001/meta Permission: -rwxr-xr-x --- PASS: TestUnversionedSlowerOpenDoesNotOverwrite (0.22s) === RUN TestAlreadyReflectedEventPreservesDirtyWrites I0824 22:18:38.759296 leveldb_store.go:48 filer store dir: /tmp/TestAlreadyReflectedEventPreservesDirtyWrites2506597607/001/meta I0824 22:18:38.759784 file_util.go:29 Folder /tmp/TestAlreadyReflectedEventPreservesDirtyWrites2506597607/001/meta Permission: -rwxr-xr-x --- PASS: TestAlreadyReflectedEventPreservesDirtyWrites (0.06s) === RUN TestDeleteTombstoneBlocksResurrection I0824 22:18:38.816220 leveldb_store.go:48 filer store dir: /tmp/TestDeleteTombstoneBlocksResurrection1637850317/001/meta I0824 22:18:38.816492 file_util.go:29 Folder /tmp/TestDeleteTombstoneBlocksResurrection1637850317/001/meta Permission: -rwxr-xr-x --- PASS: TestDeleteTombstoneBlocksResurrection (0.05s) === RUN TestServerSideCopyInstallEnrollsInBase I0824 22:18:38.882642 leveldb_store.go:48 filer store dir: /tmp/TestServerSideCopyInstallEnrollsInBase4228158714/001/meta I0824 22:18:38.910216 file_util.go:29 Folder /tmp/TestServerSideCopyInstallEnrollsInBase4228158714/001/meta Permission: -rwxr-xr-x --- PASS: TestServerSideCopyInstallEnrollsInBase (0.12s) === RUN TestAbsenceFloorBlocksGhostCreate I0824 22:18:38.991592 leveldb_store.go:48 filer store dir: /tmp/TestAbsenceFloorBlocksGhostCreate3153620051/001/meta I0824 22:18:38.992034 file_util.go:29 Folder /tmp/TestAbsenceFloorBlocksGhostCreate3153620051/001/meta Permission: -rwxr-xr-x --- PASS: TestAbsenceFloorBlocksGhostCreate (0.03s) === RUN TestVersionedDeleteOfMissingEntryLeavesTombstone I0824 22:18:39.022633 leveldb_store.go:48 filer store dir: /tmp/TestVersionedDeleteOfMissingEntryLeavesTombstone1848690826/001/meta I0824 22:18:39.023017 file_util.go:29 Folder /tmp/TestVersionedDeleteOfMissingEntryLeavesTombstone1848690826/001/meta Permission: -rwxr-xr-x --- PASS: TestVersionedDeleteOfMissingEntryLeavesTombstone (0.07s) === RUN TestCommittedCopyWithFailedReadbackAdoptsItsEvent I0824 22:18:39.093369 leveldb_store.go:48 filer store dir: /tmp/TestCommittedCopyWithFailedReadbackAdoptsItsEvent3440742687/001/meta I0824 22:18:39.093843 file_util.go:29 Folder /tmp/TestCommittedCopyWithFailedReadbackAdoptsItsEvent3440742687/001/meta Permission: -rwxr-xr-x --- PASS: TestCommittedCopyWithFailedReadbackAdoptsItsEvent (0.03s) === RUN TestNewerAbsenceFloorOverridesOlderTombstone I0824 22:18:39.122308 leveldb_store.go:48 filer store dir: /tmp/TestNewerAbsenceFloorOverridesOlderTombstone2109764495/001/meta I0824 22:18:39.122593 file_util.go:29 Folder /tmp/TestNewerAbsenceFloorOverridesOlderTombstone2109764495/001/meta Permission: -rwxr-xr-x --- PASS: TestNewerAbsenceFloorOverridesOlderTombstone (0.05s) === RUN TestFlushAckCancelsPendingCopyEventAdoption I0824 22:18:39.171209 leveldb_store.go:48 filer store dir: /tmp/TestFlushAckCancelsPendingCopyEventAdoption3722656645/001/meta I0824 22:18:39.171559 file_util.go:29 Folder /tmp/TestFlushAckCancelsPendingCopyEventAdoption3722656645/001/meta Permission: -rwxr-xr-x --- PASS: TestFlushAckCancelsPendingCopyEventAdoption (0.11s) === RUN TestAckedSaveInstallsIntoRacingOpenHandle I0824 22:18:39.286919 leveldb_store.go:48 filer store dir: /tmp/TestAckedSaveInstallsIntoRacingOpenHandle917165651/001/meta I0824 22:18:39.287252 file_util.go:29 Folder /tmp/TestAckedSaveInstallsIntoRacingOpenHandle917165651/001/meta Permission: -rwxr-xr-x --- PASS: TestAckedSaveInstallsIntoRacingOpenHandle (0.28s) === RUN TestEmptyListingTrailerSnapshotSetsAbsenceFloor I0824 22:18:39.567365 leveldb_store.go:48 filer store dir: /tmp/TestEmptyListingTrailerSnapshotSetsAbsenceFloor1280853617/001/meta I0824 22:18:39.567807 file_util.go:29 Folder /tmp/TestEmptyListingTrailerSnapshotSetsAbsenceFloor1280853617/001/meta Permission: -rwxr-xr-x --- PASS: TestEmptyListingTrailerSnapshotSetsAbsenceFloor (0.15s) === RUN TestVacateEventPreservesDirtyPages I0824 22:18:39.718699 leveldb_store.go:48 filer store dir: /tmp/TestVacateEventPreservesDirtyPages1544340490/001/meta I0824 22:18:39.718838 file_util.go:29 Folder /tmp/TestVacateEventPreservesDirtyPages1544340490/001/meta Permission: -rwxr-xr-x --- PASS: TestVacateEventPreservesDirtyPages (0.03s) === RUN TestCopyAdoptRejectsForeignEvent I0824 22:18:39.747671 leveldb_store.go:48 filer store dir: /tmp/TestCopyAdoptRejectsForeignEvent172394607/001/meta I0824 22:18:39.747890 file_util.go:29 Folder /tmp/TestCopyAdoptRejectsForeignEvent172394607/001/meta Permission: -rwxr-xr-x --- PASS: TestCopyAdoptRejectsForeignEvent (0.05s) === RUN TestRemoteDownloadBaseMappedToLocal I0824 22:18:39.799086 leveldb_store.go:48 filer store dir: /tmp/TestRemoteDownloadBaseMappedToLocal1520954365/001/meta I0824 22:18:39.799307 file_util.go:29 Folder /tmp/TestRemoteDownloadBaseMappedToLocal1520954365/001/meta Permission: -rwxr-xr-x --- PASS: TestRemoteDownloadBaseMappedToLocal (0.04s) === RUN TestForeignDeleteMarksHandleDeleted I0824 22:18:39.842064 leveldb_store.go:48 filer store dir: /tmp/TestForeignDeleteMarksHandleDeleted609919784/001/meta I0824 22:18:39.842289 file_util.go:29 Folder /tmp/TestForeignDeleteMarksHandleDeleted609919784/001/meta Permission: -rwxr-xr-x --- PASS: TestForeignDeleteMarksHandleDeleted (0.02s) === RUN TestEventlessSaveVersionsCacheEntry I0824 22:18:39.859088 leveldb_store.go:48 filer store dir: /tmp/TestEventlessSaveVersionsCacheEntry2279658862/001/meta I0824 22:18:39.859284 file_util.go:29 Folder /tmp/TestEventlessSaveVersionsCacheEntry2279658862/001/meta Permission: -rwxr-xr-x --- PASS: TestEventlessSaveVersionsCacheEntry (0.05s) === RUN TestStaleRemoteDownloadDoesNotRollBack I0824 22:18:39.911616 leveldb_store.go:48 filer store dir: /tmp/TestStaleRemoteDownloadDoesNotRollBack3525987274/001/meta I0824 22:18:39.912162 file_util.go:29 Folder /tmp/TestStaleRemoteDownloadDoesNotRollBack3525987274/001/meta Permission: -rwxr-xr-x --- PASS: TestStaleRemoteDownloadDoesNotRollBack (0.04s) === RUN TestRemoteOnlyHandleTakesUnversionedDownload I0824 22:18:39.949580 leveldb_store.go:48 filer store dir: /tmp/TestRemoteOnlyHandleTakesUnversionedDownload1994921306/001/meta I0824 22:18:39.949844 file_util.go:29 Folder /tmp/TestRemoteOnlyHandleTakesUnversionedDownload1994921306/001/meta Permission: -rwxr-xr-x --- PASS: TestRemoteOnlyHandleTakesUnversionedDownload (0.07s) === RUN TestCopyAdoptRejectsForeignMetadataChange I0824 22:18:40.018615 leveldb_store.go:48 filer store dir: /tmp/TestCopyAdoptRejectsForeignMetadataChange363714289/001/meta I0824 22:18:40.018853 file_util.go:29 Folder /tmp/TestCopyAdoptRejectsForeignMetadataChange363714289/001/meta Permission: -rwxr-xr-x --- PASS: TestCopyAdoptRejectsForeignMetadataChange (0.01s) === RUN TestForeignRenameDoesNotMarkHandleDeleted I0824 22:18:40.032709 leveldb_store.go:48 filer store dir: /tmp/TestForeignRenameDoesNotMarkHandleDeleted1023845957/001/meta I0824 22:18:40.032935 file_util.go:29 Folder /tmp/TestForeignRenameDoesNotMarkHandleDeleted1023845957/001/meta Permission: -rwxr-xr-x --- PASS: TestForeignRenameDoesNotMarkHandleDeleted (0.04s) === RUN TestCopyAdoptAppliesForeignTouchToCleanHandle I0824 22:18:40.076058 leveldb_store.go:48 filer store dir: /tmp/TestCopyAdoptAppliesForeignTouchToCleanHandle1006703525/001/meta I0824 22:18:40.076346 file_util.go:29 Folder /tmp/TestCopyAdoptAppliesForeignTouchToCleanHandle1006703525/001/meta Permission: -rwxr-xr-x --- PASS: TestCopyAdoptAppliesForeignTouchToCleanHandle (0.02s) === RUN TestUnversionedRemoteDownloadDoesNotOverwriteVersioned I0824 22:18:40.095995 leveldb_store.go:48 filer store dir: /tmp/TestUnversionedRemoteDownloadDoesNotOverwriteVersioned477010629/001/meta I0824 22:18:40.096265 file_util.go:29 Folder /tmp/TestUnversionedRemoteDownloadDoesNotOverwriteVersioned477010629/001/meta Permission: -rwxr-xr-x --- PASS: TestUnversionedRemoteDownloadDoesNotOverwriteVersioned (0.05s) === RUN TestEventFromAnotherFilerIsNotFencedByForeignClock I0824 22:18:40.149332 leveldb_store.go:48 filer store dir: /tmp/TestEventFromAnotherFilerIsNotFencedByForeignClock414939610/001/meta I0824 22:18:40.149747 file_util.go:29 Folder /tmp/TestEventFromAnotherFilerIsNotFencedByForeignClock414939610/001/meta Permission: -rwxr-xr-x --- PASS: TestEventFromAnotherFilerIsNotFencedByForeignClock (0.03s) === RUN TestTouchBeforeCopyEventKeepsPostCopyWrites I0824 22:18:40.176374 leveldb_store.go:48 filer store dir: /tmp/TestTouchBeforeCopyEventKeepsPostCopyWrites3978213494/001/meta I0824 22:18:40.176616 file_util.go:29 Folder /tmp/TestTouchBeforeCopyEventKeepsPostCopyWrites3978213494/001/meta Permission: -rwxr-xr-x --- PASS: TestTouchBeforeCopyEventKeepsPostCopyWrites (0.03s) === RUN TestRejectedDownloadDoesNotRollBackCache I0824 22:18:40.223887 leveldb_store.go:48 filer store dir: /tmp/TestRejectedDownloadDoesNotRollBackCache1204315864/001/meta I0824 22:18:40.224258 file_util.go:29 Folder /tmp/TestRejectedDownloadDoesNotRollBackCache1204315864/001/meta Permission: -rwxr-xr-x --- PASS: TestRejectedDownloadDoesNotRollBackCache (0.05s) === RUN TestRemoteOnlyHandleRefusesKnownOlderDownload I0824 22:18:40.257892 leveldb_store.go:48 filer store dir: /tmp/TestRemoteOnlyHandleRefusesKnownOlderDownload1896170952/001/meta I0824 22:18:40.258088 file_util.go:29 Folder /tmp/TestRemoteOnlyHandleRefusesKnownOlderDownload1896170952/001/meta Permission: -rwxr-xr-x --- PASS: TestRemoteOnlyHandleRefusesKnownOlderDownload (0.07s) === RUN TestForeignChmodPreservesDirtyPages I0824 22:18:40.323876 leveldb_store.go:48 filer store dir: /tmp/TestForeignChmodPreservesDirtyPages1796903320/001/meta I0824 22:18:40.324295 file_util.go:29 Folder /tmp/TestForeignChmodPreservesDirtyPages1796903320/001/meta Permission: -rwxr-xr-x --- PASS: TestForeignChmodPreservesDirtyPages (0.01s) === RUN TestRenameOverExistingMarksReplacedHandleDeleted I0824 22:18:40.337549 leveldb_store.go:48 filer store dir: /tmp/TestRenameOverExistingMarksReplacedHandleDeleted2157993831/001/meta I0824 22:18:40.337931 file_util.go:29 Folder /tmp/TestRenameOverExistingMarksReplacedHandleDeleted2157993831/001/meta Permission: -rwxr-xr-x --- PASS: TestRenameOverExistingMarksReplacedHandleDeleted (0.03s) === RUN TestAckFromAnotherFilerIsNotDroppedBehindForeignFence I0824 22:18:40.365710 leveldb_store.go:48 filer store dir: /tmp/TestAckFromAnotherFilerIsNotDroppedBehindForeignFence3316826574/001/meta I0824 22:18:40.365977 file_util.go:29 Folder /tmp/TestAckFromAnotherFilerIsNotDroppedBehindForeignFence3316826574/001/meta Permission: -rwxr-xr-x --- PASS: TestAckFromAnotherFilerIsNotDroppedBehindForeignFence (0.02s) === RUN TestFlushFileMetadataPreservesUserMtime --- PASS: TestFlushFileMetadataPreservesUserMtime (0.00s) === RUN TestPosixLockTypeMapping --- PASS: TestPosixLockTypeMapping (0.00s) === RUN TestPosixPollAcquireGrantedImmediately --- PASS: TestPosixPollAcquireGrantedImmediately (0.00s) === RUN TestPosixPollAcquireRetriesThenGrants --- PASS: TestPosixPollAcquireRetriesThenGrants (0.03s) === RUN TestPosixPollAcquireError --- PASS: TestPosixPollAcquireError (0.00s) === RUN TestPosixPollAcquireCancel --- PASS: TestPosixPollAcquireCancel (0.00s) === RUN TestPosixLockHint --- PASS: TestPosixLockHint (0.00s) === RUN TestUncommittedBytesTracking --- PASS: TestUncommittedBytesTracking (0.00s) === RUN TestUncommittedBytesDoesNotGoNegative --- PASS: TestUncommittedBytesDoesNotGoNegative (0.00s) === RUN TestIsOverQuotaWithUncommitted === RUN TestIsOverQuotaWithUncommitted/no_quota_set === RUN TestIsOverQuotaWithUncommitted/under_quota === RUN TestIsOverQuotaWithUncommitted/over_quota_with_uncommitted === RUN TestIsOverQuotaWithUncommitted/already_over_quota_flag_set === RUN TestIsOverQuotaWithUncommitted/exactly_at_quota === RUN TestIsOverQuotaWithUncommitted/one_byte_over_quota --- PASS: TestIsOverQuotaWithUncommitted (0.00s) --- PASS: TestIsOverQuotaWithUncommitted/no_quota_set (0.00s) --- PASS: TestIsOverQuotaWithUncommitted/under_quota (0.00s) --- PASS: TestIsOverQuotaWithUncommitted/over_quota_with_uncommitted (0.00s) --- PASS: TestIsOverQuotaWithUncommitted/already_over_quota_flag_set (0.00s) --- PASS: TestIsOverQuotaWithUncommitted/exactly_at_quota (0.00s) --- PASS: TestIsOverQuotaWithUncommitted/one_byte_over_quota (0.00s) === RUN TestGetQuotaCheckInterval === RUN TestGetQuotaCheckInterval/no_quota === RUN TestGetQuotaCheckInterval/under_90%_threshold === RUN TestGetQuotaCheckInterval/at_90%_threshold === RUN TestGetQuotaCheckInterval/over_90%_with_uncommitted --- PASS: TestGetQuotaCheckInterval (0.00s) --- PASS: TestGetQuotaCheckInterval/no_quota (0.00s) --- PASS: TestGetQuotaCheckInterval/under_90%_threshold (0.00s) --- PASS: TestGetQuotaCheckInterval/at_90%_threshold (0.00s) --- PASS: TestGetQuotaCheckInterval/over_90%_with_uncommitted (0.00s) === RUN TestNoQuotaTrackingWhenDisabled --- PASS: TestNoQuotaTrackingWhenDisabled (0.00s) === RUN TestRemoteRenameMovesInodePaths I0824 22:18:40.421871 leveldb_store.go:48 filer store dir: /tmp/TestRemoteRenameMovesInodePaths1338383098/001/meta I0824 22:18:40.422255 file_util.go:29 Folder /tmp/TestRemoteRenameMovesInodePaths1338383098/001/meta Permission: -rwxr-xr-x --- PASS: TestRemoteRenameMovesInodePaths (0.01s) === RUN TestRemoteRenameMovesOnceWithAnOpenHandle I0824 22:18:40.435331 leveldb_store.go:48 filer store dir: /tmp/TestRemoteRenameMovesOnceWithAnOpenHandle2502944890/001/meta I0824 22:18:40.435543 file_util.go:29 Folder /tmp/TestRemoteRenameMovesOnceWithAnOpenHandle2502944890/001/meta Permission: -rwxr-xr-x --- PASS: TestRemoteRenameMovesOnceWithAnOpenHandle (0.03s) === RUN TestRemoteRenameReplayLeavesTheTargetAlone I0824 22:18:40.466649 leveldb_store.go:48 filer store dir: /tmp/TestRemoteRenameReplayLeavesTheTargetAlone2140978509/001/meta I0824 22:18:40.466905 file_util.go:29 Folder /tmp/TestRemoteRenameReplayLeavesTheTargetAlone2140978509/001/meta Permission: -rwxr-xr-x --- PASS: TestRemoteRenameReplayLeavesTheTargetAlone (0.01s) === RUN TestRenameBelowTheVersionFenceStillMoves I0824 22:18:40.476115 leveldb_store.go:48 filer store dir: /tmp/TestRenameBelowTheVersionFenceStillMoves989388073/001/meta I0824 22:18:40.476382 file_util.go:29 Folder /tmp/TestRenameBelowTheVersionFenceStillMoves989388073/001/meta Permission: -rwxr-xr-x --- PASS: TestRenameBelowTheVersionFenceStillMoves (0.11s) === RUN TestHandleRenameResponseLeavesUncachedTargetOutOfCache I0824 22:18:40.591071 leveldb_store.go:48 filer store dir: /tmp/TestHandleRenameResponseLeavesUncachedTargetOutOfCache3603138522/001/meta I0824 22:18:40.591474 file_util.go:29 Folder /tmp/TestHandleRenameResponseLeavesUncachedTargetOutOfCache3603138522/001/meta Permission: -rwxr-xr-x --- PASS: TestHandleRenameResponseLeavesUncachedTargetOutOfCache (0.02s) === RUN TestDiskSizes --- PASS: TestDiskSizes (0.00s) === RUN TestSetXAttrCopiesValueBuffer --- PASS: TestSetXAttrCopiesValueBuffer (0.00s) === RUN TestSanitizeFuseName_InvalidBytesReplaced --- PASS: TestSanitizeFuseName_InvalidBytesReplaced (0.00s) === RUN TestSanitizeFuseName_PassThroughValidUTF8 --- PASS: TestSanitizeFuseName_PassThroughValidUTF8 (0.00s) === RUN TestCheckName_SanitizesBeforeLengthCheck --- PASS: TestCheckName_SanitizesBeforeLengthCheck (0.00s) PASS ok github.com/seaweedfs/seaweedfs/weed/mount 31.818s === RUN TestApplyMetadataResponseAppliesEventsInOrder I0824 22:18:26.745853 leveldb_store.go:48 filer store dir: /tmp/TestApplyMetadataResponseAppliesEventsInOrder1534493582/001/meta I0824 22:18:26.762715 file_util.go:29 Folder /tmp/TestApplyMetadataResponseAppliesEventsInOrder1534493582/001/meta Permission: -rwxr-xr-x --- PASS: TestApplyMetadataResponseAppliesEventsInOrder (0.36s) === RUN TestApplyMetadataResponseRenamesAcrossCachedDirectories I0824 22:18:27.106825 leveldb_store.go:48 filer store dir: /tmp/TestApplyMetadataResponseRenamesAcrossCachedDirectories304538099/001/meta I0824 22:18:27.107343 file_util.go:29 Folder /tmp/TestApplyMetadataResponseRenamesAcrossCachedDirectories304538099/001/meta Permission: -rwxr-xr-x --- PASS: TestApplyMetadataResponseRenamesAcrossCachedDirectories (0.03s) === RUN TestApplyMetadataResponseLocalOptionsSkipInvalidations I0824 22:18:27.142485 leveldb_store.go:48 filer store dir: /tmp/TestApplyMetadataResponseLocalOptionsSkipInvalidations1215510691/001/meta I0824 22:18:27.142998 file_util.go:29 Folder /tmp/TestApplyMetadataResponseLocalOptionsSkipInvalidations1215510691/001/meta Permission: -rwxr-xr-x --- PASS: TestApplyMetadataResponseLocalOptionsSkipInvalidations (0.08s) === RUN TestApplyMetadataResponseDeduplicatesRepeatedFilerEvent I0824 22:18:27.220909 leveldb_store.go:48 filer store dir: /tmp/TestApplyMetadataResponseDeduplicatesRepeatedFilerEvent3583576129/001/meta I0824 22:18:27.221342 file_util.go:29 Folder /tmp/TestApplyMetadataResponseDeduplicatesRepeatedFilerEvent3583576129/001/meta Permission: -rwxr-xr-x --- PASS: TestApplyMetadataResponseDeduplicatesRepeatedFilerEvent (0.07s) === RUN TestApplyMetadataResponseSkipsHiddenSystemEntryWhenDisabled I0824 22:18:27.290262 leveldb_store.go:48 filer store dir: /tmp/TestApplyMetadataResponseSkipsHiddenSystemEntryWhenDisabled2123207289/001/meta I0824 22:18:27.290662 file_util.go:29 Folder /tmp/TestApplyMetadataResponseSkipsHiddenSystemEntryWhenDisabled2123207289/001/meta Permission: -rwxr-xr-x --- PASS: TestApplyMetadataResponseSkipsHiddenSystemEntryWhenDisabled (0.13s) === RUN TestApplyMetadataResponsePurgesHiddenDestinationPath I0824 22:18:27.440754 leveldb_store.go:48 filer store dir: /tmp/TestApplyMetadataResponsePurgesHiddenDestinationPath33828248/001/meta I0824 22:18:27.441102 file_util.go:29 Folder /tmp/TestApplyMetadataResponsePurgesHiddenDestinationPath33828248/001/meta Permission: -rwxr-xr-x --- PASS: TestApplyMetadataResponsePurgesHiddenDestinationPath (0.09s) === RUN TestCollectEntryInvalidationsCarryAuthoritativeEntries --- PASS: TestCollectEntryInvalidationsCarryAuthoritativeEntries (0.00s) === RUN TestUnversionedWriteClearsEntryVersion I0824 22:18:27.509573 leveldb_store.go:48 filer store dir: /tmp/TestUnversionedWriteClearsEntryVersion901288076/001/meta I0824 22:18:27.509841 file_util.go:29 Folder /tmp/TestUnversionedWriteClearsEntryVersion901288076/001/meta Permission: -rwxr-xr-x --- PASS: TestUnversionedWriteClearsEntryVersion (0.08s) === RUN TestUnversionedRebuildClearsStaleVersions I0824 22:18:27.590123 leveldb_store.go:48 filer store dir: /tmp/TestUnversionedRebuildClearsStaleVersions1415887207/001/meta I0824 22:18:27.590936 file_util.go:29 Folder /tmp/TestUnversionedRebuildClearsStaleVersions1415887207/001/meta Permission: -rwxr-xr-x --- PASS: TestUnversionedRebuildClearsStaleVersions (0.04s) === RUN TestTombstonesScopedToCachedDirectories I0824 22:18:27.631127 leveldb_store.go:48 filer store dir: /tmp/TestTombstonesScopedToCachedDirectories1798497289/001/meta I0824 22:18:27.631794 file_util.go:29 Folder /tmp/TestTombstonesScopedToCachedDirectories1798497289/001/meta Permission: -rwxr-xr-x --- PASS: TestTombstonesScopedToCachedDirectories (0.12s) === RUN TestBuildCompletionPrunesSupersededTombstones I0824 22:18:27.750315 leveldb_store.go:48 filer store dir: /tmp/TestBuildCompletionPrunesSupersededTombstones3006783228/001/meta I0824 22:18:27.750799 file_util.go:29 Folder /tmp/TestBuildCompletionPrunesSupersededTombstones3006783228/001/meta Permission: -rwxr-xr-x --- PASS: TestBuildCompletionPrunesSupersededTombstones (0.09s) === RUN TestDirectoryEvictionClearsVersionRecords I0824 22:18:27.848046 leveldb_store.go:48 filer store dir: /tmp/TestDirectoryEvictionClearsVersionRecords2177209016/001/meta I0824 22:18:27.856487 file_util.go:29 Folder /tmp/TestDirectoryEvictionClearsVersionRecords2177209016/001/meta Permission: -rwxr-xr-x --- PASS: TestDirectoryEvictionClearsVersionRecords (0.20s) === RUN TestBuildFloorVersionsChildrenWithoutPerChildRecords I0824 22:18:28.084193 leveldb_store.go:48 filer store dir: /tmp/TestBuildFloorVersionsChildrenWithoutPerChildRecords2860867536/001/meta I0824 22:18:28.085319 file_util.go:29 Folder /tmp/TestBuildFloorVersionsChildrenWithoutPerChildRecords2860867536/001/meta Permission: -rwxr-xr-x --- PASS: TestBuildFloorVersionsChildrenWithoutPerChildRecords (0.09s) === RUN TestExpiredEntryIsJudgedByDirectoryFloor I0824 22:18:28.149821 leveldb_store.go:48 filer store dir: /tmp/TestExpiredEntryIsJudgedByDirectoryFloor1421185762/001/meta I0824 22:18:28.150179 file_util.go:29 Folder /tmp/TestExpiredEntryIsJudgedByDirectoryFloor1421185762/001/meta Permission: -rwxr-xr-x --- PASS: TestExpiredEntryIsJudgedByDirectoryFloor (0.07s) === RUN TestUnversionedWriteDoesNotInheritDirectoryFloor I0824 22:18:28.218198 leveldb_store.go:48 filer store dir: /tmp/TestUnversionedWriteDoesNotInheritDirectoryFloor3896735373/001/meta I0824 22:18:28.218395 file_util.go:29 Folder /tmp/TestUnversionedWriteDoesNotInheritDirectoryFloor3896735373/001/meta Permission: -rwxr-xr-x --- PASS: TestUnversionedWriteDoesNotInheritDirectoryFloor (0.02s) === RUN TestEnsureVisitedReplaysBufferedEventsAfterSnapshot I0824 22:18:28.241945 leveldb_store.go:48 filer store dir: /tmp/TestEnsureVisitedReplaysBufferedEventsAfterSnapshot4237914202/001/meta I0824 22:18:28.242105 file_util.go:29 Folder /tmp/TestEnsureVisitedReplaysBufferedEventsAfterSnapshot4237914202/001/meta Permission: -rwxr-xr-x --- PASS: TestEnsureVisitedReplaysBufferedEventsAfterSnapshot (0.06s) === RUN TestDirectoryNotificationsSuppressedDuringBuild I0824 22:18:28.304622 leveldb_store.go:48 filer store dir: /tmp/TestDirectoryNotificationsSuppressedDuringBuild2255993155/001/meta I0824 22:18:28.305814 file_util.go:29 Folder /tmp/TestDirectoryNotificationsSuppressedDuringBuild2255993155/001/meta Permission: -rwxr-xr-x --- PASS: TestDirectoryNotificationsSuppressedDuringBuild (0.05s) === RUN TestEmptyDirectoryBuildReplaysAllBufferedEvents I0824 22:18:28.351465 leveldb_store.go:48 filer store dir: /tmp/TestEmptyDirectoryBuildReplaysAllBufferedEvents3017420117/001/meta I0824 22:18:28.351854 file_util.go:29 Folder /tmp/TestEmptyDirectoryBuildReplaysAllBufferedEvents3017420117/001/meta Permission: -rwxr-xr-x --- PASS: TestEmptyDirectoryBuildReplaysAllBufferedEvents (0.09s) === RUN TestBuildCompletionSurvivesCallerCancellation I0824 22:18:28.440369 leveldb_store.go:48 filer store dir: /tmp/TestBuildCompletionSurvivesCallerCancellation3881880445/001/meta I0824 22:18:28.440547 file_util.go:29 Folder /tmp/TestBuildCompletionSurvivesCallerCancellation3881880445/001/meta Permission: -rwxr-xr-x --- PASS: TestBuildCompletionSurvivesCallerCancellation (0.03s) === RUN TestBufferedRenameUpdatesOtherDirectoryBeforeBuildCompletes I0824 22:18:28.470636 leveldb_store.go:48 filer store dir: /tmp/TestBufferedRenameUpdatesOtherDirectoryBeforeBuildCompletes83342833/001/meta I0824 22:18:28.470821 file_util.go:29 Folder /tmp/TestBufferedRenameUpdatesOtherDirectoryBeforeBuildCompletes83342833/001/meta Permission: -rwxr-xr-x --- PASS: TestBufferedRenameUpdatesOtherDirectoryBeforeBuildCompletes (0.05s) === RUN TestEnsureVisitedPreservesLocalOnlyEntry I0824 22:18:28.516795 leveldb_store.go:48 filer store dir: /tmp/TestEnsureVisitedPreservesLocalOnlyEntry4277118614/001/meta I0824 22:18:28.516948 file_util.go:29 Folder /tmp/TestEnsureVisitedPreservesLocalOnlyEntry4277118614/001/meta Permission: -rwxr-xr-x --- PASS: TestEnsureVisitedPreservesLocalOnlyEntry (0.02s) === RUN TestEnsureVisitedDropsUnpinnedStaleEntry I0824 22:18:28.540230 leveldb_store.go:48 filer store dir: /tmp/TestEnsureVisitedDropsUnpinnedStaleEntry2326297701/001/meta I0824 22:18:28.540622 file_util.go:29 Folder /tmp/TestEnsureVisitedDropsUnpinnedStaleEntry2326297701/001/meta Permission: -rwxr-xr-x --- PASS: TestEnsureVisitedDropsUnpinnedStaleEntry (0.07s) === RUN TestEnsureVisitedConfirmsTransientEmptyListing I0824 22:18:28.606999 leveldb_store.go:48 filer store dir: /tmp/TestEnsureVisitedConfirmsTransientEmptyListing3157586196/001/meta I0824 22:18:28.607951 file_util.go:29 Folder /tmp/TestEnsureVisitedConfirmsTransientEmptyListing3157586196/001/meta Permission: -rwxr-xr-x W0824 22:18:28.634946 meta_cache_init.go:214 rebuild of /dir saw a transient empty listing, recovered 1 entries on confirmation --- PASS: TestEnsureVisitedConfirmsTransientEmptyListing (0.04s) === RUN TestEnsureVisitedCachesGenuinelyEmptyDirectory I0824 22:18:28.650417 leveldb_store.go:48 filer store dir: /tmp/TestEnsureVisitedCachesGenuinelyEmptyDirectory371710134/001/meta I0824 22:18:28.651187 file_util.go:29 Folder /tmp/TestEnsureVisitedCachesGenuinelyEmptyDirectory371710134/001/meta Permission: -rwxr-xr-x --- PASS: TestEnsureVisitedCachesGenuinelyEmptyDirectory (0.10s) === RUN TestApplyLoopInvalidateDoesNotDeadlockWithLockHoldingEnqueuer I0824 22:18:28.754326 leveldb_store.go:48 filer store dir: /tmp/TestApplyLoopInvalidateDoesNotDeadlockWithLockHoldingEnqueuer2135141371/001/meta I0824 22:18:28.754522 file_util.go:29 Folder /tmp/TestApplyLoopInvalidateDoesNotDeadlockWithLockHoldingEnqueuer2135141371/001/meta Permission: -rwxr-xr-x --- PASS: TestApplyLoopInvalidateDoesNotDeadlockWithLockHoldingEnqueuer (0.07s) === RUN TestPurgeSkippedWhileDirectoryBuilding I0824 22:18:28.823463 leveldb_store.go:48 filer store dir: /tmp/TestPurgeSkippedWhileDirectoryBuilding2201297648/001/meta I0824 22:18:28.824062 file_util.go:29 Folder /tmp/TestPurgeSkippedWhileDirectoryBuilding2201297648/001/meta Permission: -rwxr-xr-x --- PASS: TestPurgeSkippedWhileDirectoryBuilding (0.07s) === RUN TestPurgeClearsWhenNotBuilding I0824 22:18:28.896539 leveldb_store.go:48 filer store dir: /tmp/TestPurgeClearsWhenNotBuilding2443857716/001/meta I0824 22:18:28.897273 file_util.go:29 Folder /tmp/TestPurgeClearsWhenNotBuilding2443857716/001/meta Permission: -rwxr-xr-x --- PASS: TestPurgeClearsWhenNotBuilding (0.05s) === RUN TestSectionOf --- PASS: TestSectionOf (0.00s) === RUN TestSectionBurstInvalidatesOnlyItsSection I0824 22:18:28.943395 leveldb_store.go:48 filer store dir: /tmp/TestSectionBurstInvalidatesOnlyItsSection2285533738/001/meta I0824 22:18:28.943976 file_util.go:29 Folder /tmp/TestSectionBurstInvalidatesOnlyItsSection2285533738/001/meta Permission: -rwxr-xr-x --- PASS: TestSectionBurstInvalidatesOnlyItsSection (0.19s) === RUN TestLocalChangesDoNotInvalidateSections I0824 22:18:29.137029 leveldb_store.go:48 filer store dir: /tmp/TestLocalChangesDoNotInvalidateSections1075817418/001/meta I0824 22:18:29.137654 file_util.go:29 Folder /tmp/TestLocalChangesDoNotInvalidateSections1075817418/001/meta Permission: -rwxr-xr-x --- PASS: TestLocalChangesDoNotInvalidateSections (0.11s) === RUN TestSectionRefreshReconciles I0824 22:18:29.249959 leveldb_store.go:48 filer store dir: /tmp/TestSectionRefreshReconciles424365512/001/meta I0824 22:18:29.250148 file_util.go:29 Folder /tmp/TestSectionRefreshReconciles424365512/001/meta Permission: -rwxr-xr-x --- PASS: TestSectionRefreshReconciles (0.50s) === RUN TestSectionRefreshSplitsOvergrownSection I0824 22:18:29.778109 leveldb_store.go:48 filer store dir: /tmp/TestSectionRefreshSplitsOvergrownSection1568354732/001/meta I0824 22:18:29.779850 file_util.go:29 Folder /tmp/TestSectionRefreshSplitsOvergrownSection1568354732/001/meta Permission: -rwxr-xr-x --- PASS: TestSectionRefreshSplitsOvergrownSection (0.49s) === RUN TestEnsureListingFreshRefreshesFromFiler I0824 22:18:30.264216 leveldb_store.go:48 filer store dir: /tmp/TestEnsureListingFreshRefreshesFromFiler2430159825/001/meta I0824 22:18:30.264418 file_util.go:29 Folder /tmp/TestEnsureListingFreshRefreshesFromFiler2430159825/001/meta Permission: -rwxr-xr-x --- PASS: TestEnsureListingFreshRefreshesFromFiler (0.64s) === RUN TestEnsureListingFreshGivesUpOnOvergrownRange I0824 22:18:30.909189 leveldb_store.go:48 filer store dir: /tmp/TestEnsureListingFreshGivesUpOnOvergrownRange2610317070/001/meta I0824 22:18:30.910067 file_util.go:29 Folder /tmp/TestEnsureListingFreshGivesUpOnOvergrownRange2610317070/001/meta Permission: -rwxr-xr-x --- PASS: TestEnsureListingFreshGivesUpOnOvergrownRange (0.72s) === RUN TestSectionNoteChangeWindowExpiry --- PASS: TestSectionNoteChangeWindowExpiry (0.00s) === RUN TestSectionCompleteRefreshGuardsChangedTable --- PASS: TestSectionCompleteRefreshGuardsChangedTable (0.00s) === RUN TestSectionCompleteRefreshSplitsMiddleSection --- PASS: TestSectionCompleteRefreshSplitsMiddleSection (0.00s) === RUN TestSectionFloorFencesAbsentNames I0824 22:18:31.643887 leveldb_store.go:48 filer store dir: /tmp/TestSectionFloorFencesAbsentNames2710865543/001/meta I0824 22:18:31.644758 file_util.go:29 Folder /tmp/TestSectionFloorFencesAbsentNames2710865543/001/meta Permission: -rwxr-xr-x --- PASS: TestSectionFloorFencesAbsentNames (0.21s) === RUN TestUnversionedRefreshStaysStale I0824 22:18:31.851881 leveldb_store.go:48 filer store dir: /tmp/TestUnversionedRefreshStaysStale1646906152/001/meta I0824 22:18:31.852092 file_util.go:29 Folder /tmp/TestUnversionedRefreshStaysStale1646906152/001/meta Permission: -rwxr-xr-x --- PASS: TestUnversionedRefreshStaysStale (0.09s) === RUN TestEnsureListingFreshCoversAllStaleSectionsAhead I0824 22:18:31.946497 leveldb_store.go:48 filer store dir: /tmp/TestEnsureListingFreshCoversAllStaleSectionsAhead3419671711/001/meta I0824 22:18:31.946862 file_util.go:29 Folder /tmp/TestEnsureListingFreshCoversAllStaleSectionsAhead3419671711/001/meta Permission: -rwxr-xr-x --- PASS: TestEnsureListingFreshCoversAllStaleSectionsAhead (0.68s) === RUN TestSectionCompleteRefreshSetsFloors --- PASS: TestSectionCompleteRefreshSetsFloors (0.00s) === RUN TestSectionFloorOutranksOlderTombstone I0824 22:18:32.626196 leveldb_store.go:48 filer store dir: /tmp/TestSectionFloorOutranksOlderTombstone2365618834/001/meta I0824 22:18:32.626351 file_util.go:29 Folder /tmp/TestSectionFloorOutranksOlderTombstone2365618834/001/meta Permission: -rwxr-xr-x --- PASS: TestSectionFloorOutranksOlderTombstone (0.11s) === RUN TestMismatchedRefreshLeavesStoreAlone I0824 22:18:32.736857 leveldb_store.go:48 filer store dir: /tmp/TestMismatchedRefreshLeavesStoreAlone1992133056/001/meta I0824 22:18:32.737441 file_util.go:29 Folder /tmp/TestMismatchedRefreshLeavesStoreAlone1992133056/001/meta Permission: -rwxr-xr-x --- PASS: TestMismatchedRefreshLeavesStoreAlone (0.10s) === RUN TestRefreshClearsUnversionedMarker I0824 22:18:32.840684 leveldb_store.go:48 filer store dir: /tmp/TestRefreshClearsUnversionedMarker3121110740/001/meta I0824 22:18:32.840982 file_util.go:29 Folder /tmp/TestRefreshClearsUnversionedMarker3121110740/001/meta Permission: -rwxr-xr-x --- PASS: TestRefreshClearsUnversionedMarker (0.17s) === RUN TestRefreshSkipsBuildingDirectory I0824 22:18:33.012936 leveldb_store.go:48 filer store dir: /tmp/TestRefreshSkipsBuildingDirectory3618879795/001/meta I0824 22:18:33.013501 file_util.go:29 Folder /tmp/TestRefreshSkipsBuildingDirectory3618879795/001/meta Permission: -rwxr-xr-x --- PASS: TestRefreshSkipsBuildingDirectory (0.11s) === RUN TestSectionBorderEntries I0824 22:18:33.131435 leveldb_store.go:48 filer store dir: /tmp/TestSectionBorderEntries3990871470/001/meta I0824 22:18:33.132634 file_util.go:29 Folder /tmp/TestSectionBorderEntries3990871470/001/meta Permission: -rwxr-xr-x --- PASS: TestSectionBorderEntries (0.19s) === RUN TestChurnBeyondBuiltEntriesLandsInEdgeSection I0824 22:18:33.324830 leveldb_store.go:48 filer store dir: /tmp/TestChurnBeyondBuiltEntriesLandsInEdgeSection2456827714/001/meta I0824 22:18:33.325852 file_util.go:29 Folder /tmp/TestChurnBeyondBuiltEntriesLandsInEdgeSection2456827714/001/meta Permission: -rwxr-xr-x --- PASS: TestChurnBeyondBuiltEntriesLandsInEdgeSection (0.11s) === RUN TestRenameAcrossSectionsCountsBoth I0824 22:18:33.440282 leveldb_store.go:48 filer store dir: /tmp/TestRenameAcrossSectionsCountsBoth4203233129/001/meta I0824 22:18:33.440904 file_util.go:29 Folder /tmp/TestRenameAcrossSectionsCountsBoth4203233129/001/meta Permission: -rwxr-xr-x --- PASS: TestRenameAcrossSectionsCountsBoth (0.07s) === RUN TestEnsureVisitedRefusesOversizedDirectory I0824 22:18:33.509157 leveldb_store.go:48 filer store dir: /tmp/TestEnsureVisitedRefusesOversizedDirectory310667296/001/meta I0824 22:18:33.509567 file_util.go:29 Folder /tmp/TestEnsureVisitedRefusesOversizedDirectory310667296/001/meta Permission: -rwxr-xr-x I0824 22:18:33.598658 meta_cache_init.go:186 directory /dir exceeds 5 entries, reading it through instead of caching W0824 22:18:33.599397 meta_cache.go:660 unsynchronized dir: /dir --- PASS: TestEnsureVisitedRefusesOversizedDirectory (0.09s) === RUN TestEnsureVisitedStepsOverOversizedAncestor I0824 22:18:33.605086 leveldb_store.go:48 filer store dir: /tmp/TestEnsureVisitedStepsOverOversizedAncestor2530676974/001/meta I0824 22:18:33.605722 file_util.go:29 Folder /tmp/TestEnsureVisitedStepsOverOversizedAncestor2530676974/001/meta Permission: -rwxr-xr-x --- PASS: TestEnsureVisitedStepsOverOversizedAncestor (0.03s) === RUN TestEnsureVisitedUnderTheLimitStillCaches I0824 22:18:33.636065 leveldb_store.go:48 filer store dir: /tmp/TestEnsureVisitedUnderTheLimitStillCaches1680401740/001/meta I0824 22:18:33.636486 file_util.go:29 Folder /tmp/TestEnsureVisitedUnderTheLimitStillCaches1680401740/001/meta Permission: -rwxr-xr-x --- PASS: TestEnsureVisitedUnderTheLimitStillCaches (0.05s) === RUN TestEnsureVisitedAncestorFoundOversizedMidVisit I0824 22:18:33.691170 leveldb_store.go:48 filer store dir: /tmp/TestEnsureVisitedAncestorFoundOversizedMidVisit605800201/001/meta I0824 22:18:33.691801 file_util.go:29 Folder /tmp/TestEnsureVisitedAncestorFoundOversizedMidVisit605800201/001/meta Permission: -rwxr-xr-x I0824 22:18:33.721454 meta_cache_init.go:186 directory /huge exceeds 5 entries, reading it through instead of caching --- PASS: TestEnsureVisitedAncestorFoundOversizedMidVisit (0.04s) PASS ok github.com/seaweedfs/seaweedfs/weed/mount/meta_cache 24.952s === RUN Test_PageChunkWrittenIntervalList --- PASS: Test_PageChunkWrittenIntervalList (0.00s) === RUN Test_PageChunkWrittenIntervalList1 --- PASS: Test_PageChunkWrittenIntervalList1 (0.00s) === RUN TestIsContiguouslyWritten === RUN TestIsContiguouslyWritten/empty === RUN TestIsContiguouslyWritten/single_full === RUN TestIsContiguouslyWritten/single_partial_at_start === RUN TestIsContiguouslyWritten/two_adjacent_halves_in_order === RUN TestIsContiguouslyWritten/two_adjacent_halves_out_of_order === RUN TestIsContiguouslyWritten/overlapping_covers_everything === RUN TestIsContiguouslyWritten/three_adjacent_thirds === RUN TestIsContiguouslyWritten/single_partial_in_middle === RUN TestIsContiguouslyWritten/first_byte_missing === RUN TestIsContiguouslyWritten/gap_in_middle === RUN TestIsContiguouslyWritten/two_intervals_with_1-byte_gap === RUN TestIsContiguouslyWritten/three_intervals_with_middle_gap --- PASS: TestIsContiguouslyWritten (0.01s) --- PASS: TestIsContiguouslyWritten/empty (0.00s) --- PASS: TestIsContiguouslyWritten/single_full (0.00s) --- PASS: TestIsContiguouslyWritten/single_partial_at_start (0.00s) --- PASS: TestIsContiguouslyWritten/two_adjacent_halves_in_order (0.00s) --- PASS: TestIsContiguouslyWritten/two_adjacent_halves_out_of_order (0.00s) --- PASS: TestIsContiguouslyWritten/overlapping_covers_everything (0.00s) --- PASS: TestIsContiguouslyWritten/three_adjacent_thirds (0.00s) --- PASS: TestIsContiguouslyWritten/single_partial_in_middle (0.00s) --- PASS: TestIsContiguouslyWritten/first_byte_missing (0.00s) --- PASS: TestIsContiguouslyWritten/gap_in_middle (0.00s) --- PASS: TestIsContiguouslyWritten/two_intervals_with_1-byte_gap (0.00s) --- PASS: TestIsContiguouslyWritten/three_intervals_with_middle_gap (0.00s) === RUN TestIsComplete_AdjacentIntervals === RUN TestIsComplete_AdjacentIntervals/single_full_write === RUN TestIsComplete_AdjacentIntervals/two_adjacent_halves_in_order === RUN TestIsComplete_AdjacentIntervals/two_adjacent_halves_out_of_order === RUN TestIsComplete_AdjacentIntervals/eight_adjacent_eighths === RUN TestIsComplete_AdjacentIntervals/gap_in_the_middle === RUN TestIsComplete_AdjacentIntervals/missing_last_byte === RUN TestIsComplete_AdjacentIntervals/missing_first_byte === RUN TestIsComplete_AdjacentIntervals/overlapping_intervals_covering_everything --- PASS: TestIsComplete_AdjacentIntervals (0.01s) --- PASS: TestIsComplete_AdjacentIntervals/single_full_write (0.00s) --- PASS: TestIsComplete_AdjacentIntervals/two_adjacent_halves_in_order (0.00s) --- PASS: TestIsComplete_AdjacentIntervals/two_adjacent_halves_out_of_order (0.00s) --- PASS: TestIsComplete_AdjacentIntervals/eight_adjacent_eighths (0.00s) --- PASS: TestIsComplete_AdjacentIntervals/gap_in_the_middle (0.00s) --- PASS: TestIsComplete_AdjacentIntervals/missing_last_byte (0.00s) --- PASS: TestIsComplete_AdjacentIntervals/missing_first_byte (0.00s) --- PASS: TestIsComplete_AdjacentIntervals/overlapping_intervals_covering_everything (0.00s) === RUN TestSwapFile_NewSwapFileChunk_Concurrent --- PASS: TestSwapFile_NewSwapFileChunk_Concurrent (0.02s) === RUN TestSwapFile_MkdirAll_Permissions --- PASS: TestSwapFile_MkdirAll_Permissions (0.00s) === RUN TestSwapFile_RecreateDir --- PASS: TestSwapFile_RecreateDir (0.01s) === RUN TestMoveToSealed_ReplaceKeepsUploadReference --- PASS: TestMoveToSealed_ReplaceKeepsUploadReference (0.00s) === RUN TestShutdown_LeavesInFlightUploadItsReference --- PASS: TestShutdown_LeavesInFlightUploadItsReference (0.01s) === RUN TestShutdown_ReleasesWritableReservation --- PASS: TestShutdown_ReleasesWritableReservation (0.02s) === RUN TestSealedChunk_FreeReferenceIsIdempotent --- PASS: TestSealedChunk_FreeReferenceIsIdempotent (0.00s) === RUN TestUploadPipeline --- PASS: TestUploadPipeline (332.67s) === RUN TestEvictOneWritableChunk_SkipsGappyChunks --- PASS: TestEvictOneWritableChunk_SkipsGappyChunks (0.11s) === RUN TestEvictOneWritableChunk_FallbackPicksOldestGappy --- PASS: TestEvictOneWritableChunk_FallbackPicksOldestGappy (0.03s) === RUN TestEvictOneWritableChunk_PrefersStrictOverFallback --- PASS: TestEvictOneWritableChunk_PrefersStrictOverFallback (0.08s) === RUN TestProactiveFlush_SkipsGappyChunks --- PASS: TestProactiveFlush_SkipsGappyChunks (0.02s) === RUN TestWriteBufferAccountant_Unlimited --- PASS: TestWriteBufferAccountant_Unlimited (0.00s) === RUN TestWriteBufferAccountant_Nil --- PASS: TestWriteBufferAccountant_Nil (0.00s) === RUN TestWriteBufferAccountant_ReserveAndRelease --- PASS: TestWriteBufferAccountant_ReserveAndRelease (0.00s) === RUN TestWriteBufferAccountant_BlocksWhenOverCap --- PASS: TestWriteBufferAccountant_BlocksWhenOverCap (0.05s) === RUN TestWriteBufferAccountant_SoftThrottle --- PASS: TestWriteBufferAccountant_SoftThrottle (0.02s) === RUN TestWriteBufferAccountant_HardThrottle --- PASS: TestWriteBufferAccountant_HardThrottle (0.06s) === RUN TestWriteBufferAccountant_NoThrottleBelowSoft --- PASS: TestWriteBufferAccountant_NoThrottleBelowSoft (0.00s) === RUN TestWriteBufferAccountant_ThrottleCounters --- PASS: TestWriteBufferAccountant_ThrottleCounters (0.08s) === RUN TestWriteBufferAccountant_GraduatedRecovery --- PASS: TestWriteBufferAccountant_GraduatedRecovery (0.08s) === RUN TestWriteBufferAccountant_AllowsOversizeWhenEmpty --- PASS: TestWriteBufferAccountant_AllowsOversizeWhenEmpty (0.00s) === RUN TestWriteBufferCap_SharedAcrossPipelines write_buffer_cap_integration_test.go:173: peak inflight uploads=4, peak Used()=262144 bytes (cap=262144) --- PASS: TestWriteBufferCap_SharedAcrossPipelines (0.87s) PASS ok github.com/seaweedfs/seaweedfs/weed/mount/page_writer 345.428s ? github.com/seaweedfs/seaweedfs/weed/mount/unmount [no test files] === RUN TestToErrnoUsesCgofuseNumbering --- PASS: TestToErrnoUsesCgofuseNumbering (0.00s) === RUN TestToErrnoUnknownBecomesEIO --- PASS: TestToErrnoUnknownBecomesEIO (0.00s) === RUN TestToErrnoNeverReturnsPositive --- PASS: TestToErrnoNeverReturnsPositive (0.00s) === RUN TestRelativeToMount --- PASS: TestRelativeToMount (0.00s) === RUN TestSplitPath --- PASS: TestSplitPath (0.00s) === RUN TestSplitParent --- PASS: TestSplitParent (0.00s) === RUN TestSplitParentRejectsRoot --- PASS: TestSplitParentRejectsRoot (0.00s) === RUN TestPathCacheLookupAndExpiry --- PASS: TestPathCacheLookupAndExpiry (0.08s) === RUN TestPathCacheStealTransfersOwnership --- PASS: TestPathCacheStealTransfersOwnership (0.16s) === RUN TestPathCacheReplaceReturnsOldReference --- PASS: TestPathCacheReplaceReturnsOldReference (0.10s) === RUN TestPathCachePurgePrefix --- PASS: TestPathCachePurgePrefix (0.00s) === RUN TestPathCacheRootNeverCached --- PASS: TestPathCacheRootNeverCached (0.00s) === RUN TestCacheKey --- PASS: TestCacheKey (0.00s) === RUN TestPathCacheInsertAfterCoveringPurgeIsDiscarded --- PASS: TestPathCacheInsertAfterCoveringPurgeIsDiscarded (0.00s) === RUN TestPathCacheInsertUnderPurgedPrefixIsDiscarded --- PASS: TestPathCacheInsertUnderPurgedPrefixIsDiscarded (0.00s) === RUN TestPathCacheUnrelatedPurgeDoesNotDiscard --- PASS: TestPathCacheUnrelatedPurgeDoesNotDiscard (0.00s) === RUN TestPathCacheInsertOlderThanPurgeWindowIsDiscarded --- PASS: TestPathCacheInsertOlderThanPurgeWindowIsDiscarded (0.00s) === RUN TestPathCacheDiscardedInsertRestsBeforeForget --- PASS: TestPathCacheDiscardedInsertRestsBeforeForget (0.09s) === RUN TestPathCacheRootPurgeCoversEveryInsert --- PASS: TestPathCacheRootPurgeCoversEveryInsert (0.00s) PASS ok github.com/seaweedfs/seaweedfs/weed/mount/winfsp 12.246s ? github.com/seaweedfs/seaweedfs/weed/mq [no test files] ? github.com/seaweedfs/seaweedfs/weed/mq/agent [no test files] === RUN TestConvertOffsetToMessagePosition === RUN TestConvertOffsetToMessagePosition/reset_to_earliest === RUN TestConvertOffsetToMessagePosition/reset_to_latest === RUN TestConvertOffsetToMessagePosition/exact_offset_zero === RUN TestConvertOffsetToMessagePosition/exact_offset_non-zero === RUN TestConvertOffsetToMessagePosition/exact_timestamp --- PASS: TestConvertOffsetToMessagePosition (0.00s) --- PASS: TestConvertOffsetToMessagePosition/reset_to_earliest (0.00s) --- PASS: TestConvertOffsetToMessagePosition/reset_to_latest (0.00s) --- PASS: TestConvertOffsetToMessagePosition/exact_offset_zero (0.00s) --- PASS: TestConvertOffsetToMessagePosition/exact_offset_non-zero (0.00s) --- PASS: TestConvertOffsetToMessagePosition/exact_timestamp (0.00s) === RUN TestConvertOffsetToMessagePosition_OffsetEncoding === RUN TestConvertOffsetToMessagePosition_OffsetEncoding/offset_10 === RUN TestConvertOffsetToMessagePosition_OffsetEncoding/offset_100 === RUN TestConvertOffsetToMessagePosition_OffsetEncoding/offset_0 === RUN TestConvertOffsetToMessagePosition_OffsetEncoding/offset_42 --- PASS: TestConvertOffsetToMessagePosition_OffsetEncoding (0.00s) --- PASS: TestConvertOffsetToMessagePosition_OffsetEncoding/offset_10 (0.00s) --- PASS: TestConvertOffsetToMessagePosition_OffsetEncoding/offset_100 (0.00s) --- PASS: TestConvertOffsetToMessagePosition_OffsetEncoding/offset_0 (0.00s) --- PASS: TestConvertOffsetToMessagePosition_OffsetEncoding/offset_42 (0.00s) === RUN TestConvertOffsetToMessagePosition_ConsistentResults --- PASS: TestConvertOffsetToMessagePosition_ConsistentResults (0.17s) === RUN TestConvertOffsetToMessagePosition_FixVerification --- PASS: TestConvertOffsetToMessagePosition_FixVerification (0.11s) === RUN TestPartitionIdentityConsistency --- PASS: TestPartitionIdentityConsistency (0.00s) === RUN TestBrokerOffsetManager_GetSubscription_Fixed --- PASS: TestBrokerOffsetManager_GetSubscription_Fixed (0.03s) === RUN TestBrokerOffsetManager_ListActiveSubscriptions_Fixed --- PASS: TestBrokerOffsetManager_ListActiveSubscriptions_Fixed (0.03s) === RUN TestMessageQueueBroker_ListActiveSubscriptions_Fixed --- PASS: TestMessageQueueBroker_ListActiveSubscriptions_Fixed (0.00s) === RUN TestSingleWriterPerPartitionCorrectness --- PASS: TestSingleWriterPerPartitionCorrectness (0.00s) === RUN TestEndToEndWorkflowAfterFixes --- PASS: TestEndToEndWorkflowAfterFixes (0.00s) === RUN TestValidateRecordValue --- PASS: TestValidateRecordValue (0.00s) === RUN TestValidateRecordValueEmptyFields --- PASS: TestValidateRecordValueEmptyFields (0.00s) === RUN TestValidateRecordValueNonKafkaTopic --- PASS: TestValidateRecordValueNonKafkaTopic (0.00s) === RUN TestValidateRecordValueNilInputs --- PASS: TestValidateRecordValueNilInputs (0.00s) === RUN TestRecordValueMarshalUnmarshalIntegration --- PASS: TestRecordValueMarshalUnmarshalIntegration (0.11s) PASS ok github.com/seaweedfs/seaweedfs/weed/mq/broker 17.607s ? github.com/seaweedfs/seaweedfs/weed/mq/client/agent_client [no test files] ? github.com/seaweedfs/seaweedfs/weed/mq/client/pub_client [no test files] ? github.com/seaweedfs/seaweedfs/weed/mq/client/sub_client [no test files] ? github.com/seaweedfs/seaweedfs/weed/mq/kafka [no test files] === RUN TestCompressionCodec_String === RUN TestCompressionCodec_String/none === RUN TestCompressionCodec_String/gzip === RUN TestCompressionCodec_String/snappy === RUN TestCompressionCodec_String/lz4 === RUN TestCompressionCodec_String/zstd === RUN TestCompressionCodec_String/unknown(99) --- PASS: TestCompressionCodec_String (0.00s) --- PASS: TestCompressionCodec_String/none (0.00s) --- PASS: TestCompressionCodec_String/gzip (0.00s) --- PASS: TestCompressionCodec_String/snappy (0.00s) --- PASS: TestCompressionCodec_String/lz4 (0.00s) --- PASS: TestCompressionCodec_String/zstd (0.00s) --- PASS: TestCompressionCodec_String/unknown(99) (0.00s) === RUN TestCompressionCodec_IsValid === RUN TestCompressionCodec_IsValid/none === RUN TestCompressionCodec_IsValid/gzip === RUN TestCompressionCodec_IsValid/snappy === RUN TestCompressionCodec_IsValid/lz4 === RUN TestCompressionCodec_IsValid/zstd === RUN TestCompressionCodec_IsValid/unknown(-1) === RUN TestCompressionCodec_IsValid/unknown(5) === RUN TestCompressionCodec_IsValid/unknown(99) --- PASS: TestCompressionCodec_IsValid (0.00s) --- PASS: TestCompressionCodec_IsValid/none (0.00s) --- PASS: TestCompressionCodec_IsValid/gzip (0.00s) --- PASS: TestCompressionCodec_IsValid/snappy (0.00s) --- PASS: TestCompressionCodec_IsValid/lz4 (0.00s) --- PASS: TestCompressionCodec_IsValid/zstd (0.00s) --- PASS: TestCompressionCodec_IsValid/unknown(-1) (0.00s) --- PASS: TestCompressionCodec_IsValid/unknown(5) (0.00s) --- PASS: TestCompressionCodec_IsValid/unknown(99) (0.00s) === RUN TestExtractCompressionCodec === RUN TestExtractCompressionCodec/None === RUN TestExtractCompressionCodec/Gzip === RUN TestExtractCompressionCodec/Snappy === RUN TestExtractCompressionCodec/Lz4 === RUN TestExtractCompressionCodec/Zstd === RUN TestExtractCompressionCodec/Gzip_with_transactional === RUN TestExtractCompressionCodec/Snappy_with_control === RUN TestExtractCompressionCodec/Lz4_with_both_flags --- PASS: TestExtractCompressionCodec (0.00s) --- PASS: TestExtractCompressionCodec/None (0.00s) --- PASS: TestExtractCompressionCodec/Gzip (0.00s) --- PASS: TestExtractCompressionCodec/Snappy (0.00s) --- PASS: TestExtractCompressionCodec/Lz4 (0.00s) --- PASS: TestExtractCompressionCodec/Zstd (0.00s) --- PASS: TestExtractCompressionCodec/Gzip_with_transactional (0.00s) --- PASS: TestExtractCompressionCodec/Snappy_with_control (0.00s) --- PASS: TestExtractCompressionCodec/Lz4_with_both_flags (0.00s) === RUN TestSetCompressionCodec === RUN TestSetCompressionCodec/Set_None === RUN TestSetCompressionCodec/Set_Gzip === RUN TestSetCompressionCodec/Set_Snappy === RUN TestSetCompressionCodec/Set_Lz4 === RUN TestSetCompressionCodec/Set_Zstd === RUN TestSetCompressionCodec/Replace_Gzip_with_Snappy === RUN TestSetCompressionCodec/Set_Gzip_preserving_transactional === RUN TestSetCompressionCodec/Set_Lz4_preserving_control === RUN TestSetCompressionCodec/Set_Zstd_preserving_both_flags --- PASS: TestSetCompressionCodec (0.00s) --- PASS: TestSetCompressionCodec/Set_None (0.00s) --- PASS: TestSetCompressionCodec/Set_Gzip (0.00s) --- PASS: TestSetCompressionCodec/Set_Snappy (0.00s) --- PASS: TestSetCompressionCodec/Set_Lz4 (0.00s) --- PASS: TestSetCompressionCodec/Set_Zstd (0.00s) --- PASS: TestSetCompressionCodec/Replace_Gzip_with_Snappy (0.00s) --- PASS: TestSetCompressionCodec/Set_Gzip_preserving_transactional (0.00s) --- PASS: TestSetCompressionCodec/Set_Lz4_preserving_control (0.00s) --- PASS: TestSetCompressionCodec/Set_Zstd_preserving_both_flags (0.00s) === RUN TestCompress_None --- PASS: TestCompress_None (0.00s) === RUN TestCompress_Gzip --- PASS: TestCompress_Gzip (1.15s) === RUN TestCompress_Snappy --- PASS: TestCompress_Snappy (0.00s) === RUN TestCompress_Lz4 --- PASS: TestCompress_Lz4 (0.51s) === RUN TestCompress_Zstd --- PASS: TestCompress_Zstd (0.26s) === RUN TestCompress_InvalidCodec --- PASS: TestCompress_InvalidCodec (0.00s) === RUN TestDecompress_None --- PASS: TestDecompress_None (0.00s) === RUN TestRoundTrip === RUN TestRoundTrip/none === RUN TestRoundTrip/none/data_0 === RUN TestRoundTrip/none/data_1 === RUN TestRoundTrip/none/data_2 === RUN TestRoundTrip/none/data_3 === RUN TestRoundTrip/none/data_4 === RUN TestRoundTrip/none/data_5 === RUN TestRoundTrip/gzip === RUN TestRoundTrip/gzip/data_0 === RUN TestRoundTrip/gzip/data_1 === RUN TestRoundTrip/gzip/data_2 === RUN TestRoundTrip/gzip/data_3 === RUN TestRoundTrip/gzip/data_4 === RUN TestRoundTrip/gzip/data_5 === RUN TestRoundTrip/snappy === RUN TestRoundTrip/snappy/data_0 === RUN TestRoundTrip/snappy/data_1 === RUN TestRoundTrip/snappy/data_2 === RUN TestRoundTrip/snappy/data_3 === RUN TestRoundTrip/snappy/data_4 === RUN TestRoundTrip/snappy/data_5 === RUN TestRoundTrip/lz4 === RUN TestRoundTrip/lz4/data_0 === RUN TestRoundTrip/lz4/data_1 === RUN TestRoundTrip/lz4/data_2 === RUN TestRoundTrip/lz4/data_3 === RUN TestRoundTrip/lz4/data_4 === RUN TestRoundTrip/lz4/data_5 === RUN TestRoundTrip/zstd === RUN TestRoundTrip/zstd/data_0 === RUN TestRoundTrip/zstd/data_1 === RUN TestRoundTrip/zstd/data_2 === RUN TestRoundTrip/zstd/data_3 === RUN TestRoundTrip/zstd/data_4 === RUN TestRoundTrip/zstd/data_5 --- PASS: TestRoundTrip (1.01s) --- PASS: TestRoundTrip/none (0.00s) --- PASS: TestRoundTrip/none/data_0 (0.00s) --- PASS: TestRoundTrip/none/data_1 (0.00s) --- PASS: TestRoundTrip/none/data_2 (0.00s) --- PASS: TestRoundTrip/none/data_3 (0.00s) --- PASS: TestRoundTrip/none/data_4 (0.00s) --- PASS: TestRoundTrip/none/data_5 (0.00s) --- PASS: TestRoundTrip/gzip (0.43s) --- PASS: TestRoundTrip/gzip/data_0 (0.01s) --- PASS: TestRoundTrip/gzip/data_1 (0.01s) --- PASS: TestRoundTrip/gzip/data_2 (0.01s) --- PASS: TestRoundTrip/gzip/data_3 (0.22s) --- PASS: TestRoundTrip/gzip/data_4 (0.12s) --- PASS: TestRoundTrip/gzip/data_5 (0.04s) --- PASS: TestRoundTrip/snappy (0.01s) --- PASS: TestRoundTrip/snappy/data_0 (0.00s) --- PASS: TestRoundTrip/snappy/data_1 (0.00s) --- PASS: TestRoundTrip/snappy/data_2 (0.00s) --- PASS: TestRoundTrip/snappy/data_3 (0.00s) --- PASS: TestRoundTrip/snappy/data_4 (0.00s) --- PASS: TestRoundTrip/snappy/data_5 (0.00s) --- PASS: TestRoundTrip/lz4 (0.03s) --- PASS: TestRoundTrip/lz4/data_0 (0.02s) --- PASS: TestRoundTrip/lz4/data_1 (0.00s) --- PASS: TestRoundTrip/lz4/data_2 (0.00s) --- PASS: TestRoundTrip/lz4/data_3 (0.01s) --- PASS: TestRoundTrip/lz4/data_4 (0.00s) --- PASS: TestRoundTrip/lz4/data_5 (0.00s) --- PASS: TestRoundTrip/zstd (0.52s) --- PASS: TestRoundTrip/zstd/data_0 (0.04s) --- PASS: TestRoundTrip/zstd/data_1 (0.02s) --- PASS: TestRoundTrip/zstd/data_2 (0.02s) --- PASS: TestRoundTrip/zstd/data_3 (0.37s) --- PASS: TestRoundTrip/zstd/data_4 (0.01s) --- PASS: TestRoundTrip/zstd/data_5 (0.05s) === RUN TestDecompress_InvalidCodec --- PASS: TestDecompress_InvalidCodec (0.00s) === RUN TestDecompress_CorruptedData === RUN TestDecompress_CorruptedData/gzip === RUN TestDecompress_CorruptedData/snappy === RUN TestDecompress_CorruptedData/lz4 === RUN TestDecompress_CorruptedData/zstd --- PASS: TestDecompress_CorruptedData (0.01s) --- PASS: TestDecompress_CorruptedData/gzip (0.01s) --- PASS: TestDecompress_CorruptedData/snappy (0.00s) --- PASS: TestDecompress_CorruptedData/lz4 (0.00s) --- PASS: TestDecompress_CorruptedData/zstd (0.00s) === RUN TestCompressRecordBatch === RUN TestCompressRecordBatch/None_codec === RUN TestCompressRecordBatch/Gzip_codec === RUN TestCompressRecordBatch/Snappy_codec --- PASS: TestCompressRecordBatch (0.01s) --- PASS: TestCompressRecordBatch/None_codec (0.00s) --- PASS: TestCompressRecordBatch/Gzip_codec (0.00s) --- PASS: TestCompressRecordBatch/Snappy_codec (0.00s) === RUN TestDecompressRecordBatch === RUN TestDecompressRecordBatch/None_codec === RUN TestDecompressRecordBatch/Round_trip_with_Gzip === RUN TestDecompressRecordBatch/Round_trip_with_Snappy --- PASS: TestDecompressRecordBatch (0.00s) --- PASS: TestDecompressRecordBatch/None_codec (0.00s) --- PASS: TestDecompressRecordBatch/Round_trip_with_Gzip (0.00s) --- PASS: TestDecompressRecordBatch/Round_trip_with_Snappy (0.00s) === RUN TestCompressionEfficiency === RUN TestCompressionEfficiency/gzip compression_test.go:304: Codec: gzip, Original: 5100 bytes, Compressed: 95 bytes, Ratio: 0.02 === RUN TestCompressionEfficiency/snappy compression_test.go:304: Codec: snappy, Original: 5100 bytes, Compressed: 291 bytes, Ratio: 0.06 === RUN TestCompressionEfficiency/lz4 compression_test.go:304: Codec: lz4, Original: 5100 bytes, Compressed: 112 bytes, Ratio: 0.02 === RUN TestCompressionEfficiency/zstd compression_test.go:304: Codec: zstd, Original: 5100 bytes, Compressed: 68 bytes, Ratio: 0.01 --- PASS: TestCompressionEfficiency (0.31s) --- PASS: TestCompressionEfficiency/gzip (0.00s) --- PASS: TestCompressionEfficiency/snappy (0.00s) --- PASS: TestCompressionEfficiency/lz4 (0.02s) --- PASS: TestCompressionEfficiency/zstd (0.28s) PASS ok github.com/seaweedfs/seaweedfs/weed/mq/kafka/compression 7.531s === RUN TestRangeAssignmentStrategy --- PASS: TestRangeAssignmentStrategy (0.00s) === RUN TestRangeAssignmentStrategy_UnevenPartitions --- PASS: TestRangeAssignmentStrategy_UnevenPartitions (0.00s) === RUN TestRangeAssignmentStrategy_MultipleTopics --- PASS: TestRangeAssignmentStrategy_MultipleTopics (0.00s) === RUN TestRoundRobinAssignmentStrategy --- PASS: TestRoundRobinAssignmentStrategy (0.00s) === RUN TestRoundRobinAssignmentStrategy_MultipleTopics --- PASS: TestRoundRobinAssignmentStrategy_MultipleTopics (0.00s) === RUN TestGetAssignmentStrategy --- PASS: TestGetAssignmentStrategy (0.00s) === RUN TestConsumerGroup_AssignPartitions --- PASS: TestConsumerGroup_AssignPartitions (0.00s) === RUN TestConsumerGroup_GetMemberAssignments --- PASS: TestConsumerGroup_GetMemberAssignments (0.00s) === RUN TestConsumerGroup_UpdateMemberSubscription --- PASS: TestConsumerGroup_UpdateMemberSubscription (0.00s) === RUN TestAssignmentStrategy_EmptyMembers --- PASS: TestAssignmentStrategy_EmptyMembers (0.00s) === RUN TestCooperativeStickyAssignmentStrategy_Name --- PASS: TestCooperativeStickyAssignmentStrategy_Name (0.00s) === RUN TestCooperativeStickyAssignmentStrategy_InitialAssignment --- PASS: TestCooperativeStickyAssignmentStrategy_InitialAssignment (0.00s) === RUN TestCooperativeStickyAssignmentStrategy_StickyBehavior --- PASS: TestCooperativeStickyAssignmentStrategy_StickyBehavior (0.00s) === RUN TestCooperativeStickyAssignmentStrategy_NewMemberJoin cooperative_sticky_test.go:187: Member1 retained 2 out of 4 original partitions --- PASS: TestCooperativeStickyAssignmentStrategy_NewMemberJoin (0.00s) === RUN TestCooperativeStickyAssignmentStrategy_MemberLeave --- PASS: TestCooperativeStickyAssignmentStrategy_MemberLeave (0.00s) === RUN TestCooperativeStickyAssignmentStrategy_MultipleTopics --- PASS: TestCooperativeStickyAssignmentStrategy_MultipleTopics (0.00s) === RUN TestCooperativeStickyAssignmentStrategy_UnevenPartitions --- PASS: TestCooperativeStickyAssignmentStrategy_UnevenPartitions (0.00s) === RUN TestCooperativeStickyAssignmentStrategy_PartialSubscription --- PASS: TestCooperativeStickyAssignmentStrategy_PartialSubscription (0.00s) === RUN TestGetAssignmentStrategy_CooperativeSticky --- PASS: TestGetAssignmentStrategy_CooperativeSticky (0.00s) === RUN TestGroupCoordinator_CreateGroup --- PASS: TestGroupCoordinator_CreateGroup (0.00s) === RUN TestGroupCoordinator_ValidateSessionTimeout --- PASS: TestGroupCoordinator_ValidateSessionTimeout (0.00s) === RUN TestGroupCoordinator_MemberManagement --- PASS: TestGroupCoordinator_MemberManagement (0.00s) === RUN TestGroupCoordinator_Stats --- PASS: TestGroupCoordinator_Stats (0.00s) === RUN TestGroupCoordinator_Cleanup --- PASS: TestGroupCoordinator_Cleanup (0.00s) === RUN TestGroupCoordinator_GenerateMemberID --- PASS: TestGroupCoordinator_GenerateMemberID (0.00s) === RUN TestGroupCoordinator_EvictExpiredMembersLocked --- PASS: TestGroupCoordinator_EvictExpiredMembersLocked (0.00s) === RUN TestGroupCoordinator_Cleanup_SurvivorsRebalance --- PASS: TestGroupCoordinator_Cleanup_SurvivorsRebalance (0.00s) === RUN TestGroupCoordinator_EvictExpiredMembersLocked_ZeroSessionTimeout --- PASS: TestGroupCoordinator_EvictExpiredMembersLocked_ZeroSessionTimeout (0.00s) === RUN TestIncrementalCooperativeAssignmentStrategy_BasicAssignment incremental_rebalancing_test.go:41: Member member-1 assigned 2 partitions: [{topic-1 0} {topic-1 1}] incremental_rebalancing_test.go:41: Member member-2 assigned 2 partitions: [{topic-1 2} {topic-1 3}] --- PASS: TestIncrementalCooperativeAssignmentStrategy_BasicAssignment (0.00s) === RUN TestIncrementalCooperativeAssignmentStrategy_RebalanceWithRevocation incremental_rebalancing_test.go:106: Revocation phase - Member-1: 2 partitions, Member-2: 0 partitions incremental_rebalancing_test.go:139: Final assignment - Member-1: 2 partitions, Member-2: 2 partitions --- PASS: TestIncrementalCooperativeAssignmentStrategy_RebalanceWithRevocation (0.02s) === RUN TestIncrementalCooperativeAssignmentStrategy_NoRevocationNeeded --- PASS: TestIncrementalCooperativeAssignmentStrategy_NoRevocationNeeded (0.00s) === RUN TestIncrementalCooperativeAssignmentStrategy_MultipleTopics incremental_rebalancing_test.go:244: All assigned partitions: map[topic-1:0:true topic-1:1:true topic-1:2:true topic-2:0:true topic-2:1:true] --- PASS: TestIncrementalCooperativeAssignmentStrategy_MultipleTopics (0.00s) === RUN TestIncrementalCooperativeAssignmentStrategy_ForceComplete --- PASS: TestIncrementalCooperativeAssignmentStrategy_ForceComplete (0.00s) === RUN TestIncrementalCooperativeAssignmentStrategy_RevocationTimeout --- PASS: TestIncrementalCooperativeAssignmentStrategy_RevocationTimeout (0.01s) === RUN TestIncrementalCooperativeAssignmentStrategy_StateTransitions --- PASS: TestIncrementalCooperativeAssignmentStrategy_StateTransitions (0.00s) === RUN TestRebalanceTimeoutManager_CheckRebalanceTimeouts --- PASS: TestRebalanceTimeoutManager_CheckRebalanceTimeouts (0.00s) === RUN TestRebalanceTimeoutManager_SessionTimeoutFallback --- PASS: TestRebalanceTimeoutManager_SessionTimeoutFallback (0.00s) === RUN TestRebalanceTimeoutManager_LeaderEviction --- PASS: TestRebalanceTimeoutManager_LeaderEviction (0.00s) === RUN TestRebalanceTimeoutManager_IsRebalanceStuck --- PASS: TestRebalanceTimeoutManager_IsRebalanceStuck (0.00s) === RUN TestRebalanceTimeoutManager_ForceCompleteRebalance --- PASS: TestRebalanceTimeoutManager_ForceCompleteRebalance (0.00s) === RUN TestRebalanceTimeoutManager_GetRebalanceStatus --- PASS: TestRebalanceTimeoutManager_GetRebalanceStatus (0.00s) === RUN TestRebalanceTimeoutManager_DefaultRebalanceTimeout --- PASS: TestRebalanceTimeoutManager_DefaultRebalanceTimeout (0.00s) === RUN TestGroupCoordinator_StaticMembership --- PASS: TestGroupCoordinator_StaticMembership (0.00s) === RUN TestGroupCoordinator_StaticMemberReconnection --- PASS: TestGroupCoordinator_StaticMemberReconnection (0.00s) === RUN TestGroupCoordinator_StaticMembershipEdgeCases --- PASS: TestGroupCoordinator_StaticMembershipEdgeCases (0.00s) === RUN TestGroupCoordinator_StaticMembershipConcurrency --- PASS: TestGroupCoordinator_StaticMembershipConcurrency (0.06s) PASS ok github.com/seaweedfs/seaweedfs/weed/mq/kafka/consumer 0.718s === RUN TestMemoryStorageCommitAndFetch --- PASS: TestMemoryStorageCommitAndFetch (0.01s) === RUN TestMemoryStorageFetchNonExistent --- PASS: TestMemoryStorageFetchNonExistent (0.00s) === RUN TestMemoryStorageFetchAllOffsets --- PASS: TestMemoryStorageFetchAllOffsets (0.00s) === RUN TestMemoryStorageDeleteGroup --- PASS: TestMemoryStorageDeleteGroup (0.00s) === RUN TestMemoryStorageListGroups --- PASS: TestMemoryStorageListGroups (0.00s) === RUN TestMemoryStorageConcurrency --- PASS: TestMemoryStorageConcurrency (0.01s) === RUN TestMemoryStorageInvalidInputs --- PASS: TestMemoryStorageInvalidInputs (0.00s) === RUN TestMemoryStorageClosedOperations --- PASS: TestMemoryStorageClosedOperations (0.00s) === RUN TestMemoryStorageOverwrite --- PASS: TestMemoryStorageOverwrite (0.00s) PASS ok github.com/seaweedfs/seaweedfs/weed/mq/kafka/consumer_offset 17.535s === RUN TestCoordinatorRegistry_DeterministicNodeID --- PASS: TestCoordinatorRegistry_DeterministicNodeID (0.00s) === RUN TestCoordinatorRegistry_BasicOperations --- PASS: TestCoordinatorRegistry_BasicOperations (0.00s) === RUN TestCoordinatorRegistry_AssignCoordinator --- PASS: TestCoordinatorRegistry_AssignCoordinator (0.03s) === RUN TestCoordinatorRegistry_HealthyGateways --- PASS: TestCoordinatorRegistry_HealthyGateways (0.00s) === RUN TestCoordinatorRegistry_ConsistentHashing --- PASS: TestCoordinatorRegistry_ConsistentHashing (0.00s) === RUN TestCoordinatorRegistry_CleanupStaleEntries --- PASS: TestCoordinatorRegistry_CleanupStaleEntries (0.00s) === RUN TestCoordinatorRegistry_GetStats --- PASS: TestCoordinatorRegistry_GetStats (0.00s) === RUN TestCoordinatorRegistry_HeartbeatGateway --- PASS: TestCoordinatorRegistry_HeartbeatGateway (0.02s) PASS ok github.com/seaweedfs/seaweedfs/weed/mq/kafka/gateway 14.975s === RUN TestNeedsRestart === RUN TestNeedsRestart/Stream_is_nil_-_needs_restart === RUN TestNeedsRestart/Offset_in_cache_-_no_restart_needed === RUN TestNeedsRestart/Offset_before_current_-_needs_restart === RUN TestNeedsRestart/Large_gap_ahead_-_needs_restart === RUN TestNeedsRestart/Small_gap_ahead_-_no_restart_needed === RUN TestNeedsRestart/Exact_match_-_no_restart_needed === RUN TestNeedsRestart/Context_is_nil_-_needs_restart --- PASS: TestNeedsRestart (0.07s) --- PASS: TestNeedsRestart/Stream_is_nil_-_needs_restart (0.00s) --- PASS: TestNeedsRestart/Offset_in_cache_-_no_restart_needed (0.00s) --- PASS: TestNeedsRestart/Offset_before_current_-_needs_restart (0.00s) --- PASS: TestNeedsRestart/Large_gap_ahead_-_needs_restart (0.00s) --- PASS: TestNeedsRestart/Small_gap_ahead_-_no_restart_needed (0.00s) --- PASS: TestNeedsRestart/Exact_match_-_no_restart_needed (0.00s) --- PASS: TestNeedsRestart/Context_is_nil_-_needs_restart (0.00s) === RUN TestNeedsRestart_CacheLogic === RUN TestNeedsRestart_CacheLogic/First_offset_in_cache === RUN TestNeedsRestart_CacheLogic/Middle_offset_in_cache === RUN TestNeedsRestart_CacheLogic/Last_offset_in_cache === RUN TestNeedsRestart_CacheLogic/Before_cache_start === RUN TestNeedsRestart_CacheLogic/Current_position === RUN TestNeedsRestart_CacheLogic/One_ahead === RUN TestNeedsRestart_CacheLogic/Large_gap_>_1000 --- PASS: TestNeedsRestart_CacheLogic (0.00s) --- PASS: TestNeedsRestart_CacheLogic/First_offset_in_cache (0.00s) --- PASS: TestNeedsRestart_CacheLogic/Middle_offset_in_cache (0.00s) --- PASS: TestNeedsRestart_CacheLogic/Last_offset_in_cache (0.00s) --- PASS: TestNeedsRestart_CacheLogic/Before_cache_start (0.00s) --- PASS: TestNeedsRestart_CacheLogic/Current_position (0.00s) --- PASS: TestNeedsRestart_CacheLogic/One_ahead (0.00s) --- PASS: TestNeedsRestart_CacheLogic/Large_gap_>_1000 (0.00s) === RUN TestNeedsRestart_EmptyCache === RUN TestNeedsRestart_EmptyCache/Before_current === RUN TestNeedsRestart_EmptyCache/At_current === RUN TestNeedsRestart_EmptyCache/Small_gap_ahead === RUN TestNeedsRestart_EmptyCache/Large_gap_ahead --- PASS: TestNeedsRestart_EmptyCache (0.00s) --- PASS: TestNeedsRestart_EmptyCache/Before_current (0.00s) --- PASS: TestNeedsRestart_EmptyCache/At_current (0.00s) --- PASS: TestNeedsRestart_EmptyCache/Small_gap_ahead (0.00s) --- PASS: TestNeedsRestart_EmptyCache/Large_gap_ahead (0.00s) === RUN TestNeedsRestart_ThreadSafety --- PASS: TestNeedsRestart_ThreadSafety (0.00s) === RUN TestRestartSubscriber_StateManagement --- PASS: TestRestartSubscriber_StateManagement (0.00s) === RUN TestMapBrokerErrorToKafka === RUN TestMapBrokerErrorToKafka/No_error === RUN TestMapBrokerErrorToKafka/Unknown_server_error === RUN TestMapBrokerErrorToKafka/Topic_not_found === RUN TestMapBrokerErrorToKafka/Partition_not_found === RUN TestMapBrokerErrorToKafka/Not_leader_or_follower === RUN TestMapBrokerErrorToKafka/Request_timed_out === RUN TestMapBrokerErrorToKafka/Broker_not_available === RUN TestMapBrokerErrorToKafka/Message_too_large === RUN TestMapBrokerErrorToKafka/Network_exception === RUN TestMapBrokerErrorToKafka/Offset_load_in_progress === RUN TestMapBrokerErrorToKafka/Invalid_record === RUN TestMapBrokerErrorToKafka/Topic_already_exists === RUN TestMapBrokerErrorToKafka/Invalid_partitions === RUN TestMapBrokerErrorToKafka/Invalid_config === RUN TestMapBrokerErrorToKafka/Publisher_not_found === RUN TestMapBrokerErrorToKafka/Connection_failed === RUN TestMapBrokerErrorToKafka/Follower_connection_failed === RUN TestMapBrokerErrorToKafka/Unknown_error_code --- PASS: TestMapBrokerErrorToKafka (0.01s) --- PASS: TestMapBrokerErrorToKafka/No_error (0.00s) --- PASS: TestMapBrokerErrorToKafka/Unknown_server_error (0.00s) --- PASS: TestMapBrokerErrorToKafka/Topic_not_found (0.00s) --- PASS: TestMapBrokerErrorToKafka/Partition_not_found (0.00s) --- PASS: TestMapBrokerErrorToKafka/Not_leader_or_follower (0.00s) --- PASS: TestMapBrokerErrorToKafka/Request_timed_out (0.00s) --- PASS: TestMapBrokerErrorToKafka/Broker_not_available (0.00s) --- PASS: TestMapBrokerErrorToKafka/Message_too_large (0.00s) --- PASS: TestMapBrokerErrorToKafka/Network_exception (0.00s) --- PASS: TestMapBrokerErrorToKafka/Offset_load_in_progress (0.00s) --- PASS: TestMapBrokerErrorToKafka/Invalid_record (0.00s) --- PASS: TestMapBrokerErrorToKafka/Topic_already_exists (0.00s) --- PASS: TestMapBrokerErrorToKafka/Invalid_partitions (0.00s) --- PASS: TestMapBrokerErrorToKafka/Invalid_config (0.00s) --- PASS: TestMapBrokerErrorToKafka/Publisher_not_found (0.00s) --- PASS: TestMapBrokerErrorToKafka/Connection_failed (0.00s) --- PASS: TestMapBrokerErrorToKafka/Follower_connection_failed (0.00s) --- PASS: TestMapBrokerErrorToKafka/Unknown_error_code (0.00s) === RUN TestHandleBrokerResponse === RUN TestHandleBrokerResponse/No_error === RUN TestHandleBrokerResponse/Structured_error_-_Not_leader === RUN TestHandleBrokerResponse/Structured_error_-_Topic_not_found === RUN TestHandleBrokerResponse/Fallback_string_parsing_-_Not_leader === RUN TestHandleBrokerResponse/Fallback_string_parsing_-_Topic_not_found === RUN TestHandleBrokerResponse/Fallback_string_parsing_-_Unknown_error --- PASS: TestHandleBrokerResponse (0.00s) --- PASS: TestHandleBrokerResponse/No_error (0.00s) --- PASS: TestHandleBrokerResponse/Structured_error_-_Not_leader (0.00s) --- PASS: TestHandleBrokerResponse/Structured_error_-_Topic_not_found (0.00s) --- PASS: TestHandleBrokerResponse/Fallback_string_parsing_-_Not_leader (0.00s) --- PASS: TestHandleBrokerResponse/Fallback_string_parsing_-_Topic_not_found (0.00s) --- PASS: TestHandleBrokerResponse/Fallback_string_parsing_-_Unknown_error (0.00s) === RUN TestParseStringErrorToKafkaCode === RUN TestParseStringErrorToKafkaCode/Empty_error === RUN TestParseStringErrorToKafkaCode/Not_leader_error === RUN TestParseStringErrorToKafkaCode/Not_leader_error_variant === RUN TestParseStringErrorToKafkaCode/Topic_not_found === RUN TestParseStringErrorToKafkaCode/Topic_does_not_exist === RUN TestParseStringErrorToKafkaCode/Partition_not_found === RUN TestParseStringErrorToKafkaCode/Timeout_error === RUN TestParseStringErrorToKafkaCode/Timeout_error_variant === RUN TestParseStringErrorToKafkaCode/Network_error === RUN TestParseStringErrorToKafkaCode/Connection_error === RUN TestParseStringErrorToKafkaCode/Message_too_large === RUN TestParseStringErrorToKafkaCode/Size_error === RUN TestParseStringErrorToKafkaCode/Unknown_error --- PASS: TestParseStringErrorToKafkaCode (0.00s) --- PASS: TestParseStringErrorToKafkaCode/Empty_error (0.00s) --- PASS: TestParseStringErrorToKafkaCode/Not_leader_error (0.00s) --- PASS: TestParseStringErrorToKafkaCode/Not_leader_error_variant (0.00s) --- PASS: TestParseStringErrorToKafkaCode/Topic_not_found (0.00s) --- PASS: TestParseStringErrorToKafkaCode/Topic_does_not_exist (0.00s) --- PASS: TestParseStringErrorToKafkaCode/Partition_not_found (0.00s) --- PASS: TestParseStringErrorToKafkaCode/Timeout_error (0.00s) --- PASS: TestParseStringErrorToKafkaCode/Timeout_error_variant (0.00s) --- PASS: TestParseStringErrorToKafkaCode/Network_error (0.00s) --- PASS: TestParseStringErrorToKafkaCode/Connection_error (0.00s) --- PASS: TestParseStringErrorToKafkaCode/Message_too_large (0.00s) --- PASS: TestParseStringErrorToKafkaCode/Size_error (0.00s) --- PASS: TestParseStringErrorToKafkaCode/Unknown_error (0.00s) === RUN TestAdaptiveFetchTimeout fetch_performance_test.go:11: Testing adaptive fetch timeout strategy... === RUN TestAdaptiveFetchTimeout/OldStrategy_50ms_Timeout fetch_performance_test.go:32: Record 1 timed out (readTime=150ms > timeout=50ms) fetch_performance_test.go:38: Old strategy: received 0/4 records in 2.033019ms fetch_performance_test.go:43: ✓ Bug reproduced: old strategy times out too quickly === RUN TestAdaptiveFetchTimeout/NewStrategy_1s_Timeout fetch_performance_test.go:56: Record 1 received (readTime=150ms) fetch_performance_test.go:56: Record 2 received (readTime=150ms) fetch_performance_test.go:56: Record 3 received (readTime=150ms) fetch_performance_test.go:56: Record 4 received (readTime=150ms) fetch_performance_test.go:64: New strategy: received 4/4 records in 176.24µs fetch_performance_test.go:69: ✓ Fix verified: new strategy receives all records === RUN TestAdaptiveFetchTimeout/SchemaRegistry_CatchUp_Scenario fetch_performance_test.go:99: Schema Registry catch-up simulation: fetch_performance_test.go:100: Old strategy: 4 round trips, ~800ms total time fetch_performance_test.go:101: New strategy: 1 round trip, ~600ms total time fetch_performance_test.go:102: Schema Registry timeout: 500ms fetch_performance_test.go:108: ✓ Old strategy exceeds timeout: 800ms > 500ms fetch_performance_test.go:112: ✓ New strategy completes within timeout: 600ms <= 700ms --- PASS: TestAdaptiveFetchTimeout (0.01s) --- PASS: TestAdaptiveFetchTimeout/OldStrategy_50ms_Timeout (0.00s) --- PASS: TestAdaptiveFetchTimeout/NewStrategy_1s_Timeout (0.00s) --- PASS: TestAdaptiveFetchTimeout/SchemaRegistry_CatchUp_Scenario (0.00s) === RUN TestFetchTimeoutProgression fetch_performance_test.go:121: Testing fetch timeout progression... fetch_performance_test.go:134: Timeout progression: fetch_performance_test.go:137: Record 1: timeout = 1s fetch_performance_test.go:137: Record 2: timeout = 1s fetch_performance_test.go:137: Record 3: timeout = 1s fetch_performance_test.go:137: Record 4: timeout = 1s fetch_performance_test.go:137: Record 5: timeout = 1s fetch_performance_test.go:137: Record 6: timeout = 100ms fetch_performance_test.go:137: Record 7: timeout = 100ms fetch_performance_test.go:137: Record 8: timeout = 100ms fetch_performance_test.go:137: Record 9: timeout = 100ms fetch_performance_test.go:137: Record 10: timeout = 100ms fetch_performance_test.go:154: ✓ Timeout progression is correct --- PASS: TestFetchTimeoutProgression (0.00s) === RUN TestSeaweedSMQRecord_Interface --- PASS: TestSeaweedSMQRecord_Interface (0.00s) === RUN TestSeaweedMQHandler_GetStoredRecords_EmptyTopic record_retrieval_test.go:96: Test obsolete: ledgers removed, SMQ broker handles offset management --- SKIP: TestSeaweedMQHandler_GetStoredRecords_EmptyTopic (0.00s) === RUN TestSeaweedMQHandler_GetStoredRecords_EmptyPartition record_retrieval_test.go:102: Test obsolete: ledgers removed, SMQ broker handles offset management --- SKIP: TestSeaweedMQHandler_GetStoredRecords_EmptyPartition (0.00s) === RUN TestSeaweedMQHandler_GetStoredRecords_OffsetBeyondHighWaterMark record_retrieval_test.go:108: Test obsolete: ledgers removed, SMQ broker handles offset management --- SKIP: TestSeaweedMQHandler_GetStoredRecords_OffsetBeyondHighWaterMark (0.00s) === RUN TestSeaweedMQHandler_GetStoredRecords_MaxRecordsLimit record_retrieval_test.go:114: Test obsolete: ledgers removed, SMQ broker handles offset management --- SKIP: TestSeaweedMQHandler_GetStoredRecords_MaxRecordsLimit (0.00s) === RUN TestSeaweedMQHandler_MapSeaweedToKafkaOffsets seaweedmq_handler_test.go:15: Test obsolete: ledger system removed, SMQ uses native offsets --- SKIP: TestSeaweedMQHandler_MapSeaweedToKafkaOffsets (0.00s) === RUN TestSeaweedMQHandler_MapSeaweedToKafkaOffsets_EmptyRecords seaweedmq_handler_test.go:21: Test obsolete: ledger system removed, SMQ uses native offsets --- SKIP: TestSeaweedMQHandler_MapSeaweedToKafkaOffsets_EmptyRecords (0.00s) === RUN TestSeaweedMQHandler_ConvertSeaweedToKafkaRecordBatch seaweedmq_handler_test.go:68: Successfully converted 2 records to 119 byte batch --- PASS: TestSeaweedMQHandler_ConvertSeaweedToKafkaRecordBatch (0.00s) === RUN TestSeaweedMQHandler_ConvertSeaweedToKafkaRecordBatch_EmptyRecords --- PASS: TestSeaweedMQHandler_ConvertSeaweedToKafkaRecordBatch_EmptyRecords (0.00s) === RUN TestSeaweedMQHandler_ConvertSingleSeaweedRecord === RUN TestSeaweedMQHandler_ConvertSingleSeaweedRecord/Record_with_key_and_value seaweedmq_handler_test.go:161: Successfully converted single record: 24 bytes === RUN TestSeaweedMQHandler_ConvertSingleSeaweedRecord/Record_with_null_key seaweedmq_handler_test.go:161: Successfully converted single record: 23 bytes === RUN TestSeaweedMQHandler_ConvertSingleSeaweedRecord/Record_with_empty_value seaweedmq_handler_test.go:161: Successfully converted single record: 26 bytes --- PASS: TestSeaweedMQHandler_ConvertSingleSeaweedRecord (0.03s) --- PASS: TestSeaweedMQHandler_ConvertSingleSeaweedRecord/Record_with_key_and_value (0.03s) --- PASS: TestSeaweedMQHandler_ConvertSingleSeaweedRecord/Record_with_null_key (0.00s) --- PASS: TestSeaweedMQHandler_ConvertSingleSeaweedRecord/Record_with_empty_value (0.00s) === RUN TestSeaweedMQHandler_Creation seaweedmq_handler_test.go:171: Integration test requires real SeaweedMQ Broker - run manually with broker available --- SKIP: TestSeaweedMQHandler_Creation (0.00s) === RUN TestSeaweedMQHandler_TopicLifecycle seaweedmq_handler_test.go:190: Integration test requires real SeaweedMQ Broker - run manually with broker available --- SKIP: TestSeaweedMQHandler_TopicLifecycle (0.00s) === RUN TestSeaweedMQHandler_ProduceRecord seaweedmq_handler_test.go:252: Integration test requires real SeaweedMQ Broker - run manually with broker available --- SKIP: TestSeaweedMQHandler_ProduceRecord (0.00s) === RUN TestSeaweedMQHandler_MultiplePartitions seaweedmq_handler_test.go:297: Integration test requires real SeaweedMQ Broker - run manually with broker available --- SKIP: TestSeaweedMQHandler_MultiplePartitions (0.00s) === RUN TestSeaweedMQHandler_FetchRecords seaweedmq_handler_test.go:341: Integration test requires real SeaweedMQ Broker - run manually with broker available --- SKIP: TestSeaweedMQHandler_FetchRecords (0.00s) === RUN TestSeaweedMQHandler_FetchRecords_ErrorHandling seaweedmq_handler_test.go:435: Integration test requires real SeaweedMQ Broker - run manually with broker available --- SKIP: TestSeaweedMQHandler_FetchRecords_ErrorHandling (0.00s) === RUN TestSeaweedMQHandler_ErrorHandling seaweedmq_handler_test.go:485: Integration test requires real SeaweedMQ Broker - run manually with broker available --- SKIP: TestSeaweedMQHandler_ErrorHandling (0.00s) PASS ok github.com/seaweedfs/seaweedfs/weed/mq/kafka/integration 13.916s === RUN TestBatchConstruction batch_crc_compat_test.go:24: Batch size: 86 bytes batch_crc_compat_test.go:25: Batch hex: 0000: 00000000000000000000004affffffff 0016: 02f4623d47000000000000000001a035 0032: da7324000001a035da7324ffffffffff 0048: ffffffffffffffffff00000001300000 0064: 0010746573742d6b657914746573742d 0080: 76616c756500 batch_crc_compat_test.go:33: Stored CRC: 0xf4623d47 batch_crc_compat_test.go:38: Calculated CRC: 0xf4623d47 (over 65 bytes) batch_crc_compat_test.go:57: CRC verification PASSED batch_crc_compat_test.go:61: === Batch Structure === batch_crc_compat_test.go:238: Base Offset: 0000000000000000 (value: 0) batch_crc_compat_test.go:238: Batch Length: 0000004a (value: 74) batch_crc_compat_test.go:238: Leader Epoch: ffffffff (value: -1) batch_crc_compat_test.go:238: Magic: 02 (value: 2) batch_crc_compat_test.go:238: CRC: f4623d47 (value: 4100078919) batch_crc_compat_test.go:238: Attributes: 0000 (value: 0) batch_crc_compat_test.go:238: Last Offset Delta: 00000000 (value: 0) batch_crc_compat_test.go:238: Base Timestamp: 000001a035da7324 (value: 1787609903908) batch_crc_compat_test.go:238: Max Timestamp: 000001a035da7324 (value: 1787609903908) batch_crc_compat_test.go:238: Record Count: 00000001 (value: 1) batch_crc_compat_test.go:79: Batch length correct: 74 --- PASS: TestBatchConstruction (0.03s) === RUN TestMultipleRecordsBatch batch_crc_compat_test.go:93: Batch 1 size: 78, CRC: 0x5613d682 batch_crc_compat_test.go:94: Batch 2 size: 78, CRC: 0x7b5b7232 batch_crc_compat_test.go:104: Batch 1 CRC valid batch_crc_compat_test.go:104: Batch 2 CRC valid --- PASS: TestMultipleRecordsBatch (0.00s) === RUN TestVarintEncoding batch_crc_compat_test.go:131: encodeVarint(0) = 00 batch_crc_compat_test.go:131: encodeVarint(1) = 02 batch_crc_compat_test.go:131: encodeVarint(-1) = 01 batch_crc_compat_test.go:131: encodeVarint(5) = 0a batch_crc_compat_test.go:131: encodeVarint(-5) = 09 batch_crc_compat_test.go:131: encodeVarint(127) = fe01 batch_crc_compat_test.go:131: encodeVarint(128) = 8002 batch_crc_compat_test.go:131: encodeVarint(-127) = fd01 batch_crc_compat_test.go:131: encodeVarint(-128) = ff01 --- PASS: TestVarintEncoding (0.00s) === RUN TestClientSideCRCValidation batch_crc_compat_test.go:259: Constructed batch: 86 bytes batch_crc_compat_test.go:268: Client read CRC from header: 0x0c0eaea2 batch_crc_compat_test.go:272: Client calculated CRC: 0x0c0eaea2 batch_crc_compat_test.go:280: CLIENT WOULD ACCEPT: CRC valid --- PASS: TestClientSideCRCValidation (0.00s) === RUN TestConcurrentBatchConstruction batch_crc_compat_test.go:316: All 10 concurrent batches have valid CRCs --- PASS: TestConcurrentBatchConstruction (0.00s) === RUN TestProductionBatchConstruction batch_crc_compat_test.go:338: Production batch size: 86 bytes batch_crc_compat_test.go:348: Production batch CRC: stored=0xa499aaed calculated=0xa499aaed batch_crc_compat_test.go:354: PRODUCTION CODE CRC VALID --- PASS: TestProductionBatchConstruction (0.00s) === RUN TestHeartbeatResponseFormat_V0 --- PASS: TestHeartbeatResponseFormat_V0 (0.00s) === RUN TestHeartbeatResponseFormat_V1ToV3 === RUN TestHeartbeatResponseFormat_V1ToV3/v1 === RUN TestHeartbeatResponseFormat_V1ToV3/v2 === RUN TestHeartbeatResponseFormat_V1ToV3/v3 --- PASS: TestHeartbeatResponseFormat_V1ToV3 (0.00s) --- PASS: TestHeartbeatResponseFormat_V1ToV3/v1 (0.00s) --- PASS: TestHeartbeatResponseFormat_V1ToV3/v2 (0.00s) --- PASS: TestHeartbeatResponseFormat_V1ToV3/v3 (0.00s) === RUN TestHeartbeatResponseFormat_V4Plus === RUN TestHeartbeatResponseFormat_V4Plus/v4 --- PASS: TestHeartbeatResponseFormat_V4Plus (0.00s) --- PASS: TestHeartbeatResponseFormat_V4Plus/v4 (0.00s) === RUN TestHeartbeatResponseFormat_ErrorCode === RUN TestHeartbeatResponseFormat_ErrorCode/None === RUN TestHeartbeatResponseFormat_ErrorCode/UnknownMemberID === RUN TestHeartbeatResponseFormat_ErrorCode/IllegalGeneration === RUN TestHeartbeatResponseFormat_ErrorCode/RebalanceInProgress --- PASS: TestHeartbeatResponseFormat_ErrorCode (0.00s) --- PASS: TestHeartbeatResponseFormat_ErrorCode/None (0.00s) --- PASS: TestHeartbeatResponseFormat_ErrorCode/UnknownMemberID (0.00s) --- PASS: TestHeartbeatResponseFormat_ErrorCode/IllegalGeneration (0.00s) --- PASS: TestHeartbeatResponseFormat_ErrorCode/RebalanceInProgress (0.00s) === RUN TestHeartbeatResponseFormat_BugReproduce heartbeat_response_format_test.go:157: This test documents the original bug - skip to avoid false failures --- SKIP: TestHeartbeatResponseFormat_BugReproduce (0.00s) === RUN TestMetadataRequestBlocking metadata_blocking_test.go:22: This test documents the original bug. The fix is in the broker's ListTopics with filer timeout. Run TestMetadataRequestWithFastMock to verify fast path works. --- SKIP: TestMetadataRequestBlocking (0.00s) === RUN TestMetadataRequestWithFastMock metadata_blocking_test.go:68: Testing Metadata handler with fast-responding backend... metadata_blocking_test.go:88: ✓ Metadata completed in 49.48µs (283 bytes) --- PASS: TestMetadataRequestWithFastMock (0.00s) === RUN TestMetadataRequestWithTimeoutFix metadata_blocking_test.go:98: Testing Metadata handler with timeout-aware backend... metadata_blocking_test.go:115: Metadata completed in 2.014860695s metadata_blocking_test.go:121: ✓ Metadata returned response (33 bytes) without blocking --- PASS: TestMetadataRequestWithTimeoutFix (2.02s) === RUN TestOffsetCommitFetchPattern offset_fetch_pattern_test.go:18: Integration test - requires mock broker setup --- SKIP: TestOffsetCommitFetchPattern (0.00s) === RUN TestOffsetFetchAfterCommit offset_fetch_pattern_test.go:98: Integration test - requires mock broker setup --- SKIP: TestOffsetFetchAfterCommit (0.00s) === RUN TestOffsetPersistencePattern offset_fetch_pattern_test.go:126: Integration test - requires mock broker setup --- SKIP: TestOffsetPersistencePattern (0.00s) === RUN TestOffsetCommitConsistency offset_fetch_pattern_test.go:167: Integration test - requires mock broker setup --- SKIP: TestOffsetCommitConsistency (0.00s) === RUN TestLongPollWithOffsetCommit offset_fetch_pattern_test.go:200: Integration test - requires mock broker setup --- SKIP: TestLongPollWithOffsetCommit (0.00s) === RUN TestRecordBatchParser_ParseRecordBatch --- PASS: TestRecordBatchParser_ParseRecordBatch (0.00s) === RUN TestRecordBatchParser_ParseRecordBatch_TooSmall --- PASS: TestRecordBatchParser_ParseRecordBatch_TooSmall (0.00s) === RUN TestRecordBatchParser_ParseRecordBatch_InvalidMagic --- PASS: TestRecordBatchParser_ParseRecordBatch_InvalidMagic (0.00s) === RUN TestRecordBatchParser_Compression === RUN TestRecordBatchParser_Compression/none === RUN TestRecordBatchParser_Compression/gzip === RUN TestRecordBatchParser_Compression/snappy === RUN TestRecordBatchParser_Compression/lz4 === RUN TestRecordBatchParser_Compression/zstd --- PASS: TestRecordBatchParser_Compression (0.18s) --- PASS: TestRecordBatchParser_Compression/none (0.00s) --- PASS: TestRecordBatchParser_Compression/gzip (0.01s) --- PASS: TestRecordBatchParser_Compression/snappy (0.00s) --- PASS: TestRecordBatchParser_Compression/lz4 (0.10s) --- PASS: TestRecordBatchParser_Compression/zstd (0.07s) === RUN TestRecordBatchParser_CRCValidation === RUN TestRecordBatchParser_CRCValidation/Valid_CRC === RUN TestRecordBatchParser_CRCValidation/Invalid_CRC === RUN TestRecordBatchParser_CRCValidation/Skip_CRC_validation --- PASS: TestRecordBatchParser_CRCValidation (0.00s) --- PASS: TestRecordBatchParser_CRCValidation/Valid_CRC (0.00s) --- PASS: TestRecordBatchParser_CRCValidation/Invalid_CRC (0.00s) --- PASS: TestRecordBatchParser_CRCValidation/Skip_CRC_validation (0.00s) === RUN TestRecordBatchParser_ExtractRecords --- PASS: TestRecordBatchParser_ExtractRecords (0.00s) === RUN TestCompressRecordBatch === RUN TestCompressRecordBatch/No_compression === RUN TestCompressRecordBatch/Gzip_compression --- PASS: TestCompressRecordBatch (0.01s) --- PASS: TestCompressRecordBatch/No_compression (0.00s) --- PASS: TestCompressRecordBatch/Gzip_compression (0.01s) === RUN TestCreateRecordBatch === RUN TestCreateRecordBatch/Uncompressed_batch === RUN TestCreateRecordBatch/Compressed_batch --- PASS: TestCreateRecordBatch (0.00s) --- PASS: TestCreateRecordBatch/Uncompressed_batch (0.00s) --- PASS: TestCreateRecordBatch/Compressed_batch (0.00s) === RUN TestRecordBatchParser_InvalidRecordCount --- PASS: TestRecordBatchParser_InvalidRecordCount (0.00s) === RUN TestExtractAllRecords_RealKafkaFormat record_extraction_test.go:125: Created batch of 85 bytes, record value: {"type":"string"} record_extraction_test.go:150: Successfully extracted record with value: {"type":"string"} --- PASS: TestExtractAllRecords_RealKafkaFormat (0.00s) === RUN TestExtractAllRecords_CompressedBatch record_extraction_test.go:157: Compressed batch test - implement after uncompressed works --- SKIP: TestExtractAllRecords_CompressedBatch (0.00s) === RUN TestResponseFormatsNoCorrelationID === RUN TestResponseFormatsNoCorrelationID/ApiVersions_v0 response_format_test.go:153: Testing ApiVersions_v0: ApiVersions v0 should not include correlation ID in body response_format_test.go:160: ✓ API Key 18 Version 0: Correlation ID should be handled by writeResponseWithHeader === RUN TestResponseFormatsNoCorrelationID/ApiVersions_v4 response_format_test.go:153: Testing ApiVersions_v4: ApiVersions v4 (flexible) should not include correlation ID in body response_format_test.go:160: ✓ API Key 18 Version 4: Correlation ID should be handled by writeResponseWithHeader === RUN TestResponseFormatsNoCorrelationID/Metadata_v0 response_format_test.go:153: Testing Metadata_v0: Metadata v0 should not include correlation ID in body response_format_test.go:160: ✓ API Key 3 Version 0: Correlation ID should be handled by writeResponseWithHeader === RUN TestResponseFormatsNoCorrelationID/Metadata_v7 response_format_test.go:153: Testing Metadata_v7: Metadata v7 should not include correlation ID in body response_format_test.go:160: ✓ API Key 3 Version 7: Correlation ID should be handled by writeResponseWithHeader === RUN TestResponseFormatsNoCorrelationID/FindCoordinator_v0 response_format_test.go:153: Testing FindCoordinator_v0: FindCoordinator v0 should not include correlation ID in body response_format_test.go:160: ✓ API Key 10 Version 0: Correlation ID should be handled by writeResponseWithHeader === RUN TestResponseFormatsNoCorrelationID/FindCoordinator_v2 response_format_test.go:153: Testing FindCoordinator_v2: FindCoordinator v2 should not include correlation ID in body response_format_test.go:160: ✓ API Key 10 Version 2: Correlation ID should be handled by writeResponseWithHeader === RUN TestResponseFormatsNoCorrelationID/DescribeConfigs_v0 response_format_test.go:153: Testing DescribeConfigs_v0: DescribeConfigs v0 should not include correlation ID in body response_format_test.go:160: ✓ API Key 32 Version 0: Correlation ID should be handled by writeResponseWithHeader === RUN TestResponseFormatsNoCorrelationID/DescribeConfigs_v4 response_format_test.go:153: Testing DescribeConfigs_v4: DescribeConfigs v4 (flexible) should not include correlation ID in body response_format_test.go:160: ✓ API Key 32 Version 4: Correlation ID should be handled by writeResponseWithHeader === RUN TestResponseFormatsNoCorrelationID/DescribeCluster_v0 response_format_test.go:153: Testing DescribeCluster_v0: DescribeCluster v0 (flexible) should not include correlation ID in body response_format_test.go:160: ✓ API Key 60 Version 0: Correlation ID should be handled by writeResponseWithHeader === RUN TestResponseFormatsNoCorrelationID/InitProducerId_v0 response_format_test.go:153: Testing InitProducerId_v0: InitProducerId v0 should not include correlation ID in body response_format_test.go:160: ✓ API Key 22 Version 0: Correlation ID should be handled by writeResponseWithHeader === RUN TestResponseFormatsNoCorrelationID/InitProducerId_v4 response_format_test.go:153: Testing InitProducerId_v4: InitProducerId v4 (flexible) should not include correlation ID in body response_format_test.go:160: ✓ API Key 22 Version 4: Correlation ID should be handled by writeResponseWithHeader === RUN TestResponseFormatsNoCorrelationID/JoinGroup_v0 response_format_test.go:153: Testing JoinGroup_v0: JoinGroup v0 should not include correlation ID in body response_format_test.go:160: ✓ API Key 11 Version 0: Correlation ID should be handled by writeResponseWithHeader === RUN TestResponseFormatsNoCorrelationID/SyncGroup_v0 response_format_test.go:153: Testing SyncGroup_v0: SyncGroup v0 should not include correlation ID in body response_format_test.go:160: ✓ API Key 14 Version 0: Correlation ID should be handled by writeResponseWithHeader === RUN TestResponseFormatsNoCorrelationID/Heartbeat_v0 response_format_test.go:153: Testing Heartbeat_v0: Heartbeat v0 should not include correlation ID in body response_format_test.go:160: ✓ API Key 12 Version 0: Correlation ID should be handled by writeResponseWithHeader === RUN TestResponseFormatsNoCorrelationID/LeaveGroup_v0 response_format_test.go:153: Testing LeaveGroup_v0: LeaveGroup v0 should not include correlation ID in body response_format_test.go:160: ✓ API Key 13 Version 0: Correlation ID should be handled by writeResponseWithHeader === RUN TestResponseFormatsNoCorrelationID/OffsetFetch_v0 response_format_test.go:153: Testing OffsetFetch_v0: OffsetFetch v0 should not include correlation ID in body response_format_test.go:160: ✓ API Key 9 Version 0: Correlation ID should be handled by writeResponseWithHeader === RUN TestResponseFormatsNoCorrelationID/OffsetCommit_v0 response_format_test.go:153: Testing OffsetCommit_v0: OffsetCommit v0 should not include correlation ID in body response_format_test.go:160: ✓ API Key 8 Version 0: Correlation ID should be handled by writeResponseWithHeader === RUN TestResponseFormatsNoCorrelationID/Produce_v0 response_format_test.go:153: Testing Produce_v0: Produce v0 should not include correlation ID in body response_format_test.go:160: ✓ API Key 0 Version 0: Correlation ID should be handled by writeResponseWithHeader === RUN TestResponseFormatsNoCorrelationID/Produce_v7 response_format_test.go:153: Testing Produce_v7: Produce v7 should not include correlation ID in body response_format_test.go:160: ✓ API Key 0 Version 7: Correlation ID should be handled by writeResponseWithHeader === RUN TestResponseFormatsNoCorrelationID/Fetch_v0 response_format_test.go:153: Testing Fetch_v0: Fetch v0 should not include correlation ID in body response_format_test.go:160: ✓ API Key 1 Version 0: Correlation ID should be handled by writeResponseWithHeader === RUN TestResponseFormatsNoCorrelationID/Fetch_v7 response_format_test.go:153: Testing Fetch_v7: Fetch v7 should not include correlation ID in body response_format_test.go:160: ✓ API Key 1 Version 7: Correlation ID should be handled by writeResponseWithHeader --- PASS: TestResponseFormatsNoCorrelationID (0.02s) --- PASS: TestResponseFormatsNoCorrelationID/ApiVersions_v0 (0.00s) --- PASS: TestResponseFormatsNoCorrelationID/ApiVersions_v4 (0.00s) --- PASS: TestResponseFormatsNoCorrelationID/Metadata_v0 (0.00s) --- PASS: TestResponseFormatsNoCorrelationID/Metadata_v7 (0.00s) --- PASS: TestResponseFormatsNoCorrelationID/FindCoordinator_v0 (0.00s) --- PASS: TestResponseFormatsNoCorrelationID/FindCoordinator_v2 (0.00s) --- PASS: TestResponseFormatsNoCorrelationID/DescribeConfigs_v0 (0.00s) --- PASS: TestResponseFormatsNoCorrelationID/DescribeConfigs_v4 (0.00s) --- PASS: TestResponseFormatsNoCorrelationID/DescribeCluster_v0 (0.00s) --- PASS: TestResponseFormatsNoCorrelationID/InitProducerId_v0 (0.00s) --- PASS: TestResponseFormatsNoCorrelationID/InitProducerId_v4 (0.00s) --- PASS: TestResponseFormatsNoCorrelationID/JoinGroup_v0 (0.00s) --- PASS: TestResponseFormatsNoCorrelationID/SyncGroup_v0 (0.00s) --- PASS: TestResponseFormatsNoCorrelationID/Heartbeat_v0 (0.00s) --- PASS: TestResponseFormatsNoCorrelationID/LeaveGroup_v0 (0.00s) --- PASS: TestResponseFormatsNoCorrelationID/OffsetFetch_v0 (0.00s) --- PASS: TestResponseFormatsNoCorrelationID/OffsetCommit_v0 (0.00s) --- PASS: TestResponseFormatsNoCorrelationID/Produce_v0 (0.00s) --- PASS: TestResponseFormatsNoCorrelationID/Produce_v7 (0.00s) --- PASS: TestResponseFormatsNoCorrelationID/Fetch_v0 (0.00s) --- PASS: TestResponseFormatsNoCorrelationID/Fetch_v7 (0.00s) === RUN TestFlexibleResponseHeaderFormat === RUN TestFlexibleResponseHeaderFormat/ApiVersions_v0 response_format_test.go:252: ✓ ApiVersions_v0: correctly identified as flexible=false === RUN TestFlexibleResponseHeaderFormat/ApiVersions_v3 response_format_test.go:252: ✓ ApiVersions_v3: correctly identified as flexible=false === RUN TestFlexibleResponseHeaderFormat/ApiVersions_v4 response_format_test.go:252: ✓ ApiVersions_v4: correctly identified as flexible=false === RUN TestFlexibleResponseHeaderFormat/Metadata_v0 response_format_test.go:252: ✓ Metadata_v0: correctly identified as flexible=false === RUN TestFlexibleResponseHeaderFormat/Metadata_v7 response_format_test.go:252: ✓ Metadata_v7: correctly identified as flexible=false === RUN TestFlexibleResponseHeaderFormat/Metadata_v9 response_format_test.go:252: ✓ Metadata_v9: correctly identified as flexible=true === RUN TestFlexibleResponseHeaderFormat/Produce_v0 response_format_test.go:252: ✓ Produce_v0: correctly identified as flexible=false === RUN TestFlexibleResponseHeaderFormat/Produce_v7 response_format_test.go:252: ✓ Produce_v7: correctly identified as flexible=false === RUN TestFlexibleResponseHeaderFormat/Produce_v9 response_format_test.go:252: ✓ Produce_v9: correctly identified as flexible=true === RUN TestFlexibleResponseHeaderFormat/Fetch_v0 response_format_test.go:252: ✓ Fetch_v0: correctly identified as flexible=false === RUN TestFlexibleResponseHeaderFormat/Fetch_v7 response_format_test.go:252: ✓ Fetch_v7: correctly identified as flexible=false === RUN TestFlexibleResponseHeaderFormat/Fetch_v12 response_format_test.go:252: ✓ Fetch_v12: correctly identified as flexible=true === RUN TestFlexibleResponseHeaderFormat/FindCoordinator_v0 response_format_test.go:252: ✓ FindCoordinator_v0: correctly identified as flexible=false === RUN TestFlexibleResponseHeaderFormat/FindCoordinator_v2 response_format_test.go:252: ✓ FindCoordinator_v2: correctly identified as flexible=false === RUN TestFlexibleResponseHeaderFormat/FindCoordinator_v3 response_format_test.go:252: ✓ FindCoordinator_v3: correctly identified as flexible=true === RUN TestFlexibleResponseHeaderFormat/JoinGroup_v0 response_format_test.go:252: ✓ JoinGroup_v0: correctly identified as flexible=false === RUN TestFlexibleResponseHeaderFormat/JoinGroup_v5 response_format_test.go:252: ✓ JoinGroup_v5: correctly identified as flexible=false === RUN TestFlexibleResponseHeaderFormat/JoinGroup_v6 response_format_test.go:252: ✓ JoinGroup_v6: correctly identified as flexible=true === RUN TestFlexibleResponseHeaderFormat/SyncGroup_v0 response_format_test.go:252: ✓ SyncGroup_v0: correctly identified as flexible=false === RUN TestFlexibleResponseHeaderFormat/SyncGroup_v3 response_format_test.go:252: ✓ SyncGroup_v3: correctly identified as flexible=false === RUN TestFlexibleResponseHeaderFormat/SyncGroup_v4 response_format_test.go:252: ✓ SyncGroup_v4: correctly identified as flexible=true === RUN TestFlexibleResponseHeaderFormat/Heartbeat_v0 response_format_test.go:252: ✓ Heartbeat_v0: correctly identified as flexible=false === RUN TestFlexibleResponseHeaderFormat/Heartbeat_v3 response_format_test.go:252: ✓ Heartbeat_v3: correctly identified as flexible=false === RUN TestFlexibleResponseHeaderFormat/Heartbeat_v4 response_format_test.go:252: ✓ Heartbeat_v4: correctly identified as flexible=true === RUN TestFlexibleResponseHeaderFormat/LeaveGroup_v0 response_format_test.go:252: ✓ LeaveGroup_v0: correctly identified as flexible=false === RUN TestFlexibleResponseHeaderFormat/LeaveGroup_v3 response_format_test.go:252: ✓ LeaveGroup_v3: correctly identified as flexible=false === RUN TestFlexibleResponseHeaderFormat/LeaveGroup_v4 response_format_test.go:252: ✓ LeaveGroup_v4: correctly identified as flexible=true === RUN TestFlexibleResponseHeaderFormat/OffsetFetch_v0 response_format_test.go:252: ✓ OffsetFetch_v0: correctly identified as flexible=false === RUN TestFlexibleResponseHeaderFormat/OffsetFetch_v5 response_format_test.go:252: ✓ OffsetFetch_v5: correctly identified as flexible=false === RUN TestFlexibleResponseHeaderFormat/OffsetFetch_v6 response_format_test.go:252: ✓ OffsetFetch_v6: correctly identified as flexible=true === RUN TestFlexibleResponseHeaderFormat/OffsetCommit_v0 response_format_test.go:252: ✓ OffsetCommit_v0: correctly identified as flexible=false === RUN TestFlexibleResponseHeaderFormat/OffsetCommit_v7 response_format_test.go:252: ✓ OffsetCommit_v7: correctly identified as flexible=false === RUN TestFlexibleResponseHeaderFormat/OffsetCommit_v8 response_format_test.go:252: ✓ OffsetCommit_v8: correctly identified as flexible=true === RUN TestFlexibleResponseHeaderFormat/DescribeConfigs_v0 response_format_test.go:252: ✓ DescribeConfigs_v0: correctly identified as flexible=false === RUN TestFlexibleResponseHeaderFormat/DescribeConfigs_v3 response_format_test.go:252: ✓ DescribeConfigs_v3: correctly identified as flexible=false === RUN TestFlexibleResponseHeaderFormat/DescribeConfigs_v4 response_format_test.go:252: ✓ DescribeConfigs_v4: correctly identified as flexible=true === RUN TestFlexibleResponseHeaderFormat/InitProducerId_v0 response_format_test.go:252: ✓ InitProducerId_v0: correctly identified as flexible=false === RUN TestFlexibleResponseHeaderFormat/InitProducerId_v1 response_format_test.go:252: ✓ InitProducerId_v1: correctly identified as flexible=false === RUN TestFlexibleResponseHeaderFormat/InitProducerId_v2 response_format_test.go:252: ✓ InitProducerId_v2: correctly identified as flexible=true === RUN TestFlexibleResponseHeaderFormat/DescribeCluster_v0 response_format_test.go:252: ✓ DescribeCluster_v0: correctly identified as flexible=true === RUN TestFlexibleResponseHeaderFormat/DescribeCluster_v1 response_format_test.go:252: ✓ DescribeCluster_v1: correctly identified as flexible=true --- PASS: TestFlexibleResponseHeaderFormat (0.03s) --- PASS: TestFlexibleResponseHeaderFormat/ApiVersions_v0 (0.00s) --- PASS: TestFlexibleResponseHeaderFormat/ApiVersions_v3 (0.00s) --- PASS: TestFlexibleResponseHeaderFormat/ApiVersions_v4 (0.00s) --- PASS: TestFlexibleResponseHeaderFormat/Metadata_v0 (0.00s) --- PASS: TestFlexibleResponseHeaderFormat/Metadata_v7 (0.00s) --- PASS: TestFlexibleResponseHeaderFormat/Metadata_v9 (0.00s) --- PASS: TestFlexibleResponseHeaderFormat/Produce_v0 (0.00s) --- PASS: TestFlexibleResponseHeaderFormat/Produce_v7 (0.00s) --- PASS: TestFlexibleResponseHeaderFormat/Produce_v9 (0.00s) --- PASS: TestFlexibleResponseHeaderFormat/Fetch_v0 (0.00s) --- PASS: TestFlexibleResponseHeaderFormat/Fetch_v7 (0.00s) --- PASS: TestFlexibleResponseHeaderFormat/Fetch_v12 (0.00s) --- PASS: TestFlexibleResponseHeaderFormat/FindCoordinator_v0 (0.00s) --- PASS: TestFlexibleResponseHeaderFormat/FindCoordinator_v2 (0.00s) --- PASS: TestFlexibleResponseHeaderFormat/FindCoordinator_v3 (0.00s) --- PASS: TestFlexibleResponseHeaderFormat/JoinGroup_v0 (0.00s) --- PASS: TestFlexibleResponseHeaderFormat/JoinGroup_v5 (0.00s) --- PASS: TestFlexibleResponseHeaderFormat/JoinGroup_v6 (0.00s) --- PASS: TestFlexibleResponseHeaderFormat/SyncGroup_v0 (0.00s) --- PASS: TestFlexibleResponseHeaderFormat/SyncGroup_v3 (0.00s) --- PASS: TestFlexibleResponseHeaderFormat/SyncGroup_v4 (0.00s) --- PASS: TestFlexibleResponseHeaderFormat/Heartbeat_v0 (0.00s) --- PASS: TestFlexibleResponseHeaderFormat/Heartbeat_v3 (0.00s) --- PASS: TestFlexibleResponseHeaderFormat/Heartbeat_v4 (0.00s) --- PASS: TestFlexibleResponseHeaderFormat/LeaveGroup_v0 (0.00s) --- PASS: TestFlexibleResponseHeaderFormat/LeaveGroup_v3 (0.00s) --- PASS: TestFlexibleResponseHeaderFormat/LeaveGroup_v4 (0.00s) --- PASS: TestFlexibleResponseHeaderFormat/OffsetFetch_v0 (0.00s) --- PASS: TestFlexibleResponseHeaderFormat/OffsetFetch_v5 (0.00s) --- PASS: TestFlexibleResponseHeaderFormat/OffsetFetch_v6 (0.00s) --- PASS: TestFlexibleResponseHeaderFormat/OffsetCommit_v0 (0.00s) --- PASS: TestFlexibleResponseHeaderFormat/OffsetCommit_v7 (0.00s) --- PASS: TestFlexibleResponseHeaderFormat/OffsetCommit_v8 (0.00s) --- PASS: TestFlexibleResponseHeaderFormat/DescribeConfigs_v0 (0.00s) --- PASS: TestFlexibleResponseHeaderFormat/DescribeConfigs_v3 (0.00s) --- PASS: TestFlexibleResponseHeaderFormat/DescribeConfigs_v4 (0.00s) --- PASS: TestFlexibleResponseHeaderFormat/InitProducerId_v0 (0.00s) --- PASS: TestFlexibleResponseHeaderFormat/InitProducerId_v1 (0.00s) --- PASS: TestFlexibleResponseHeaderFormat/InitProducerId_v2 (0.00s) --- PASS: TestFlexibleResponseHeaderFormat/DescribeCluster_v0 (0.00s) --- PASS: TestFlexibleResponseHeaderFormat/DescribeCluster_v1 (0.00s) === RUN TestCorrelationIDNotInResponseBody === RUN TestCorrelationIDNotInResponseBody/DetectCorrelationIDInBody response_format_test.go:283: ✓ Successfully detected correlation ID in body (bad response) response_format_test.go:294: ✓ Correctly identified response without correlation ID --- PASS: TestCorrelationIDNotInResponseBody (0.00s) --- PASS: TestCorrelationIDNotInResponseBody/DetectCorrelationIDInBody (0.00s) === RUN TestWireProtocolFormat response_format_test.go:301: Kafka Wire Protocol Format (KIP-482): response_format_test.go:302: Non-flexible responses: response_format_test.go:303: [Size: 4 bytes][Correlation ID: 4 bytes][Response Body] response_format_test.go:304: response_format_test.go:305: Flexible responses (header version 1+): response_format_test.go:306: [Size: 4 bytes][Correlation ID: 4 bytes][Tagged Fields: 1+ bytes][Response Body] response_format_test.go:307: response_format_test.go:308: Size field: includes correlation ID + tagged fields + body response_format_test.go:309: Tagged Fields: varint-encoded, 0x00 for empty response_format_test.go:310: response_format_test.go:311: CRITICAL: Response body should NEVER include correlation ID! response_format_test.go:312: It is written ONLY by writeResponseWithHeader --- PASS: TestWireProtocolFormat (0.00s) === RUN TestJoinGroupResponseStructure response_validation_example_test.go:14: This is a demonstration test - shows what we SHOULD check --- SKIP: TestJoinGroupResponseStructure (0.00s) === RUN TestProduceResponseStructure response_validation_example_test.go:48: This is a demonstration test - shows what we SHOULD check --- SKIP: TestProduceResponseStructure (0.00s) === RUN TestCompareWithReferenceImplementation response_validation_example_test.go:64: This would require a reference Kafka broker or client library --- SKIP: TestCompareWithReferenceImplementation (0.00s) === RUN TestCurrentTestingApproach response_validation_example_test.go:82: Current testing strategy (as of Oct 2025): response_validation_example_test.go:83: response_validation_example_test.go:84: LEVEL 1: Static Code Analysis response_validation_example_test.go:85: Tool: check_responses.sh response_validation_example_test.go:86: Checks: Correlation ID patterns response_validation_example_test.go:87: Coverage: Good for known issues response_validation_example_test.go:88: response_validation_example_test.go:89: LEVEL 2: Protocol Format Tests response_validation_example_test.go:90: Tool: TestFlexibleResponseHeaderFormat response_validation_example_test.go:91: Checks: Flexible vs non-flexible classification response_validation_example_test.go:92: Coverage: Header format only response_validation_example_test.go:93: response_validation_example_test.go:94: LEVEL 3: Integration Testing response_validation_example_test.go:95: Tool: Schema Registry, kafka-go, Sarama, Java client response_validation_example_test.go:96: Checks: Real client compatibility response_validation_example_test.go:97: Coverage: Complete but requires manual debugging response_validation_example_test.go:98: response_validation_example_test.go:99: MISSING: Field-level response body validation response_validation_example_test.go:100: This is why JoinGroup issue wasn't caught by unit tests --- PASS: TestCurrentTestingApproach (0.00s) === RUN TestMetadataResponseHasBrokers response_validation_example_test.go:105: Example of what a real field-level test would look like --- SKIP: TestMetadataResponseHasBrokers (0.00s) === RUN TestSyncGroup_RaceCondition_BugDocumentation syncgroup_assignment_test.go:52: Original bug: Non-leader in Stable state would trigger server-side assignment syncgroup_assignment_test.go:53: This caused duplicate partition assignments and message re-reads (66.7% duplicates) syncgroup_assignment_test.go:54: Fix: Check if member is non-leader with empty assignments, regardless of group state --- PASS: TestSyncGroup_RaceCondition_BugDocumentation (0.00s) === RUN TestSyncGroup_FixVerification === RUN TestSyncGroup_FixVerification/Leader_with_assignments syncgroup_assignment_test.go:125: ✓ Leader provides client-side assignments, processes them: isLeader=true hasAssignments=true shouldWait=false === RUN TestSyncGroup_FixVerification/Non-leader_without_assignments_(PreparingRebalance) syncgroup_assignment_test.go:125: ✓ Non-leader waits for leader to provide assignments: isLeader=false hasAssignments=false shouldWait=true === RUN TestSyncGroup_FixVerification/Non-leader_without_assignments_(Stable)_-_THE_BUG_CASE syncgroup_assignment_test.go:125: ✓ Non-leader retrieves assignment from leader (already processed): isLeader=false hasAssignments=false shouldWait=true === RUN TestSyncGroup_FixVerification/Leader_without_assignments syncgroup_assignment_test.go:125: ✓ Edge case: server-side assignment (should not happen with Sarama): isLeader=true hasAssignments=false shouldWait=false --- PASS: TestSyncGroup_FixVerification (0.01s) --- PASS: TestSyncGroup_FixVerification/Leader_with_assignments (0.00s) --- PASS: TestSyncGroup_FixVerification/Non-leader_without_assignments_(PreparingRebalance) (0.00s) --- PASS: TestSyncGroup_FixVerification/Non-leader_without_assignments_(Stable)_-_THE_BUG_CASE (0.00s) --- PASS: TestSyncGroup_FixVerification/Leader_without_assignments (0.00s) === RUN TestParseSyncGroupRequestV5SkipsProtocolFields --- PASS: TestParseSyncGroupRequestV5SkipsProtocolFields (0.00s) === RUN TestSyncGroupSchemaRegistryV5LeaderAssignmentDoesNotRebalance --- PASS: TestSyncGroupSchemaRegistryV5LeaderAssignmentDoesNotRebalance (0.05s) === RUN TestEnsureTopicExistsTreatsAlreadyExistsAsSuccess --- PASS: TestEnsureTopicExistsTreatsAlreadyExistsAsSuccess (0.00s) === RUN TestEnsureTopicExistsReportsGenuineFailure --- PASS: TestEnsureTopicExistsReportsGenuineFailure (0.00s) === RUN TestEnsureTopicExistsWhenPresent --- PASS: TestEnsureTopicExistsWhenPresent (0.00s) PASS ok github.com/seaweedfs/seaweedfs/weed/mq/kafka/protocol 17.418s === RUN TestNewAvroDecoder === RUN TestNewAvroDecoder/valid_record_schema === RUN TestNewAvroDecoder/valid_enum_schema === RUN TestNewAvroDecoder/invalid_schema === RUN TestNewAvroDecoder/empty_schema --- PASS: TestNewAvroDecoder (0.01s) --- PASS: TestNewAvroDecoder/valid_record_schema (0.00s) --- PASS: TestNewAvroDecoder/valid_enum_schema (0.00s) --- PASS: TestNewAvroDecoder/invalid_schema (0.00s) --- PASS: TestNewAvroDecoder/empty_schema (0.00s) === RUN TestAvroDecoder_Decode --- PASS: TestAvroDecoder_Decode (0.00s) === RUN TestAvroDecoder_DecodeToRecordValue --- PASS: TestAvroDecoder_DecodeToRecordValue (0.00s) === RUN TestMapToRecordValue --- PASS: TestMapToRecordValue (0.02s) === RUN TestGoValueToSchemaValue === RUN TestGoValueToSchemaValue/nil_value === RUN TestGoValueToSchemaValue/bool_value === RUN TestGoValueToSchemaValue/int32_value === RUN TestGoValueToSchemaValue/int64_value === RUN TestGoValueToSchemaValue/string_value === RUN TestGoValueToSchemaValue/bytes_value === RUN TestGoValueToSchemaValue/time_value --- PASS: TestGoValueToSchemaValue (0.00s) --- PASS: TestGoValueToSchemaValue/nil_value (0.00s) --- PASS: TestGoValueToSchemaValue/bool_value (0.00s) --- PASS: TestGoValueToSchemaValue/int32_value (0.00s) --- PASS: TestGoValueToSchemaValue/int64_value (0.00s) --- PASS: TestGoValueToSchemaValue/string_value (0.00s) --- PASS: TestGoValueToSchemaValue/bytes_value (0.00s) --- PASS: TestGoValueToSchemaValue/time_value (0.00s) === RUN TestInferRecordTypeFromMap --- PASS: TestInferRecordTypeFromMap (0.02s) === RUN TestInferTypeFromValue === RUN TestInferTypeFromValue/nil === RUN TestInferTypeFromValue/bool === RUN TestInferTypeFromValue/int32 === RUN TestInferTypeFromValue/int64 === RUN TestInferTypeFromValue/int === RUN TestInferTypeFromValue/float32 === RUN TestInferTypeFromValue/float64 === RUN TestInferTypeFromValue/string === RUN TestInferTypeFromValue/bytes === RUN TestInferTypeFromValue/time --- PASS: TestInferTypeFromValue (0.00s) --- PASS: TestInferTypeFromValue/nil (0.00s) --- PASS: TestInferTypeFromValue/bool (0.00s) --- PASS: TestInferTypeFromValue/int32 (0.00s) --- PASS: TestInferTypeFromValue/int64 (0.00s) --- PASS: TestInferTypeFromValue/int (0.00s) --- PASS: TestInferTypeFromValue/float32 (0.00s) --- PASS: TestInferTypeFromValue/float64 (0.00s) --- PASS: TestInferTypeFromValue/string (0.00s) --- PASS: TestInferTypeFromValue/bytes (0.00s) --- PASS: TestInferTypeFromValue/time (0.00s) === RUN TestAvroDecoder_Integration --- PASS: TestAvroDecoder_Integration (0.00s) === RUN TestBrokerClient_FetchIntegration === RUN TestBrokerClient_FetchIntegration/Fetch_Schema_Integration I0824 22:18:25.897088 connect_to_sub_coordinator.go:33 broker coordinator on localhost:17777: getOrCreateConnection localhost:17777: fail to dial localhost:17777: grpc: no transport security set (use grpc.WithTransportCredentials(insecure.NewCredentials()) explicitly or set credentials) I0824 22:18:25.899948 connect_to_sub_coordinator.go:100 subscriber kafka.fetch-test-topic/kafka-gateway waiting for more assignments broker_client_fetch_test.go:57: Fetch integration test completed - connection failed as expected with mock broker: failed to get subscriber for topic fetch-test-topic: failed to connect to brokers: connection timeout === RUN TestBrokerClient_FetchIntegration/Envelope_Reconstruction broker_client_fetch_test.go:88: Expected error in envelope reconstruction due to schema mismatch: failed to encode RecordValue: failed to encode to Avro binary: cannot encode binary record "FetchTest" field "id": schema does not specify default value and no value provided === RUN TestBrokerClient_FetchIntegration/Subscriber_Management I0824 22:18:25.991260 connect_to_sub_coordinator.go:33 broker coordinator on localhost:17777: getOrCreateConnection localhost:17777: fail to dial localhost:17777: grpc: no transport security set (use grpc.WithTransportCredentials(insecure.NewCredentials()) explicitly or set credentials) I0824 22:18:25.991306 connect_to_sub_coordinator.go:100 subscriber kafka.subscriber-test-topic/kafka-gateway waiting for more assignments broker_client_fetch_test.go:106: Subscriber creation failed as expected with mock brokers: failed to connect to brokers: connection timeout broker_client_fetch_test.go:118: Active subscribers: 0 --- PASS: TestBrokerClient_FetchIntegration (0.37s) --- PASS: TestBrokerClient_FetchIntegration/Fetch_Schema_Integration (0.20s) --- PASS: TestBrokerClient_FetchIntegration/Envelope_Reconstruction (0.04s) --- PASS: TestBrokerClient_FetchIntegration/Subscriber_Management (0.10s) === RUN TestBrokerClient_RoundTripIntegration === RUN TestBrokerClient_RoundTripIntegration/Complete_Schema_Workflow broker_client_fetch_test.go:182: Round-trip test completed - schema validation and processing successful === RUN TestBrokerClient_RoundTripIntegration/Error_Handling_in_Fetch I0824 22:18:26.114942 connect_to_sub_coordinator.go:33 broker coordinator on localhost:17777: getOrCreateConnection localhost:17777: fail to dial localhost:17777: grpc: no transport security set (use grpc.WithTransportCredentials(insecure.NewCredentials()) explicitly or set credentials) I0824 22:18:26.115027 connect_to_sub_coordinator.go:100 subscriber kafka.non-existent-topic/kafka-gateway waiting for more assignments broker_client_fetch_test.go:200: Reconstruction result: failed to encode RecordValue: failed to get schema for encoding: schema registry error 404: {"error_code": 40403, "message": "Schema not found"} --- PASS: TestBrokerClient_RoundTripIntegration (0.13s) --- PASS: TestBrokerClient_RoundTripIntegration/Complete_Schema_Workflow (0.01s) --- PASS: TestBrokerClient_RoundTripIntegration/Error_Handling_in_Fetch (0.11s) === RUN TestBrokerClient_SubscriberConfiguration === RUN TestBrokerClient_SubscriberConfiguration/Subscriber_Cache_Management I0824 22:18:26.234359 connect_to_sub_coordinator.go:33 broker coordinator on localhost:17777: getOrCreateConnection localhost:17777: fail to dial localhost:17777: grpc: no transport security set (use grpc.WithTransportCredentials(insecure.NewCredentials()) explicitly or set credentials) I0824 22:18:26.234460 connect_to_sub_coordinator.go:100 subscriber kafka.cache-test-topic/kafka-gateway waiting for more assignments I0824 22:18:26.335300 connect_to_sub_coordinator.go:33 broker coordinator on localhost:17777: getOrCreateConnection localhost:17777: fail to dial localhost:17777: grpc: no transport security set (use grpc.WithTransportCredentials(insecure.NewCredentials()) explicitly or set credentials) I0824 22:18:26.335474 connect_to_sub_coordinator.go:100 subscriber kafka.cache-test-topic/kafka-gateway waiting for more assignments broker_client_fetch_test.go:230: Subscriber creation results: err1=failed to connect to brokers: connection timeout, err2=failed to connect to brokers: connection timeout broker_client_fetch_test.go:235: Broker client remains functional after subscriber creation attempts === RUN TestBrokerClient_SubscriberConfiguration/Multiple_Topic_Subscribers I0824 22:18:26.447853 connect_to_sub_coordinator.go:33 broker coordinator on localhost:17777: getOrCreateConnection localhost:17777: fail to dial localhost:17777: grpc: no transport security set (use grpc.WithTransportCredentials(insecure.NewCredentials()) explicitly or set credentials) I0824 22:18:26.448034 connect_to_sub_coordinator.go:100 subscriber kafka.topic-a/kafka-gateway waiting for more assignments broker_client_fetch_test.go:244: Subscriber creation for topic-a: failed to connect to brokers: connection timeout I0824 22:18:26.555267 connect_to_sub_coordinator.go:33 broker coordinator on localhost:17777: getOrCreateConnection localhost:17777: fail to dial localhost:17777: grpc: no transport security set (use grpc.WithTransportCredentials(insecure.NewCredentials()) explicitly or set credentials) I0824 22:18:26.555508 connect_to_sub_coordinator.go:100 subscriber kafka.topic-b/kafka-gateway waiting for more assignments broker_client_fetch_test.go:244: Subscriber creation for topic-b: failed to connect to brokers: connection timeout I0824 22:18:26.665110 connect_to_sub_coordinator.go:33 broker coordinator on localhost:17777: getOrCreateConnection localhost:17777: fail to dial localhost:17777: grpc: no transport security set (use grpc.WithTransportCredentials(insecure.NewCredentials()) explicitly or set credentials) I0824 22:18:26.665162 connect_to_sub_coordinator.go:100 subscriber kafka.topic-c/kafka-gateway waiting for more assignments broker_client_fetch_test.go:244: Subscriber creation for topic-c: failed to connect to brokers: connection timeout --- PASS: TestBrokerClient_SubscriberConfiguration (0.54s) --- PASS: TestBrokerClient_SubscriberConfiguration/Subscriber_Cache_Management (0.21s) --- PASS: TestBrokerClient_SubscriberConfiguration/Multiple_Topic_Subscribers (0.33s) === RUN TestBrokerClient_SchematizedMessage === RUN TestBrokerClient_SchematizedMessage/Avro_Schematized_Message broker_client_test.go:78: Known issue: Integer value decoded as 0 instead of 42 broker_client_test.go:87: Successfully validated schematized message with schema ID 1 === RUN TestBrokerClient_SchematizedMessage/RecordType_Creation === RUN TestBrokerClient_SchematizedMessage/Publisher_Stats --- PASS: TestBrokerClient_SchematizedMessage (0.07s) --- PASS: TestBrokerClient_SchematizedMessage/Avro_Schematized_Message (0.04s) --- PASS: TestBrokerClient_SchematizedMessage/RecordType_Creation (0.02s) --- PASS: TestBrokerClient_SchematizedMessage/Publisher_Stats (0.00s) === RUN TestBrokerClient_ErrorHandling === RUN TestBrokerClient_ErrorHandling/Invalid_Schematized_Message === RUN TestBrokerClient_ErrorHandling/Non-Schematized_Message === RUN TestBrokerClient_ErrorHandling/Unknown_Schema_ID === RUN TestBrokerClient_ErrorHandling/Invalid_RecordType_Creation --- PASS: TestBrokerClient_ErrorHandling (0.05s) --- PASS: TestBrokerClient_ErrorHandling/Invalid_Schematized_Message (0.01s) --- PASS: TestBrokerClient_ErrorHandling/Non-Schematized_Message (0.00s) --- PASS: TestBrokerClient_ErrorHandling/Unknown_Schema_ID (0.00s) --- PASS: TestBrokerClient_ErrorHandling/Invalid_RecordType_Creation (0.04s) === RUN TestBrokerClient_Integration === RUN TestBrokerClient_Integration/Multiple_Schema_Formats broker_client_test.go:251: Successfully validated JSON Schema message with schema ID 11 === RUN TestBrokerClient_Integration/Cache_Behavior --- PASS: TestBrokerClient_Integration (0.02s) --- PASS: TestBrokerClient_Integration/Multiple_Schema_Formats (0.01s) --- PASS: TestBrokerClient_Integration/Cache_Behavior (0.00s) === RUN TestBasicSchemaDecodeEncode === RUN TestBasicSchemaDecodeEncode/Simple_Avro_String_Record === RUN TestBasicSchemaDecodeEncode/JSON_Schema_with_String_Field === RUN TestBasicSchemaDecodeEncode/Cache_Performance --- PASS: TestBasicSchemaDecodeEncode (0.05s) --- PASS: TestBasicSchemaDecodeEncode/Simple_Avro_String_Record (0.01s) --- PASS: TestBasicSchemaDecodeEncode/JSON_Schema_with_String_Field (0.02s) --- PASS: TestBasicSchemaDecodeEncode/Cache_Performance (0.02s) === RUN TestSchemaValidation === RUN TestSchemaValidation/Valid_Schema_Message === RUN TestSchemaValidation/Non-Schematized_Message === RUN TestSchemaValidation/Invalid_Envelope --- PASS: TestSchemaValidation (0.03s) --- PASS: TestSchemaValidation/Valid_Schema_Message (0.03s) --- PASS: TestSchemaValidation/Non-Schematized_Message (0.00s) --- PASS: TestSchemaValidation/Invalid_Envelope (0.00s) === RUN TestSchemaDecodeEncode_Avro === RUN TestSchemaDecodeEncode_Avro/Simple_User_Record === RUN TestSchemaDecodeEncode_Avro/Complex_Record_with_Arrays === RUN TestSchemaDecodeEncode_Avro/Union_Types --- PASS: TestSchemaDecodeEncode_Avro (0.02s) --- PASS: TestSchemaDecodeEncode_Avro/Simple_User_Record (0.01s) --- PASS: TestSchemaDecodeEncode_Avro/Complex_Record_with_Arrays (0.01s) --- PASS: TestSchemaDecodeEncode_Avro/Union_Types (0.01s) === RUN TestSchemaDecodeEncode_JSONSchema === RUN TestSchemaDecodeEncode_JSONSchema/Product_Schema === RUN TestSchemaDecodeEncode_JSONSchema/Nested_Object_Schema --- PASS: TestSchemaDecodeEncode_JSONSchema (0.02s) --- PASS: TestSchemaDecodeEncode_JSONSchema/Product_Schema (0.01s) --- PASS: TestSchemaDecodeEncode_JSONSchema/Nested_Object_Schema (0.01s) === RUN TestSchemaDecodeEncode_Protobuf --- PASS: TestSchemaDecodeEncode_Protobuf (0.08s) === RUN TestSchemaDecodeEncode_ErrorHandling === RUN TestSchemaDecodeEncode_ErrorHandling/Invalid_Confluent_Envelope === RUN TestSchemaDecodeEncode_ErrorHandling/Schema_Not_Found === RUN TestSchemaDecodeEncode_ErrorHandling/Invalid_Avro_Data === RUN TestSchemaDecodeEncode_ErrorHandling/Invalid_JSON_Data --- PASS: TestSchemaDecodeEncode_ErrorHandling (0.05s) --- PASS: TestSchemaDecodeEncode_ErrorHandling/Invalid_Confluent_Envelope (0.00s) --- PASS: TestSchemaDecodeEncode_ErrorHandling/Schema_Not_Found (0.02s) --- PASS: TestSchemaDecodeEncode_ErrorHandling/Invalid_Avro_Data (0.01s) --- PASS: TestSchemaDecodeEncode_ErrorHandling/Invalid_JSON_Data (0.01s) === RUN TestSchemaDecodeEncode_CachePerformance --- PASS: TestSchemaDecodeEncode_CachePerformance (0.01s) === RUN TestParseConfluentEnvelope === RUN TestParseConfluentEnvelope/valid_Avro_message === RUN TestParseConfluentEnvelope/valid_message_with_larger_schema_ID === RUN TestParseConfluentEnvelope/too_short_message === RUN TestParseConfluentEnvelope/no_magic_byte === RUN TestParseConfluentEnvelope/empty_message === RUN TestParseConfluentEnvelope/minimal_valid_message --- PASS: TestParseConfluentEnvelope (0.00s) --- PASS: TestParseConfluentEnvelope/valid_Avro_message (0.00s) --- PASS: TestParseConfluentEnvelope/valid_message_with_larger_schema_ID (0.00s) --- PASS: TestParseConfluentEnvelope/too_short_message (0.00s) --- PASS: TestParseConfluentEnvelope/no_magic_byte (0.00s) --- PASS: TestParseConfluentEnvelope/empty_message (0.00s) --- PASS: TestParseConfluentEnvelope/minimal_valid_message (0.00s) === RUN TestIsSchematized === RUN TestIsSchematized/schematized_message === RUN TestIsSchematized/non-schematized_message === RUN TestIsSchematized/empty_message --- PASS: TestIsSchematized (0.00s) --- PASS: TestIsSchematized/schematized_message (0.00s) --- PASS: TestIsSchematized/non-schematized_message (0.00s) --- PASS: TestIsSchematized/empty_message (0.00s) === RUN TestExtractSchemaID === RUN TestExtractSchemaID/valid_schema_ID === RUN TestExtractSchemaID/large_schema_ID === RUN TestExtractSchemaID/no_magic_byte === RUN TestExtractSchemaID/too_short --- PASS: TestExtractSchemaID (0.00s) --- PASS: TestExtractSchemaID/valid_schema_ID (0.00s) --- PASS: TestExtractSchemaID/large_schema_ID (0.00s) --- PASS: TestExtractSchemaID/no_magic_byte (0.00s) --- PASS: TestExtractSchemaID/too_short (0.00s) === RUN TestCreateConfluentEnvelope === RUN TestCreateConfluentEnvelope/simple_Avro_message === RUN TestCreateConfluentEnvelope/large_schema_ID === RUN TestCreateConfluentEnvelope/empty_payload --- PASS: TestCreateConfluentEnvelope (0.00s) --- PASS: TestCreateConfluentEnvelope/simple_Avro_message (0.00s) --- PASS: TestCreateConfluentEnvelope/large_schema_ID (0.00s) --- PASS: TestCreateConfluentEnvelope/empty_payload (0.00s) === RUN TestEnvelopeValidate === RUN TestEnvelopeValidate/valid_Avro_envelope === RUN TestEnvelopeValidate/zero_schema_ID === RUN TestEnvelopeValidate/empty_payload === RUN TestEnvelopeValidate/unknown_format --- PASS: TestEnvelopeValidate (0.00s) --- PASS: TestEnvelopeValidate/valid_Avro_envelope (0.00s) --- PASS: TestEnvelopeValidate/zero_schema_ID (0.00s) --- PASS: TestEnvelopeValidate/empty_payload (0.00s) --- PASS: TestEnvelopeValidate/unknown_format (0.00s) === RUN TestEnvelopeMetadata --- PASS: TestEnvelopeMetadata (0.00s) === RUN TestEncodeDecodeVarint === RUN TestEncodeDecodeVarint/zero === RUN TestEncodeDecodeVarint/small === RUN TestEncodeDecodeVarint/medium === RUN TestEncodeDecodeVarint/large === RUN TestEncodeDecodeVarint/very_large === RUN TestEncodeDecodeVarint/max_uint32 --- PASS: TestEncodeDecodeVarint (0.00s) --- PASS: TestEncodeDecodeVarint/zero (0.00s) --- PASS: TestEncodeDecodeVarint/small (0.00s) --- PASS: TestEncodeDecodeVarint/medium (0.00s) --- PASS: TestEncodeDecodeVarint/large (0.00s) --- PASS: TestEncodeDecodeVarint/very_large (0.00s) --- PASS: TestEncodeDecodeVarint/max_uint32 (0.00s) === RUN TestCreateConfluentEnvelopeWithProtobufIndexes === RUN TestCreateConfluentEnvelopeWithProtobufIndexes/avro_no_indexes === RUN TestCreateConfluentEnvelopeWithProtobufIndexes/protobuf_no_indexes === RUN TestCreateConfluentEnvelopeWithProtobufIndexes/protobuf_single_index === RUN TestCreateConfluentEnvelopeWithProtobufIndexes/protobuf_multiple_indexes --- PASS: TestCreateConfluentEnvelopeWithProtobufIndexes (0.00s) --- PASS: TestCreateConfluentEnvelopeWithProtobufIndexes/avro_no_indexes (0.00s) --- PASS: TestCreateConfluentEnvelopeWithProtobufIndexes/protobuf_no_indexes (0.00s) --- PASS: TestCreateConfluentEnvelopeWithProtobufIndexes/protobuf_single_index (0.00s) --- PASS: TestCreateConfluentEnvelopeWithProtobufIndexes/protobuf_multiple_indexes (0.00s) === RUN TestProtobufEnvelopeRoundTrip --- PASS: TestProtobufEnvelopeRoundTrip (0.00s) === RUN TestVarintEdgeCases === RUN TestVarintEdgeCases/empty_data === RUN TestVarintEdgeCases/incomplete_varint === RUN TestVarintEdgeCases/max_varint_length --- PASS: TestVarintEdgeCases (0.00s) --- PASS: TestVarintEdgeCases/empty_data (0.00s) --- PASS: TestVarintEdgeCases/incomplete_varint (0.00s) --- PASS: TestVarintEdgeCases/max_varint_length (0.00s) === RUN TestProtobufEnvelopeValidation === RUN TestProtobufEnvelopeValidation/valid_envelope === RUN TestProtobufEnvelopeValidation/zero_schema_id === RUN TestProtobufEnvelopeValidation/empty_payload --- PASS: TestProtobufEnvelopeValidation (0.00s) --- PASS: TestProtobufEnvelopeValidation/valid_envelope (0.00s) --- PASS: TestProtobufEnvelopeValidation/zero_schema_id (0.00s) --- PASS: TestProtobufEnvelopeValidation/empty_payload (0.00s) === RUN TestSchemaEvolutionChecker_AvroBackwardCompatibility === RUN TestSchemaEvolutionChecker_AvroBackwardCompatibility/Compatible_-_Add_optional_field === RUN TestSchemaEvolutionChecker_AvroBackwardCompatibility/Incompatible_-_Remove_field === RUN TestSchemaEvolutionChecker_AvroBackwardCompatibility/Incompatible_-_Add_required_field === RUN TestSchemaEvolutionChecker_AvroBackwardCompatibility/Compatible_-_Type_promotion --- PASS: TestSchemaEvolutionChecker_AvroBackwardCompatibility (0.00s) --- PASS: TestSchemaEvolutionChecker_AvroBackwardCompatibility/Compatible_-_Add_optional_field (0.00s) --- PASS: TestSchemaEvolutionChecker_AvroBackwardCompatibility/Incompatible_-_Remove_field (0.00s) --- PASS: TestSchemaEvolutionChecker_AvroBackwardCompatibility/Incompatible_-_Add_required_field (0.00s) --- PASS: TestSchemaEvolutionChecker_AvroBackwardCompatibility/Compatible_-_Type_promotion (0.00s) === RUN TestSchemaEvolutionChecker_AvroForwardCompatibility === RUN TestSchemaEvolutionChecker_AvroForwardCompatibility/Compatible_-_Remove_optional_field === RUN TestSchemaEvolutionChecker_AvroForwardCompatibility/Incompatible_-_Add_field_without_default_in_old_schema --- PASS: TestSchemaEvolutionChecker_AvroForwardCompatibility (0.00s) --- PASS: TestSchemaEvolutionChecker_AvroForwardCompatibility/Compatible_-_Remove_optional_field (0.00s) --- PASS: TestSchemaEvolutionChecker_AvroForwardCompatibility/Incompatible_-_Add_field_without_default_in_old_schema (0.00s) === RUN TestSchemaEvolutionChecker_AvroFullCompatibility === RUN TestSchemaEvolutionChecker_AvroFullCompatibility/Compatible_-_Add_optional_field_with_default === RUN TestSchemaEvolutionChecker_AvroFullCompatibility/Incompatible_-_Remove_field --- PASS: TestSchemaEvolutionChecker_AvroFullCompatibility (0.00s) --- PASS: TestSchemaEvolutionChecker_AvroFullCompatibility/Compatible_-_Add_optional_field_with_default (0.00s) --- PASS: TestSchemaEvolutionChecker_AvroFullCompatibility/Incompatible_-_Remove_field (0.00s) === RUN TestSchemaEvolutionChecker_JSONSchemaCompatibility === RUN TestSchemaEvolutionChecker_JSONSchemaCompatibility/Compatible_-_Add_optional_property === RUN TestSchemaEvolutionChecker_JSONSchemaCompatibility/Incompatible_-_Add_required_property === RUN TestSchemaEvolutionChecker_JSONSchemaCompatibility/Incompatible_-_Remove_property --- PASS: TestSchemaEvolutionChecker_JSONSchemaCompatibility (0.00s) --- PASS: TestSchemaEvolutionChecker_JSONSchemaCompatibility/Compatible_-_Add_optional_property (0.00s) --- PASS: TestSchemaEvolutionChecker_JSONSchemaCompatibility/Incompatible_-_Add_required_property (0.00s) --- PASS: TestSchemaEvolutionChecker_JSONSchemaCompatibility/Incompatible_-_Remove_property (0.00s) === RUN TestSchemaEvolutionChecker_ProtobufCompatibility === RUN TestSchemaEvolutionChecker_ProtobufCompatibility/Simplified_Protobuf_check --- PASS: TestSchemaEvolutionChecker_ProtobufCompatibility (0.00s) --- PASS: TestSchemaEvolutionChecker_ProtobufCompatibility/Simplified_Protobuf_check (0.00s) === RUN TestSchemaEvolutionChecker_NoCompatibility --- PASS: TestSchemaEvolutionChecker_NoCompatibility (0.00s) === RUN TestSchemaEvolutionChecker_TypePromotion === RUN TestSchemaEvolutionChecker_TypePromotion/int_to_long === RUN TestSchemaEvolutionChecker_TypePromotion/int_to_float === RUN TestSchemaEvolutionChecker_TypePromotion/int_to_double === RUN TestSchemaEvolutionChecker_TypePromotion/long_to_float === RUN TestSchemaEvolutionChecker_TypePromotion/long_to_double === RUN TestSchemaEvolutionChecker_TypePromotion/float_to_double === RUN TestSchemaEvolutionChecker_TypePromotion/string_to_bytes === RUN TestSchemaEvolutionChecker_TypePromotion/bytes_to_string === RUN TestSchemaEvolutionChecker_TypePromotion/long_to_int === RUN TestSchemaEvolutionChecker_TypePromotion/double_to_float === RUN TestSchemaEvolutionChecker_TypePromotion/string_to_int --- PASS: TestSchemaEvolutionChecker_TypePromotion (0.01s) --- PASS: TestSchemaEvolutionChecker_TypePromotion/int_to_long (0.00s) --- PASS: TestSchemaEvolutionChecker_TypePromotion/int_to_float (0.00s) --- PASS: TestSchemaEvolutionChecker_TypePromotion/int_to_double (0.00s) --- PASS: TestSchemaEvolutionChecker_TypePromotion/long_to_float (0.00s) --- PASS: TestSchemaEvolutionChecker_TypePromotion/long_to_double (0.00s) --- PASS: TestSchemaEvolutionChecker_TypePromotion/float_to_double (0.00s) --- PASS: TestSchemaEvolutionChecker_TypePromotion/string_to_bytes (0.00s) --- PASS: TestSchemaEvolutionChecker_TypePromotion/bytes_to_string (0.00s) --- PASS: TestSchemaEvolutionChecker_TypePromotion/long_to_int (0.00s) --- PASS: TestSchemaEvolutionChecker_TypePromotion/double_to_float (0.00s) --- PASS: TestSchemaEvolutionChecker_TypePromotion/string_to_int (0.00s) === RUN TestSchemaEvolutionChecker_SuggestEvolution === RUN TestSchemaEvolutionChecker_SuggestEvolution/Compatible_schema === RUN TestSchemaEvolutionChecker_SuggestEvolution/Incompatible_schema_with_suggestions --- PASS: TestSchemaEvolutionChecker_SuggestEvolution (0.00s) --- PASS: TestSchemaEvolutionChecker_SuggestEvolution/Compatible_schema (0.00s) --- PASS: TestSchemaEvolutionChecker_SuggestEvolution/Incompatible_schema_with_suggestions (0.00s) === RUN TestSchemaEvolutionChecker_CanEvolve --- PASS: TestSchemaEvolutionChecker_CanEvolve (0.00s) === RUN TestSchemaEvolutionChecker_ExtractFields === RUN TestSchemaEvolutionChecker_ExtractFields/Extract_Avro_fields === RUN TestSchemaEvolutionChecker_ExtractFields/Extract_JSON_Schema_required_fields === RUN TestSchemaEvolutionChecker_ExtractFields/Extract_JSON_Schema_properties --- PASS: TestSchemaEvolutionChecker_ExtractFields (0.00s) --- PASS: TestSchemaEvolutionChecker_ExtractFields/Extract_Avro_fields (0.00s) --- PASS: TestSchemaEvolutionChecker_ExtractFields/Extract_JSON_Schema_required_fields (0.00s) --- PASS: TestSchemaEvolutionChecker_ExtractFields/Extract_JSON_Schema_properties (0.00s) === RUN TestFullIntegration_AvroWorkflow === RUN TestFullIntegration_AvroWorkflow/Producer_Workflow integration_test.go:88: Successfully processed producer message with 5 fields === RUN TestFullIntegration_AvroWorkflow/Consumer_Workflow integration_test.go:140: Successfully reconstructed consumer message: 38 bytes === RUN TestFullIntegration_AvroWorkflow/Round_Trip_Integrity integration_test.go:203: Round-trip integrity test passed --- PASS: TestFullIntegration_AvroWorkflow (0.02s) --- PASS: TestFullIntegration_AvroWorkflow/Producer_Workflow (0.01s) --- PASS: TestFullIntegration_AvroWorkflow/Consumer_Workflow (0.00s) --- PASS: TestFullIntegration_AvroWorkflow/Round_Trip_Integrity (0.00s) === RUN TestFullIntegration_MultiFormatSupport === RUN TestFullIntegration_MultiFormatSupport/Avro_Format integration_test.go:281: Successfully processed Avro_Format format === RUN TestFullIntegration_MultiFormatSupport/JSON_Schema_Format integration_test.go:281: Successfully processed JSON_Schema_Format format --- PASS: TestFullIntegration_MultiFormatSupport (0.02s) --- PASS: TestFullIntegration_MultiFormatSupport/Avro_Format (0.01s) --- PASS: TestFullIntegration_MultiFormatSupport/JSON_Schema_Format (0.01s) === RUN TestIntegration_CachePerformance integration_test.go:343: Cache performance: First decode: 8.439517ms, Average cached: 1.267518ms integration_test.go:345: Cache stats: 1 decoders, 1 schemas, 0 subjects --- PASS: TestIntegration_CachePerformance (0.14s) === RUN TestIntegration_ErrorHandling === RUN TestIntegration_ErrorHandling/Non_Schematized_Message integration_test.go:405: Expected error occurred: message is not schematized === RUN TestIntegration_ErrorHandling/Invalid_Schema_ID integration_test.go:405: Expected error occurred: failed to get schema 999: schema registry error 404: === RUN TestIntegration_ErrorHandling/Empty_Payload integration_test.go:405: Expected error occurred: invalid envelope: empty payload === RUN TestIntegration_ErrorHandling/Corrupted_Avro_Data integration_test.go:405: Expected error occurred: failed to decode Avro message: strict validation failed: failed to decode Avro data: cannot decode binary record "User" field "name": cannot decode binary string: cannot decode binary bytes: short buffer --- PASS: TestIntegration_ErrorHandling (0.02s) --- PASS: TestIntegration_ErrorHandling/Non_Schematized_Message (0.00s) --- PASS: TestIntegration_ErrorHandling/Invalid_Schema_ID (0.01s) --- PASS: TestIntegration_ErrorHandling/Empty_Payload (0.00s) --- PASS: TestIntegration_ErrorHandling/Corrupted_Avro_Data (0.01s) === RUN TestIntegration_SchemaEvolution === RUN TestIntegration_SchemaEvolution/Schema_V1_Message === RUN TestIntegration_SchemaEvolution/Schema_V2_Message --- PASS: TestIntegration_SchemaEvolution (0.01s) --- PASS: TestIntegration_SchemaEvolution/Schema_V1_Message (0.01s) --- PASS: TestIntegration_SchemaEvolution/Schema_V2_Message (0.00s) === RUN TestNewJSONSchemaDecoder === RUN TestNewJSONSchemaDecoder/valid_object_schema === RUN TestNewJSONSchemaDecoder/valid_array_schema === RUN TestNewJSONSchemaDecoder/valid_string_schema_with_format === RUN TestNewJSONSchemaDecoder/invalid_JSON === RUN TestNewJSONSchemaDecoder/empty_schema --- PASS: TestNewJSONSchemaDecoder (0.00s) --- PASS: TestNewJSONSchemaDecoder/valid_object_schema (0.00s) --- PASS: TestNewJSONSchemaDecoder/valid_array_schema (0.00s) --- PASS: TestNewJSONSchemaDecoder/valid_string_schema_with_format (0.00s) --- PASS: TestNewJSONSchemaDecoder/invalid_JSON (0.00s) --- PASS: TestNewJSONSchemaDecoder/empty_schema (0.00s) === RUN TestJSONSchemaDecoder_Decode === RUN TestJSONSchemaDecoder_Decode/valid_complete_data === RUN TestJSONSchemaDecoder_Decode/valid_minimal_data === RUN TestJSONSchemaDecoder_Decode/missing_required_field === RUN TestJSONSchemaDecoder_Decode/invalid_type === RUN TestJSONSchemaDecoder_Decode/invalid_email_format === RUN TestJSONSchemaDecoder_Decode/negative_age --- PASS: TestJSONSchemaDecoder_Decode (0.01s) --- PASS: TestJSONSchemaDecoder_Decode/valid_complete_data (0.00s) --- PASS: TestJSONSchemaDecoder_Decode/valid_minimal_data (0.00s) --- PASS: TestJSONSchemaDecoder_Decode/missing_required_field (0.00s) --- PASS: TestJSONSchemaDecoder_Decode/invalid_type (0.00s) --- PASS: TestJSONSchemaDecoder_Decode/invalid_email_format (0.00s) --- PASS: TestJSONSchemaDecoder_Decode/negative_age (0.00s) === RUN TestJSONSchemaDecoder_DecodeToRecordValue --- PASS: TestJSONSchemaDecoder_DecodeToRecordValue (0.00s) === RUN TestJSONSchemaDecoder_InferRecordType --- PASS: TestJSONSchemaDecoder_InferRecordType (0.00s) === RUN TestJSONSchemaDecoder_EncodeFromRecordValue --- PASS: TestJSONSchemaDecoder_EncodeFromRecordValue (0.00s) === RUN TestJSONSchemaDecoder_ArrayAndPrimitiveSchemas === RUN TestJSONSchemaDecoder_ArrayAndPrimitiveSchemas/array_schema === RUN TestJSONSchemaDecoder_ArrayAndPrimitiveSchemas/string_schema === RUN TestJSONSchemaDecoder_ArrayAndPrimitiveSchemas/number_schema === RUN TestJSONSchemaDecoder_ArrayAndPrimitiveSchemas/boolean_schema --- PASS: TestJSONSchemaDecoder_ArrayAndPrimitiveSchemas (0.00s) --- PASS: TestJSONSchemaDecoder_ArrayAndPrimitiveSchemas/array_schema (0.00s) --- PASS: TestJSONSchemaDecoder_ArrayAndPrimitiveSchemas/string_schema (0.00s) --- PASS: TestJSONSchemaDecoder_ArrayAndPrimitiveSchemas/number_schema (0.00s) --- PASS: TestJSONSchemaDecoder_ArrayAndPrimitiveSchemas/boolean_schema (0.00s) === RUN TestJSONSchemaDecoder_GetSchemaInfo --- PASS: TestJSONSchemaDecoder_GetSchemaInfo (0.00s) === RUN TestAvroLoadTestDecoding loadtest_decode_test.go:127: Avro encoded size: 68 bytes loadtest_decode_test.go:133: Confluent wire format size: 73 bytes loadtest_decode_test.go:170: ✅ Avro decoding successful: 7 fields --- PASS: TestAvroLoadTestDecoding (0.00s) === RUN TestJSONSchemaLoadTestDecoding loadtest_decode_test.go:183: JSON encoded size: 174 bytes loadtest_decode_test.go:184: JSON content: {"id":"msg-test-123","timestamp":1787609907465933251,"producer_id":0,"counter":42,"user_id":"user-789","event_type":"click","properties":{"browser":"chrome","version":"1.0"}} loadtest_decode_test.go:190: Confluent wire format size: 179 bytes loadtest_decode_test.go:227: ✅ JSON Schema decoding successful: 7 fields --- PASS: TestJSONSchemaLoadTestDecoding (0.00s) === RUN TestProtobufLoadTestDecoding loadtest_decode_test.go:241: JSON (for Protobuf) encoded size: 174 bytes loadtest_decode_test.go:242: JSON content: {"id":"msg-test-123","timestamp":1787609907469722109,"producer_id":0,"counter":42,"user_id":"user-789","event_type":"click","properties":{"browser":"chrome","version":"1.0"}} loadtest_decode_test.go:248: Confluent wire format size: 179 bytes loadtest_decode_test.go:275: Unexpectedly succeeded in decoding JSON as Protobuf loadtest_decode_test.go:277: RecordValue has 7 fields --- PASS: TestProtobufLoadTestDecoding (0.03s) === RUN TestManager_SchemaEvolution === RUN TestManager_SchemaEvolution/Compatible_Avro_evolution === RUN TestManager_SchemaEvolution/Incompatible_Avro_evolution === RUN TestManager_SchemaEvolution/Schema_evolution_suggestions === RUN TestManager_SchemaEvolution/JSON_Schema_evolution === RUN TestManager_SchemaEvolution/Full_compatibility_check === RUN TestManager_SchemaEvolution/Type_promotion_compatibility --- PASS: TestManager_SchemaEvolution (0.01s) --- PASS: TestManager_SchemaEvolution/Compatible_Avro_evolution (0.00s) --- PASS: TestManager_SchemaEvolution/Incompatible_Avro_evolution (0.00s) --- PASS: TestManager_SchemaEvolution/Schema_evolution_suggestions (0.00s) --- PASS: TestManager_SchemaEvolution/JSON_Schema_evolution (0.00s) --- PASS: TestManager_SchemaEvolution/Full_compatibility_check (0.00s) --- PASS: TestManager_SchemaEvolution/Type_promotion_compatibility (0.00s) === RUN TestManager_CompatibilityLevels === RUN TestManager_CompatibilityLevels/Get_default_compatibility_level === RUN TestManager_CompatibilityLevels/Set_compatibility_level --- PASS: TestManager_CompatibilityLevels (0.00s) --- PASS: TestManager_CompatibilityLevels/Get_default_compatibility_level (0.00s) --- PASS: TestManager_CompatibilityLevels/Set_compatibility_level (0.00s) === RUN TestManager_CanEvolveSchema === RUN TestManager_CanEvolveSchema/Compatible_evolution === RUN TestManager_CanEvolveSchema/Incompatible_evolution --- PASS: TestManager_CanEvolveSchema (0.00s) --- PASS: TestManager_CanEvolveSchema/Compatible_evolution (0.00s) --- PASS: TestManager_CanEvolveSchema/Incompatible_evolution (0.00s) === RUN TestManager_SchemaEvolutionWorkflow === RUN TestManager_SchemaEvolutionWorkflow/Complete_evolution_workflow --- PASS: TestManager_SchemaEvolutionWorkflow (0.00s) --- PASS: TestManager_SchemaEvolutionWorkflow/Complete_evolution_workflow (0.00s) === RUN TestManager_DecodeMessage --- PASS: TestManager_DecodeMessage (0.01s) === RUN TestManager_IsSchematized === RUN TestManager_IsSchematized/schematized_message === RUN TestManager_IsSchematized/non-schematized_message === RUN TestManager_IsSchematized/empty_message --- PASS: TestManager_IsSchematized (0.00s) --- PASS: TestManager_IsSchematized/schematized_message (0.00s) --- PASS: TestManager_IsSchematized/non-schematized_message (0.00s) --- PASS: TestManager_IsSchematized/empty_message (0.00s) === RUN TestManager_GetSchemaInfo --- PASS: TestManager_GetSchemaInfo (0.01s) === RUN TestManager_CacheManagement --- PASS: TestManager_CacheManagement (0.00s) === RUN TestManager_EncodeMessage --- PASS: TestManager_EncodeMessage (0.02s) === RUN TestProtobufDecoder_BasicDecoding === RUN TestProtobufDecoder_BasicDecoding/NewProtobufDecoder_with_binary_descriptor protobuf_decoder_test.go:40: Protobuf decoder creation succeeded - Phase E3 is working! === RUN TestProtobufDecoder_BasicDecoding/NewProtobufDecoder_with_empty_message_name protobuf_decoder_test.go:60: Empty message name resolution succeeded - Phase E3 is working! --- PASS: TestProtobufDecoder_BasicDecoding (0.00s) --- PASS: TestProtobufDecoder_BasicDecoding/NewProtobufDecoder_with_binary_descriptor (0.00s) --- PASS: TestProtobufDecoder_BasicDecoding/NewProtobufDecoder_with_empty_message_name (0.00s) === RUN TestProtobufDecoder_Integration === RUN TestProtobufDecoder_Integration/Parse_complex_descriptor protobuf_decoder_test.go:86: Empty message name resolution succeeded! protobuf_decoder_test.go:100: Complex message resolution succeeded! --- PASS: TestProtobufDecoder_Integration (0.00s) --- PASS: TestProtobufDecoder_Integration/Parse_complex_descriptor (0.00s) === RUN TestProtobufDecoder_Caching === RUN TestProtobufDecoder_Caching/Decoder_creation_uses_cache --- PASS: TestProtobufDecoder_Caching (0.00s) --- PASS: TestProtobufDecoder_Caching/Decoder_creation_uses_cache (0.00s) === RUN TestProtobufDecoder_ErrorHandling === RUN TestProtobufDecoder_ErrorHandling/Invalid_binary_data === RUN TestProtobufDecoder_ErrorHandling/Empty_binary_data === RUN TestProtobufDecoder_ErrorHandling/FileDescriptorSet_with_no_messages --- PASS: TestProtobufDecoder_ErrorHandling (0.00s) --- PASS: TestProtobufDecoder_ErrorHandling/Invalid_binary_data (0.00s) --- PASS: TestProtobufDecoder_ErrorHandling/Empty_binary_data (0.00s) --- PASS: TestProtobufDecoder_ErrorHandling/FileDescriptorSet_with_no_messages (0.00s) === RUN TestProtobufDescriptorParser_BasicParsing === RUN TestProtobufDescriptorParser_BasicParsing/Parse_Simple_Message_Descriptor protobuf_descriptor_test.go:42: Simple message descriptor resolution succeeded - Phase E3 is working! === RUN TestProtobufDescriptorParser_BasicParsing/Parse_Complex_Message_Descriptor === RUN TestProtobufDescriptorParser_BasicParsing/Cache_Functionality protobuf_descriptor_test.go:101: Cache functionality working with successful descriptor resolution! --- PASS: TestProtobufDescriptorParser_BasicParsing (0.00s) --- PASS: TestProtobufDescriptorParser_BasicParsing/Parse_Simple_Message_Descriptor (0.00s) --- PASS: TestProtobufDescriptorParser_BasicParsing/Parse_Complex_Message_Descriptor (0.00s) --- PASS: TestProtobufDescriptorParser_BasicParsing/Cache_Functionality (0.00s) === RUN TestProtobufDescriptorParser_Validation === RUN TestProtobufDescriptorParser_Validation/Invalid_Binary_Data === RUN TestProtobufDescriptorParser_Validation/Empty_FileDescriptorSet === RUN TestProtobufDescriptorParser_Validation/FileDescriptor_Without_Name === RUN TestProtobufDescriptorParser_Validation/FileDescriptor_Without_Package --- PASS: TestProtobufDescriptorParser_Validation (0.00s) --- PASS: TestProtobufDescriptorParser_Validation/Invalid_Binary_Data (0.00s) --- PASS: TestProtobufDescriptorParser_Validation/Empty_FileDescriptorSet (0.00s) --- PASS: TestProtobufDescriptorParser_Validation/FileDescriptor_Without_Name (0.00s) --- PASS: TestProtobufDescriptorParser_Validation/FileDescriptor_Without_Package (0.00s) === RUN TestProtobufDescriptorParser_MessageSearch === RUN TestProtobufDescriptorParser_MessageSearch/Message_Not_Found === RUN TestProtobufDescriptorParser_MessageSearch/Nested_Message_Search protobuf_descriptor_test.go:235: Nested message resolution succeeded - Phase E3 is working! --- PASS: TestProtobufDescriptorParser_MessageSearch (0.00s) --- PASS: TestProtobufDescriptorParser_MessageSearch/Message_Not_Found (0.00s) --- PASS: TestProtobufDescriptorParser_MessageSearch/Nested_Message_Search (0.00s) === RUN TestProtobufDescriptorParser_Dependencies === RUN TestProtobufDescriptorParser_Dependencies/Extract_Dependencies --- PASS: TestProtobufDescriptorParser_Dependencies (0.00s) --- PASS: TestProtobufDescriptorParser_Dependencies/Extract_Dependencies (0.00s) === RUN TestProtobufSchema_Methods === RUN TestProtobufSchema_Methods/GetMessageFields_Implemented === RUN TestProtobufSchema_Methods/GetFieldByName_Implemented === RUN TestProtobufSchema_Methods/GetFieldByNumber_Implemented === RUN TestProtobufSchema_Methods/ValidateMessage_Requires_MessageDescriptor --- PASS: TestProtobufSchema_Methods (0.00s) --- PASS: TestProtobufSchema_Methods/GetMessageFields_Implemented (0.00s) --- PASS: TestProtobufSchema_Methods/GetFieldByName_Implemented (0.00s) --- PASS: TestProtobufSchema_Methods/GetFieldByNumber_Implemented (0.00s) --- PASS: TestProtobufSchema_Methods/ValidateMessage_Requires_MessageDescriptor (0.00s) === RUN TestProtobufDescriptorParser_CacheManagement --- PASS: TestProtobufDescriptorParser_CacheManagement (0.00s) === RUN TestSchemaReconstruction_Avro reconstruction_test.go:77: Original Avro binary length: 11 reconstruction_test.go:78: Original Confluent message length: 16 reconstruction_test.go:85: Parsed envelope - SchemaID: 1, Format: AVRO, Payload length: 11 reconstruction_test.go:124: Successfully completed round-trip: Original -> Decode -> Encode -> Decode reconstruction_test.go:125: Original message size: 16 bytes reconstruction_test.go:126: Reconstructed message size: 16 bytes --- PASS: TestSchemaReconstruction_Avro (0.01s) === RUN TestSchemaReconstruction_MultipleFormats === RUN TestSchemaReconstruction_MultipleFormats/Avro === RUN TestSchemaReconstruction_MultipleFormats/Protobuf === RUN TestSchemaReconstruction_MultipleFormats/JSON_Schema --- PASS: TestSchemaReconstruction_MultipleFormats (0.14s) --- PASS: TestSchemaReconstruction_MultipleFormats/Avro (0.09s) --- PASS: TestSchemaReconstruction_MultipleFormats/Protobuf (0.01s) --- PASS: TestSchemaReconstruction_MultipleFormats/JSON_Schema (0.01s) === RUN TestConfluentEnvelope_RoundTrip === RUN TestConfluentEnvelope_RoundTrip/Avro_message reconstruction_test.go:259: Successfully round-tripped Avro message envelope: 17 bytes === RUN TestConfluentEnvelope_RoundTrip/Protobuf_message_with_indexes reconstruction_test.go:259: Successfully round-tripped Protobuf message with indexes envelope: 21 bytes === RUN TestConfluentEnvelope_RoundTrip/JSON_Schema_message reconstruction_test.go:259: Successfully round-tripped JSON Schema message envelope: 17 bytes --- PASS: TestConfluentEnvelope_RoundTrip (0.00s) --- PASS: TestConfluentEnvelope_RoundTrip/Avro_message (0.00s) --- PASS: TestConfluentEnvelope_RoundTrip/Protobuf_message_with_indexes (0.00s) --- PASS: TestConfluentEnvelope_RoundTrip/JSON_Schema_message (0.00s) === RUN TestSchemaMetadata_Preservation reconstruction_test.go:306: Successfully preserved and reconstructed schema metadata --- PASS: TestSchemaMetadata_Preservation (0.00s) === RUN TestNewRegistryClient --- PASS: TestNewRegistryClient (0.00s) === RUN TestRegistryClient_GetSchemaByID === RUN TestRegistryClient_GetSchemaByID/successful_fetch === RUN TestRegistryClient_GetSchemaByID/schema_not_found === RUN TestRegistryClient_GetSchemaByID/cache_hit --- PASS: TestRegistryClient_GetSchemaByID (0.01s) --- PASS: TestRegistryClient_GetSchemaByID/successful_fetch (0.01s) --- PASS: TestRegistryClient_GetSchemaByID/schema_not_found (0.00s) --- PASS: TestRegistryClient_GetSchemaByID/cache_hit (0.00s) === RUN TestRegistryClient_GetLatestSchema --- PASS: TestRegistryClient_GetLatestSchema (0.02s) === RUN TestRegistryClient_RegisterSchema --- PASS: TestRegistryClient_RegisterSchema (0.03s) === RUN TestRegistryClient_CheckCompatibility --- PASS: TestRegistryClient_CheckCompatibility (0.01s) === RUN TestRegistryClient_ListSubjects --- PASS: TestRegistryClient_ListSubjects (0.01s) === RUN TestRegistryClient_DetectSchemaFormat === RUN TestRegistryClient_DetectSchemaFormat/Avro_record_schema === RUN TestRegistryClient_DetectSchemaFormat/Avro_enum_schema === RUN TestRegistryClient_DetectSchemaFormat/JSON_Schema === RUN TestRegistryClient_DetectSchemaFormat/Protobuf_(non-JSON) === RUN TestRegistryClient_DetectSchemaFormat/Simple_Avro_primitive --- PASS: TestRegistryClient_DetectSchemaFormat (0.00s) --- PASS: TestRegistryClient_DetectSchemaFormat/Avro_record_schema (0.00s) --- PASS: TestRegistryClient_DetectSchemaFormat/Avro_enum_schema (0.00s) --- PASS: TestRegistryClient_DetectSchemaFormat/JSON_Schema (0.00s) --- PASS: TestRegistryClient_DetectSchemaFormat/Protobuf_(non-JSON) (0.00s) --- PASS: TestRegistryClient_DetectSchemaFormat/Simple_Avro_primitive (0.00s) === RUN TestRegistryClient_CacheManagement --- PASS: TestRegistryClient_CacheManagement (0.00s) === RUN TestRegistryClient_HealthCheck === RUN TestRegistryClient_HealthCheck/healthy_registry === RUN TestRegistryClient_HealthCheck/unhealthy_registry --- PASS: TestRegistryClient_HealthCheck (0.02s) --- PASS: TestRegistryClient_HealthCheck/healthy_registry (0.01s) --- PASS: TestRegistryClient_HealthCheck/unhealthy_registry (0.01s) PASS ok github.com/seaweedfs/seaweedfs/weed/mq/kafka/schema 19.001s === RUN TestWriteRowsNoPanic --- PASS: TestWriteRowsNoPanic (0.36s) PASS ok github.com/seaweedfs/seaweedfs/weed/mq/logstore 14.710s === RUN TestConsumerGroupPosition_JSON === RUN TestConsumerGroupPosition_JSON/offset-based_position consumer_group_storage_test.go:53: JSON: {"type":"offset","value":12345,"offset_type":"EXACT_OFFSET","committed_at":1787609903435,"metadata":"test metadata"} === RUN TestConsumerGroupPosition_JSON/timestamp-based_position consumer_group_storage_test.go:53: JSON: {"type":"timestamp","value":1787609903435676621,"offset_type":"EXACT_TS_NS","committed_at":1787609903435,"metadata":"checkpoint at 2024-10-05"} === RUN TestConsumerGroupPosition_JSON/minimal_position consumer_group_storage_test.go:53: JSON: {"type":"offset","value":42,"offset_type":"","committed_at":0,"metadata":""} --- PASS: TestConsumerGroupPosition_JSON (0.01s) --- PASS: TestConsumerGroupPosition_JSON/offset-based_position (0.00s) --- PASS: TestConsumerGroupPosition_JSON/timestamp-based_position (0.00s) --- PASS: TestConsumerGroupPosition_JSON/minimal_position (0.00s) === RUN TestConsumerGroupPosition_JSONExamples consumer_group_storage_test.go:93: Example 0: Type=offset, Value=12345 consumer_group_storage_test.go:93: Example 1: Type=timestamp, Value=1696521600000000000 consumer_group_storage_test.go:93: Example 2: Type=offset, Value=42 --- PASS: TestConsumerGroupPosition_JSONExamples (0.00s) === RUN TestConsumerGroupPosition_TypeValidation --- PASS: TestConsumerGroupPosition_TypeValidation (0.03s) PASS ok github.com/seaweedfs/seaweedfs/weed/mq/offset 14.551s === RUN Test_allocateOneBroker === RUN Test_allocateOneBroker/test_only_one_broker I0824 22:18:25.705449 allocate.go:54 allocate topic partitions 1: [partition:{ring_size:2520 range_stop:2520 unix_time_ns:1787609905665875501} leader_broker:"localhost:17777"] --- PASS: Test_allocateOneBroker (0.50s) --- PASS: Test_allocateOneBroker/test_only_one_broker (0.42s) === RUN TestEnsureAssignmentsToActiveBrokersX === RUN TestEnsureAssignmentsToActiveBrokersX/test_empty_leader test empty leader before [partition:{} follower_broker:"localhost:2"] test empty leader after [partition:{} leader_broker:"localhost:3" follower_broker:"localhost:2"] === RUN TestEnsureAssignmentsToActiveBrokersX/test_empty_follower test empty follower before [partition:{} leader_broker:"localhost:1"] test empty follower after [partition:{} leader_broker:"localhost:1" follower_broker:"localhost:3"] === RUN TestEnsureAssignmentsToActiveBrokersX/test_dead_follower test dead follower before [partition:{} leader_broker:"localhost:1" follower_broker:"localhost:200"] test dead follower after [partition:{} leader_broker:"localhost:1" follower_broker:"localhost:6"] === RUN TestEnsureAssignmentsToActiveBrokersX/test_dead_leader_and_follower test dead leader and follower before [partition:{} leader_broker:"localhost:100" follower_broker:"localhost:200"] test dead leader and follower after [partition:{} leader_broker:"localhost:1" follower_broker:"localhost:6"] === RUN TestEnsureAssignmentsToActiveBrokersX/test_low_active_brokers test low active brokers before [partition:{} leader_broker:"localhost:1" follower_broker:"localhost:2"] test low active brokers after [partition:{} leader_broker:"localhost:1" follower_broker:"localhost:2"] === RUN TestEnsureAssignmentsToActiveBrokersX/test_low_active_brokers_with_one_follower test low active brokers with one follower before [partition:{} leader_broker:"localhost:1"] test low active brokers with one follower after [partition:{} leader_broker:"localhost:1" follower_broker:"localhost:2"] === RUN TestEnsureAssignmentsToActiveBrokersX/test_single_active_broker test single active broker before [partition:{} leader_broker:"localhost:1" follower_broker:"localhost:2"] test single active broker after [partition:{} leader_broker:"localhost:1" follower_broker:"localhost:1"] --- PASS: TestEnsureAssignmentsToActiveBrokersX (0.39s) --- PASS: TestEnsureAssignmentsToActiveBrokersX/test_empty_leader (0.08s) --- PASS: TestEnsureAssignmentsToActiveBrokersX/test_empty_follower (0.06s) --- PASS: TestEnsureAssignmentsToActiveBrokersX/test_dead_follower (0.08s) --- PASS: TestEnsureAssignmentsToActiveBrokersX/test_dead_leader_and_follower (0.07s) --- PASS: TestEnsureAssignmentsToActiveBrokersX/test_low_active_brokers (0.00s) --- PASS: TestEnsureAssignmentsToActiveBrokersX/test_low_active_brokers_with_one_follower (0.01s) --- PASS: TestEnsureAssignmentsToActiveBrokersX/test_single_active_broker (0.00s) === RUN TestBalanceTopicPartitionOnBrokers === RUN TestBalanceTopicPartitionOnBrokers/test --- PASS: TestBalanceTopicPartitionOnBrokers (0.03s) --- PASS: TestBalanceTopicPartitionOnBrokers/test (0.01s) PASS ok github.com/seaweedfs/seaweedfs/weed/mq/pub_balancer 17.643s === RUN TestSplitFlatSchemaToKeyValue --- PASS: TestSplitFlatSchemaToKeyValue (0.04s) === RUN TestSplitFlatSchemaToKeyValueMissingColumns --- PASS: TestSplitFlatSchemaToKeyValueMissingColumns (0.08s) === RUN TestCombineFlatSchemaFromKeyValue --- PASS: TestCombineFlatSchemaFromKeyValue (0.08s) === RUN TestExtractKeyColumnsFromCombinedSchema --- PASS: TestExtractKeyColumnsFromCombinedSchema (0.00s) === RUN TestValidateKeyColumns --- PASS: TestValidateKeyColumns (0.00s) === RUN TestEnumScalarType === RUN TestEnumScalarType/Boolean === RUN TestEnumScalarType/Integer === RUN TestEnumScalarType/Long === RUN TestEnumScalarType/Float === RUN TestEnumScalarType/Double === RUN TestEnumScalarType/Bytes === RUN TestEnumScalarType/String --- PASS: TestEnumScalarType (0.00s) --- PASS: TestEnumScalarType/Boolean (0.00s) --- PASS: TestEnumScalarType/Integer (0.00s) --- PASS: TestEnumScalarType/Long (0.00s) --- PASS: TestEnumScalarType/Float (0.00s) --- PASS: TestEnumScalarType/Double (0.00s) --- PASS: TestEnumScalarType/Bytes (0.00s) --- PASS: TestEnumScalarType/String (0.00s) === RUN TestField --- PASS: TestField (0.00s) === RUN TestRecordType fields: < name: "field_key" field_index: 1 type: < scalar_type: INT32 > > fields: < name: "field_record" field_index: 2 type: < record_type: < fields: < name: "field_1" field_index: 1 type: < scalar_type: INT32 > > fields: < name: "field_2" field_index: 2 type: < scalar_type: STRING > > > > > {"fields":[{"name":"field_key","field_index":1,"type":{"Kind":{"ScalarType":1}}},{"name":"field_record","field_index":2,"type":{"Kind":{"RecordType":{"fields":[{"name":"field_1","field_index":1,"type":{"Kind":{"ScalarType":1}}},{"name":"field_2","field_index":2,"type":{"Kind":{"ScalarType":7}}}]}}}}]} --- PASS: TestRecordType (0.60s) === RUN TestStructToSchema === RUN TestStructToSchema/scalar_type === RUN TestStructToSchema/simple_struct_type === RUN TestStructToSchema/simple_list === RUN TestStructToSchema/simple_[]byte === RUN TestStructToSchema/nested_simple_structs === RUN TestStructToSchema/nested_struct_type --- PASS: TestStructToSchema (0.00s) --- PASS: TestStructToSchema/scalar_type (0.00s) --- PASS: TestStructToSchema/simple_struct_type (0.00s) --- PASS: TestStructToSchema/simple_list (0.00s) --- PASS: TestStructToSchema/simple_[]byte (0.00s) --- PASS: TestStructToSchema/nested_simple_structs (0.00s) --- PASS: TestStructToSchema/nested_struct_type (0.00s) === RUN TestToParquetLevels === RUN TestToParquetLevels/nested_type --- PASS: TestToParquetLevels (0.00s) --- PASS: TestToParquetLevels/nested_type (0.00s) === RUN TestToParquetValue_BasicTypes === RUN TestToParquetValue_BasicTypes/BoolValue_true === RUN TestToParquetValue_BasicTypes/Int32Value === RUN TestToParquetValue_BasicTypes/Int64Value === RUN TestToParquetValue_BasicTypes/FloatValue === RUN TestToParquetValue_BasicTypes/DoubleValue === RUN TestToParquetValue_BasicTypes/BytesValue === RUN TestToParquetValue_BasicTypes/BytesValue_empty === RUN TestToParquetValue_BasicTypes/StringValue --- PASS: TestToParquetValue_BasicTypes (0.00s) --- PASS: TestToParquetValue_BasicTypes/BoolValue_true (0.00s) --- PASS: TestToParquetValue_BasicTypes/Int32Value (0.00s) --- PASS: TestToParquetValue_BasicTypes/Int64Value (0.00s) --- PASS: TestToParquetValue_BasicTypes/FloatValue (0.00s) --- PASS: TestToParquetValue_BasicTypes/DoubleValue (0.00s) --- PASS: TestToParquetValue_BasicTypes/BytesValue (0.00s) --- PASS: TestToParquetValue_BasicTypes/BytesValue_empty (0.00s) --- PASS: TestToParquetValue_BasicTypes/StringValue (0.00s) === RUN TestToParquetValue_TimestampValue === RUN TestToParquetValue_TimestampValue/Valid_TimestampValue_UTC === RUN TestToParquetValue_TimestampValue/Valid_TimestampValue_local === RUN TestToParquetValue_TimestampValue/TimestampValue_zero === RUN TestToParquetValue_TimestampValue/TimestampValue_negative_(before_epoch) === RUN TestToParquetValue_TimestampValue/TimestampValue_nil_pointer --- PASS: TestToParquetValue_TimestampValue (0.00s) --- PASS: TestToParquetValue_TimestampValue/Valid_TimestampValue_UTC (0.00s) --- PASS: TestToParquetValue_TimestampValue/Valid_TimestampValue_local (0.00s) --- PASS: TestToParquetValue_TimestampValue/TimestampValue_zero (0.00s) --- PASS: TestToParquetValue_TimestampValue/TimestampValue_negative_(before_epoch) (0.00s) --- PASS: TestToParquetValue_TimestampValue/TimestampValue_nil_pointer (0.00s) === RUN TestToParquetValue_DateValue === RUN TestToParquetValue_DateValue/Valid_DateValue_(2024-01-01) === RUN TestToParquetValue_DateValue/DateValue_epoch_(1970-01-01) === RUN TestToParquetValue_DateValue/DateValue_before_epoch === RUN TestToParquetValue_DateValue/DateValue_nil_pointer --- PASS: TestToParquetValue_DateValue (0.00s) --- PASS: TestToParquetValue_DateValue/Valid_DateValue_(2024-01-01) (0.00s) --- PASS: TestToParquetValue_DateValue/DateValue_epoch_(1970-01-01) (0.00s) --- PASS: TestToParquetValue_DateValue/DateValue_before_epoch (0.00s) --- PASS: TestToParquetValue_DateValue/DateValue_nil_pointer (0.00s) === RUN TestToParquetValue_DecimalValue === RUN TestToParquetValue_DecimalValue/Small_Decimal_(precision_<=_9)_-_positive === RUN TestToParquetValue_DecimalValue/Small_Decimal_(precision_<=_9)_-_negative === RUN TestToParquetValue_DecimalValue/Medium_Decimal_(9_<_precision_<=_18) === RUN TestToParquetValue_DecimalValue/Large_Decimal_(precision_>_18) === RUN TestToParquetValue_DecimalValue/Decimal_with_zero_precision === RUN TestToParquetValue_DecimalValue/Decimal_nil_pointer === RUN TestToParquetValue_DecimalValue/Decimal_with_nil_Value_bytes === RUN TestToParquetValue_DecimalValue/Decimal_with_empty_Value_bytes === RUN TestToParquetValue_DecimalValue/Decimal_out_of_int32_range_(stored_as_binary) === RUN TestToParquetValue_DecimalValue/Decimal_out_of_int64_range_(stored_as_binary) === RUN TestToParquetValue_DecimalValue/Decimal_extremely_large_value_(should_be_rejected) --- PASS: TestToParquetValue_DecimalValue (0.01s) --- PASS: TestToParquetValue_DecimalValue/Small_Decimal_(precision_<=_9)_-_positive (0.00s) --- PASS: TestToParquetValue_DecimalValue/Small_Decimal_(precision_<=_9)_-_negative (0.00s) --- PASS: TestToParquetValue_DecimalValue/Medium_Decimal_(9_<_precision_<=_18) (0.00s) --- PASS: TestToParquetValue_DecimalValue/Large_Decimal_(precision_>_18) (0.00s) --- PASS: TestToParquetValue_DecimalValue/Decimal_with_zero_precision (0.00s) --- PASS: TestToParquetValue_DecimalValue/Decimal_nil_pointer (0.00s) --- PASS: TestToParquetValue_DecimalValue/Decimal_with_nil_Value_bytes (0.00s) --- PASS: TestToParquetValue_DecimalValue/Decimal_with_empty_Value_bytes (0.00s) --- PASS: TestToParquetValue_DecimalValue/Decimal_out_of_int32_range_(stored_as_binary) (0.00s) --- PASS: TestToParquetValue_DecimalValue/Decimal_out_of_int64_range_(stored_as_binary) (0.00s) --- PASS: TestToParquetValue_DecimalValue/Decimal_extremely_large_value_(should_be_rejected) (0.00s) === RUN TestToParquetValue_TimeValue === RUN TestToParquetValue_TimeValue/Valid_TimeValue_(12:34:56.789) === RUN TestToParquetValue_TimeValue/TimeValue_midnight === RUN TestToParquetValue_TimeValue/TimeValue_end_of_day_(23:59:59.999999) === RUN TestToParquetValue_TimeValue/TimeValue_nil_pointer --- PASS: TestToParquetValue_TimeValue (0.00s) --- PASS: TestToParquetValue_TimeValue/Valid_TimeValue_(12:34:56.789) (0.00s) --- PASS: TestToParquetValue_TimeValue/TimeValue_midnight (0.00s) --- PASS: TestToParquetValue_TimeValue/TimeValue_end_of_day_(23:59:59.999999) (0.00s) --- PASS: TestToParquetValue_TimeValue/TimeValue_nil_pointer (0.00s) === RUN TestToParquetValue_EdgeCases === RUN TestToParquetValue_EdgeCases/Nil_value === RUN TestToParquetValue_EdgeCases/Completely_nil_value === RUN TestToParquetValue_EdgeCases/BytesValue_with_nil_slice --- PASS: TestToParquetValue_EdgeCases (0.00s) --- PASS: TestToParquetValue_EdgeCases/Nil_value (0.00s) --- PASS: TestToParquetValue_EdgeCases/Completely_nil_value (0.00s) --- PASS: TestToParquetValue_EdgeCases/BytesValue_with_nil_slice (0.00s) === RUN TestWriteReadParquet RecordType: fields:{name:"Address" type:{record_type:{fields:{name:"City" type:{scalar_type:STRING}} fields:{name:"Street" type:{scalar_type:STRING}}}}} fields:{name:"Company" type:{scalar_type:STRING}} fields:{name:"CreatedAt" type:{scalar_type:INT64}} fields:{name:"ID" type:{scalar_type:INT64}} fields:{name:"Person" type:{record_type:{fields:{name:"emails" type:{list_type:{element_type:{scalar_type:STRING}}}} fields:{name:"zName" type:{scalar_type:STRING}}}}} ParquetSchema: message example { optional group Address { optional binary City; optional binary Street; } optional binary Company; optional int64 CreatedAt (INT(64,true)); optional int64 ID (INT(64,true)); optional group Person { repeated binary emails; optional binary zName; } } Go Type: struct { Address *struct { City *[]uint8 "parquet:\"City\""; Street *[]uint8 "parquet:\"Street\"" } "parquet:\"Address\""; Company *[]uint8 "parquet:\"Company\""; CreatedAt *int64 "parquet:\"CreatedAt\""; ID *int64 "parquet:\"ID\""; Person *struct { Emails []*[]uint8 "parquet:\"emails\""; ZName *[]uint8 "parquet:\"zName\"" } "parquet:\"Person\"" } Write RecordValue: fields:{key:"Company" value:{string_value:"company_0"}} fields:{key:"CreatedAt" value:{int64_value:2}} fields:{key:"ID" value:{int64_value:1}} fields:{key:"Person" value:{record_value:{fields:{key:"emails" value:{list_value:{values:{string_value:"john_0@a.com"} values:{string_value:"john_0@b.com"} values:{string_value:"john_0@c.com"} values:{string_value:"john_0@d.com"} values:{string_value:"john_0@e.com"}}}} fields:{key:"zName" value:{string_value:"john_0"}}}}} Build Row: [C:0 D:2 R:0 V: C:1 D:2 R:0 V: C:2 D:1 R:0 V:company_0 C:3 D:1 R:0 V:2 C:4 D:1 R:0 V:1 C:5 D:2 R:0 V:john_0@a.com C:5 D:2 R:1 V:john_0@b.com C:5 D:2 R:1 V:john_0@c.com C:5 D:2 R:1 V:john_0@d.com C:5 D:2 R:1 V:john_0@e.com C:6 D:2 R:0 V:john_0] Write RecordValue: fields:{key:"Company" value:{string_value:"company_1"}} fields:{key:"CreatedAt" value:{int64_value:4}} fields:{key:"ID" value:{int64_value:2}} fields:{key:"Person" value:{record_value:{fields:{key:"emails" value:{list_value:{values:{string_value:"john_1@a.com"} values:{string_value:"john_1@b.com"} values:{string_value:"john_1@c.com"} values:{string_value:"john_1@d.com"} values:{string_value:"john_1@e.com"}}}} fields:{key:"zName" value:{string_value:"john_1"}}}}} Build Row: [C:0 D:2 R:0 V: C:1 D:2 R:0 V: C:2 D:1 R:0 V:company_1 C:3 D:1 R:0 V:4 C:4 D:1 R:0 V:2 C:5 D:2 R:0 V:john_1@a.com C:5 D:2 R:1 V:john_1@b.com C:5 D:2 R:1 V:john_1@c.com C:5 D:2 R:1 V:john_1@d.com C:5 D:2 R:1 V:john_1@e.com C:6 D:2 R:0 V:john_1] Write RecordValue: fields:{key:"Company" value:{string_value:"company_2"}} fields:{key:"CreatedAt" value:{int64_value:6}} fields:{key:"ID" value:{int64_value:3}} fields:{key:"Person" value:{record_value:{fields:{key:"emails" value:{list_value:{values:{string_value:"john_2@a.com"} values:{string_value:"john_2@b.com"} values:{string_value:"john_2@c.com"} values:{string_value:"john_2@d.com"} values:{string_value:"john_2@e.com"}}}} fields:{key:"zName" value:{string_value:"john_2"}}}}} Build Row: [C:0 D:2 R:0 V: C:1 D:2 R:0 V: C:2 D:1 R:0 V:company_2 C:3 D:1 R:0 V:6 C:4 D:1 R:0 V:3 C:5 D:2 R:0 V:john_2@a.com C:5 D:2 R:1 V:john_2@b.com C:5 D:2 R:1 V:john_2@c.com C:5 D:2 R:1 V:john_2@d.com C:5 D:2 R:1 V:john_2@e.com C:6 D:2 R:0 V:john_2] Read RecordValue: fields:{key:"Address" value:{record_value:{fields:{key:"City" value:{string_value:""}} fields:{key:"Street" value:{string_value:""}}}}} fields:{key:"Company" value:{string_value:"company_0"}} fields:{key:"CreatedAt" value:{int64_value:2}} fields:{key:"ID" value:{int64_value:1}} fields:{key:"Person" value:{record_value:{fields:{key:"emails" value:{list_value:{values:{string_value:"john_0@a.com"} values:{string_value:"john_0@b.com"} values:{string_value:"john_0@c.com"} values:{string_value:"john_0@d.com"} values:{string_value:"john_0@e.com"}}}} fields:{key:"zName" value:{string_value:"john_0"}}}}} Read RecordValue: fields:{key:"Address" value:{record_value:{fields:{key:"City" value:{string_value:""}} fields:{key:"Street" value:{string_value:""}}}}} fields:{key:"Company" value:{string_value:"company_1"}} fields:{key:"CreatedAt" value:{int64_value:4}} fields:{key:"ID" value:{int64_value:2}} fields:{key:"Person" value:{record_value:{fields:{key:"emails" value:{list_value:{values:{string_value:"john_1@a.com"} values:{string_value:"john_1@b.com"} values:{string_value:"john_1@c.com"} values:{string_value:"john_1@d.com"} values:{string_value:"john_1@e.com"}}}} fields:{key:"zName" value:{string_value:"john_1"}}}}} Read RecordValue: fields:{key:"Address" value:{record_value:{fields:{key:"City" value:{string_value:""}} fields:{key:"Street" value:{string_value:""}}}}} fields:{key:"Company" value:{string_value:"company_2"}} fields:{key:"CreatedAt" value:{int64_value:6}} fields:{key:"ID" value:{int64_value:3}} fields:{key:"Person" value:{record_value:{fields:{key:"emails" value:{list_value:{values:{string_value:"john_2@a.com"} values:{string_value:"john_2@b.com"} values:{string_value:"john_2@c.com"} values:{string_value:"john_2@d.com"} values:{string_value:"john_2@e.com"}}}} fields:{key:"zName" value:{string_value:"john_2"}}}}} total: 3 --- PASS: TestWriteReadParquet (1.14s) PASS ok github.com/seaweedfs/seaweedfs/weed/mq/schema 9.101s ? github.com/seaweedfs/seaweedfs/weed/mq/segment [no test files] === RUN TestAddConsumerInstance &{isAssign:true partition:{RangeStart:0 RangeStop:1 RingSize:3 UnixTimeNs:0} consumer:first ts:{wall:14022888368696733180 ext:12791774265 loc:0x160a640}} &{isAssign:true partition:{RangeStart:1 RangeStop:2 RingSize:3 UnixTimeNs:0} consumer:first ts:{wall:14022888368696788760 ext:12791826185 loc:0x160a640}} --- PASS: TestAddConsumerInstance (1.03s) === RUN TestMultipleConsumerInstances &{isAssign:true partition:{RangeStart:0 RangeStop:1 RingSize:3 UnixTimeNs:0} consumer:third ts:{wall:14022888369771676373 ext:13792977314 loc:0x160a640}} &{isAssign:true partition:{RangeStart:1 RangeStop:2 RingSize:3 UnixTimeNs:0} consumer:first ts:{wall:14022888369771726653 ext:13793024114 loc:0x160a640}} &{isAssign:true partition:{RangeStart:2 RangeStop:3 RingSize:3 UnixTimeNs:0} consumer:second ts:{wall:14022888369771746073 ext:13793041894 loc:0x160a640}} --- PASS: TestMultipleConsumerInstances (1.01s) === RUN TestConfirmAdjustment &{isAssign:true partition:{RangeStart:0 RangeStop:1 RingSize:3 UnixTimeNs:0} consumer:third ts:{wall:14022888370862662761 ext:14810217338 loc:0x160a640}} &{isAssign:true partition:{RangeStart:1 RangeStop:2 RingSize:3 UnixTimeNs:0} consumer:first ts:{wall:14022888370862745181 ext:14810299718 loc:0x160a640}} &{isAssign:true partition:{RangeStart:2 RangeStop:3 RingSize:3 UnixTimeNs:0} consumer:second ts:{wall:14022888370862757381 ext:14810311098 loc:0x160a640}} &{isAssign:true partition:{RangeStart:0 RangeStop:1 RingSize:3 UnixTimeNs:0} consumer:first ts:{wall:14022888373010207449 ext:16810277858 loc:0x160a640}} --- PASS: TestConfirmAdjustment (5.01s) PASS ok github.com/seaweedfs/seaweedfs/weed/mq/sub_coordinator 24.658s === RUN TestOffsetBasedSubscribe_AllDataInMemory === RUN TestOffsetBasedSubscribe_AllDataInMemory/ReadFromOffset0 === RUN TestOffsetBasedSubscribe_AllDataInMemory/ReadFromOffset2 --- PASS: TestOffsetBasedSubscribe_AllDataInMemory (0.03s) --- PASS: TestOffsetBasedSubscribe_AllDataInMemory/ReadFromOffset0 (0.00s) --- PASS: TestOffsetBasedSubscribe_AllDataInMemory/ReadFromOffset2 (0.00s) === RUN TestOffsetBasedSubscribe_DataOnDisk === RUN TestOffsetBasedSubscribe_DataOnDisk/ReadFromOffset0_OnDisk === RUN TestOffsetBasedSubscribe_DataOnDisk/ReadFromOffset5_OnDisk === RUN TestOffsetBasedSubscribe_DataOnDisk/ReadFromOffset11_InMemory --- PASS: TestOffsetBasedSubscribe_DataOnDisk (0.00s) --- PASS: TestOffsetBasedSubscribe_DataOnDisk/ReadFromOffset0_OnDisk (0.00s) --- PASS: TestOffsetBasedSubscribe_DataOnDisk/ReadFromOffset5_OnDisk (0.00s) --- PASS: TestOffsetBasedSubscribe_DataOnDisk/ReadFromOffset11_InMemory (0.00s) === RUN TestTimestampBasedSubscribe === RUN TestTimestampBasedSubscribe/ReadFromBeginning === RUN TestTimestampBasedSubscribe/ReadFromMiddleTimestamp --- PASS: TestTimestampBasedSubscribe (0.00s) --- PASS: TestTimestampBasedSubscribe/ReadFromBeginning (0.00s) --- PASS: TestTimestampBasedSubscribe/ReadFromMiddleTimestamp (0.00s) === RUN TestConcurrentSubscribers --- PASS: TestConcurrentSubscribers (0.00s) === RUN TestResumeFromDiskError --- PASS: TestResumeFromDiskError (0.00s) PASS ok github.com/seaweedfs/seaweedfs/weed/mq/topic 13.745s ? github.com/seaweedfs/seaweedfs/weed/notification [no test files] ? github.com/seaweedfs/seaweedfs/weed/notification/aws_sqs [no test files] ? github.com/seaweedfs/seaweedfs/weed/notification/gocdk_pub_sub [no test files] ? github.com/seaweedfs/seaweedfs/weed/notification/google_pub_sub [no test files] === RUN TestConfigureSASLTLSRejectsPartialMTLSConfig === RUN TestConfigureSASLTLSRejectsPartialMTLSConfig/missing_key === RUN TestConfigureSASLTLSRejectsPartialMTLSConfig/missing_cert --- PASS: TestConfigureSASLTLSRejectsPartialMTLSConfig (0.00s) --- PASS: TestConfigureSASLTLSRejectsPartialMTLSConfig/missing_key (0.00s) --- PASS: TestConfigureSASLTLSRejectsPartialMTLSConfig/missing_cert (0.00s) === RUN TestConfigureSASLTLSConfiguresSCRAMSHA256 --- PASS: TestConfigureSASLTLSConfiguresSCRAMSHA256 (0.00s) PASS ok github.com/seaweedfs/seaweedfs/weed/notification/kafka 12.545s ? github.com/seaweedfs/seaweedfs/weed/notification/log [no test files] === RUN TestFilterEventTypes === RUN TestFilterEventTypes/create_event_-_allowed === RUN TestFilterEventTypes/create_event_-_not_allowed === RUN TestFilterEventTypes/delete_event_-_allowed === RUN TestFilterEventTypes/update_event_-_allowed === RUN TestFilterEventTypes/rename_event_-_allowed === RUN TestFilterEventTypes/rename_event_same_name_different_parent_-_allowed === RUN TestFilterEventTypes/rename_event_-_not_allowed === RUN TestFilterEventTypes/all_events_allowed_when_empty --- PASS: TestFilterEventTypes (0.04s) --- PASS: TestFilterEventTypes/create_event_-_allowed (0.00s) --- PASS: TestFilterEventTypes/create_event_-_not_allowed (0.00s) --- PASS: TestFilterEventTypes/delete_event_-_allowed (0.00s) --- PASS: TestFilterEventTypes/update_event_-_allowed (0.00s) --- PASS: TestFilterEventTypes/rename_event_-_allowed (0.00s) --- PASS: TestFilterEventTypes/rename_event_same_name_different_parent_-_allowed (0.00s) --- PASS: TestFilterEventTypes/rename_event_-_not_allowed (0.00s) --- PASS: TestFilterEventTypes/all_events_allowed_when_empty (0.00s) === RUN TestFilterPathPrefixes === RUN TestFilterPathPrefixes/matches_single_prefix === RUN TestFilterPathPrefixes/matches_one_of_multiple_prefixes === RUN TestFilterPathPrefixes/no_match === RUN TestFilterPathPrefixes/empty_prefixes_allows_all === RUN TestFilterPathPrefixes/exact_prefix_match === RUN TestFilterPathPrefixes/partial_match_not_allowed === RUN TestFilterPathPrefixes/rename_matches_destination_prefix --- PASS: TestFilterPathPrefixes (0.00s) --- PASS: TestFilterPathPrefixes/matches_single_prefix (0.00s) --- PASS: TestFilterPathPrefixes/matches_one_of_multiple_prefixes (0.00s) --- PASS: TestFilterPathPrefixes/no_match (0.00s) --- PASS: TestFilterPathPrefixes/empty_prefixes_allows_all (0.00s) --- PASS: TestFilterPathPrefixes/exact_prefix_match (0.00s) --- PASS: TestFilterPathPrefixes/partial_match_not_allowed (0.00s) --- PASS: TestFilterPathPrefixes/rename_matches_destination_prefix (0.00s) === RUN TestFilterCombined === RUN TestFilterCombined/allowed_event_and_path === RUN TestFilterCombined/allowed_event_but_wrong_path === RUN TestFilterCombined/wrong_event_but_allowed_path === RUN TestFilterCombined/wrong_event_and_wrong_path --- PASS: TestFilterCombined (0.00s) --- PASS: TestFilterCombined/allowed_event_and_path (0.00s) --- PASS: TestFilterCombined/allowed_event_but_wrong_path (0.00s) --- PASS: TestFilterCombined/wrong_event_but_allowed_path (0.00s) --- PASS: TestFilterCombined/wrong_event_and_wrong_path (0.00s) === RUN TestHttpClientSendMessage --- PASS: TestHttpClientSendMessage (0.12s) === RUN TestHttpClientSendMessageWithoutToken --- PASS: TestHttpClientSendMessageWithoutToken (0.04s) === RUN TestHttpClientSendMessageServerError --- PASS: TestHttpClientSendMessageServerError (0.02s) === RUN TestHttpClientSendMessageNetworkError --- PASS: TestHttpClientSendMessageNetworkError (0.00s) === RUN TestHttpClientFollowsRedirectAsPost --- PASS: TestHttpClientFollowsRedirectAsPost (0.04s) === RUN TestHttpClientUsesCachedRedirect --- PASS: TestHttpClientUsesCachedRedirect (0.03s) === RUN TestHttpClientPreservesPostMethod --- PASS: TestHttpClientPreservesPostMethod (0.02s) === RUN TestHttpClientInvalidatesCacheOnError --- PASS: TestHttpClientInvalidatesCacheOnError (0.07s) === RUN TestHttpClientInvalidatesCacheOnNetworkError --- PASS: TestHttpClientInvalidatesCacheOnNetworkError (0.05s) === RUN TestConfigValidation === RUN TestConfigValidation/valid_config === RUN TestConfigValidation/empty_endpoint === RUN TestConfigValidation/invalid_URL === RUN TestConfigValidation/timeout_too_large === RUN TestConfigValidation/too_many_retries === RUN TestConfigValidation/too_many_workers === RUN TestConfigValidation/buffer_too_large --- PASS: TestConfigValidation (0.00s) --- PASS: TestConfigValidation/valid_config (0.00s) --- PASS: TestConfigValidation/empty_endpoint (0.00s) --- PASS: TestConfigValidation/invalid_URL (0.00s) --- PASS: TestConfigValidation/timeout_too_large (0.00s) --- PASS: TestConfigValidation/too_many_retries (0.00s) --- PASS: TestConfigValidation/too_many_workers (0.00s) --- PASS: TestConfigValidation/buffer_too_large (0.00s) === RUN TestWebhookMessageSerialization --- PASS: TestWebhookMessageSerialization (0.02s) === RUN TestQueueInitialize [watermill] 2026/08/24 22:18:21.826799 router.go:280: level=INFO msg="Adding handler" handler_name=webhook_handler topic=webhook_topic [watermill] 2026/08/24 22:18:21.836272 router.go:434: level=INFO msg="Running router handlers" count=1 [watermill] 2026/08/24 22:18:21.844707 router.go:646: level=INFO msg="Starting handler" subscriber_name=webhook_handler topic=webhook_topic [watermill] 2026/08/24 22:18:21.844989 router.go:481: level=INFO msg="Subscriber stopped" subscriber_name=webhook_handler topic=webhook_topic [watermill] 2026/08/24 22:18:21.845107 router.go:578: level=INFO msg="Closing router" [watermill] 2026/08/24 22:18:21.848385 router.go:413: level=INFO msg="Waiting for messages" timeout=1m0s [watermill] 2026/08/24 22:18:21.848638 router.go:589: level=INFO msg="Router closed" [watermill] 2026/08/24 22:18:21.848776 router.go:419: level=INFO msg="All messages processed" I0824 22:18:21.848951 webhook_queue.go:170 webhook pubsub worker stopped --- PASS: TestQueueInitialize (0.15s) === RUN TestQueueSendMessage [watermill] 2026/08/24 22:18:21.939764 router.go:280: level=INFO msg="Adding handler" handler_name=webhook_handler topic=webhook_topic [watermill] 2026/08/24 22:18:21.941729 pubsub.go:160: level=INFO msg="No subscribers to send message" message_uuid=f972f0d6-804a-4e51-87df-fe7c42c4996b pubsub_uuid=aGCxYTPBm8zu8XyepdzuHf topic=webhook_topic [watermill] 2026/08/24 22:18:21.942639 router.go:434: level=INFO msg="Running router handlers" count=1 [watermill] 2026/08/24 22:18:21.943435 router.go:646: level=INFO msg="Starting handler" subscriber_name=webhook_handler topic=webhook_topic [watermill] 2026/08/24 22:18:21.943779 router.go:481: level=INFO msg="Subscriber stopped" subscriber_name=webhook_handler topic=webhook_topic [watermill] 2026/08/24 22:18:21.944158 router.go:578: level=INFO msg="Closing router" [watermill] 2026/08/24 22:18:21.948495 router.go:413: level=INFO msg="Waiting for messages" timeout=1m0s [watermill] 2026/08/24 22:18:21.948572 router.go:419: level=INFO msg="All messages processed" I0824 22:18:21.950610 webhook_queue.go:170 webhook pubsub worker stopped [watermill] 2026/08/24 22:18:21.948525 router.go:589: level=INFO msg="Router closed" --- PASS: TestQueueSendMessage (0.11s) === RUN TestQueueHandleWebhook --- PASS: TestQueueHandleWebhook (0.01s) === RUN TestQueueEndToEnd [watermill] 2026/08/24 22:18:22.065979 router.go:280: level=INFO msg="Adding handler" handler_name=webhook_handler topic=webhook_topic [watermill] 2026/08/24 22:18:22.066488 pubsub.go:160: level=INFO msg="No subscribers to send message" message_uuid=f2377840-fd40-48c8-a8a8-5ee829632224 pubsub_uuid=excyd44U4THxeDWtkHBek9 topic=webhook_topic [watermill] 2026/08/24 22:18:22.067373 router.go:434: level=INFO msg="Running router handlers" count=1 [watermill] 2026/08/24 22:18:22.068370 router.go:646: level=INFO msg="Starting handler" subscriber_name=webhook_handler topic=webhook_topic [watermill] 2026/08/24 22:18:22.068657 router.go:481: level=INFO msg="Subscriber stopped" subscriber_name=webhook_handler topic=webhook_topic [watermill] 2026/08/24 22:18:22.068775 router.go:578: level=INFO msg="Closing router" [watermill] 2026/08/24 22:18:22.069190 router.go:413: level=INFO msg="Waiting for messages" timeout=1m0s [watermill] 2026/08/24 22:18:22.069571 router.go:589: level=INFO msg="Router closed" [watermill] 2026/08/24 22:18:22.069689 router.go:419: level=INFO msg="All messages processed" I0824 22:18:22.069855 webhook_queue.go:170 webhook pubsub worker stopped --- PASS: TestQueueEndToEnd (0.12s) === RUN TestQueueRetryMechanism [watermill] 2026/08/24 22:18:22.188695 router.go:280: level=INFO msg="Adding handler" handler_name=webhook_handler topic=webhook_topic [watermill] 2026/08/24 22:18:22.190176 pubsub.go:160: level=INFO msg="No subscribers to send message" message_uuid=052dd324-1829-4fa3-a460-95151793c292 pubsub_uuid=DkgvHueKysifRAtPtpUqkM topic=webhook_topic [watermill] 2026/08/24 22:18:22.191635 router.go:434: level=INFO msg="Running router handlers" count=1 [watermill] 2026/08/24 22:18:22.192486 router.go:646: level=INFO msg="Starting handler" subscriber_name=webhook_handler topic=webhook_topic [watermill] 2026/08/24 22:18:22.192915 router.go:481: level=INFO msg="Subscriber stopped" subscriber_name=webhook_handler topic=webhook_topic [watermill] 2026/08/24 22:18:22.193150 router.go:578: level=INFO msg="Closing router" [watermill] 2026/08/24 22:18:22.193803 router.go:413: level=INFO msg="Waiting for messages" timeout=1m0s [watermill] 2026/08/24 22:18:22.194371 router.go:589: level=INFO msg="Router closed" [watermill] 2026/08/24 22:18:22.194642 router.go:419: level=INFO msg="All messages processed" I0824 22:18:22.194744 webhook_queue.go:170 webhook pubsub worker stopped --- PASS: TestQueueRetryMechanism (0.11s) === RUN TestQueueNoDuplicateWebhooks === RUN TestQueueNoDuplicateWebhooks/1_worker [watermill] 2026/08/24 22:18:22.299134 router.go:280: level=INFO msg="Adding handler" handler_name=webhook_handler topic=webhook_topic [watermill] 2026/08/24 22:18:22.300778 router.go:434: level=INFO msg="Running router handlers" count=1 [watermill] 2026/08/24 22:18:22.301561 router.go:646: level=INFO msg="Starting handler" subscriber_name=webhook_handler topic=webhook_topic [watermill] 2026/08/24 22:18:23.014567 router.go:481: level=INFO msg="Subscriber stopped" subscriber_name=webhook_handler topic=webhook_topic [watermill] 2026/08/24 22:18:23.014797 router.go:578: level=INFO msg="Closing router" [watermill] 2026/08/24 22:18:23.015078 router.go:413: level=INFO msg="Waiting for messages" timeout=1m0s [watermill] 2026/08/24 22:18:23.015151 router.go:589: level=INFO msg="Router closed" [watermill] 2026/08/24 22:18:23.015196 router.go:419: level=INFO msg="All messages processed" I0824 22:18:23.015246 webhook_queue.go:170 webhook pubsub worker stopped === RUN TestQueueNoDuplicateWebhooks/5_workers [watermill] 2026/08/24 22:18:23.126648 router.go:280: level=INFO msg="Adding handler" handler_name=webhook_handler topic=webhook_topic [watermill] 2026/08/24 22:18:23.128140 router.go:434: level=INFO msg="Running router handlers" count=1 [watermill] 2026/08/24 22:18:23.128591 router.go:646: level=INFO msg="Starting handler" subscriber_name=webhook_handler topic=webhook_topic [watermill] 2026/08/24 22:18:23.844538 router.go:481: level=INFO msg="Subscriber stopped" subscriber_name=webhook_handler topic=webhook_topic [watermill] 2026/08/24 22:18:23.844869 router.go:578: level=INFO msg="Closing router" [watermill] 2026/08/24 22:18:23.845069 router.go:413: level=INFO msg="Waiting for messages" timeout=1m0s [watermill] 2026/08/24 22:18:23.845231 router.go:589: level=INFO msg="Router closed" [watermill] 2026/08/24 22:18:23.845300 router.go:419: level=INFO msg="All messages processed" I0824 22:18:23.845382 webhook_queue.go:170 webhook pubsub worker stopped === RUN TestQueueNoDuplicateWebhooks/10_workers [watermill] 2026/08/24 22:18:23.961746 router.go:280: level=INFO msg="Adding handler" handler_name=webhook_handler topic=webhook_topic [watermill] 2026/08/24 22:18:23.966427 router.go:434: level=INFO msg="Running router handlers" count=1 [watermill] 2026/08/24 22:18:23.966938 router.go:646: level=INFO msg="Starting handler" subscriber_name=webhook_handler topic=webhook_topic [watermill] 2026/08/24 22:18:24.690481 router.go:481: level=INFO msg="Subscriber stopped" subscriber_name=webhook_handler topic=webhook_topic [watermill] 2026/08/24 22:18:24.690860 router.go:578: level=INFO msg="Closing router" [watermill] 2026/08/24 22:18:24.691031 router.go:413: level=INFO msg="Waiting for messages" timeout=1m0s [watermill] 2026/08/24 22:18:24.691306 router.go:589: level=INFO msg="Router closed" [watermill] 2026/08/24 22:18:24.692544 router.go:419: level=INFO msg="All messages processed" I0824 22:18:24.693649 webhook_queue.go:170 webhook pubsub worker stopped --- PASS: TestQueueNoDuplicateWebhooks (2.50s) --- PASS: TestQueueNoDuplicateWebhooks/1_worker (0.82s) --- PASS: TestQueueNoDuplicateWebhooks/5_workers (0.83s) --- PASS: TestQueueNoDuplicateWebhooks/10_workers (0.84s) === RUN TestQueueSendMessageWithFilter === RUN TestQueueSendMessageWithFilter/allowed_event_type === RUN TestQueueSendMessageWithFilter/filtered_event_type === RUN TestQueueSendMessageWithFilter/allowed_path_prefix === RUN TestQueueSendMessageWithFilter/filtered_path_prefix === RUN TestQueueSendMessageWithFilter/combined_filters_-_both_pass === RUN TestQueueSendMessageWithFilter/combined_filters_-_event_fails === RUN TestQueueSendMessageWithFilter/combined_filters_-_path_fails --- PASS: TestQueueSendMessageWithFilter (0.00s) --- PASS: TestQueueSendMessageWithFilter/allowed_event_type (0.00s) --- PASS: TestQueueSendMessageWithFilter/filtered_event_type (0.00s) --- PASS: TestQueueSendMessageWithFilter/allowed_path_prefix (0.00s) --- PASS: TestQueueSendMessageWithFilter/filtered_path_prefix (0.00s) --- PASS: TestQueueSendMessageWithFilter/combined_filters_-_both_pass (0.00s) --- PASS: TestQueueSendMessageWithFilter/combined_filters_-_event_fails (0.00s) --- PASS: TestQueueSendMessageWithFilter/combined_filters_-_path_fails (0.00s) PASS ok github.com/seaweedfs/seaweedfs/weed/notification/webhook 15.842s === RUN TestAssignRetriesOnUnavailable --- PASS: TestAssignRetriesOnUnavailable (5.17s) === RUN TestAssignStopsOnContextCancel --- PASS: TestAssignStopsOnContextCancel (2.03s) === RUN TestLookupVolumeIdCacheInvalidation --- PASS: TestLookupVolumeIdCacheInvalidation (0.05s) === RUN TestLookupRaftLeaderMasterFromFollower --- PASS: TestLookupRaftLeaderMasterFromFollower (0.03s) === RUN TestLookupRaftLeaderMasterTriesNextPeer --- PASS: TestLookupRaftLeaderMasterTriesNextPeer (0.03s) === RUN TestLookupRaftLeaderMasterNoPeers --- PASS: TestLookupRaftLeaderMasterNoPeers (0.00s) === RUN TestLookupRaftLeaderMasterCanceledContext --- PASS: TestLookupRaftLeaderMasterCanceledContext (0.00s) === RUN TestCaching vid 123 locations = [{a.com:8080 0}] --- PASS: TestCaching (2.01s) === RUN TestCachingDelete --- PASS: TestCachingDelete (0.00s) === RUN TestCachingLargeVolumeId --- PASS: TestCachingLargeVolumeId (0.00s) === RUN TestCreateNeedleFromRequest needle: 0f084d17353afda0 Size:0, DataSize:0, Name:t.txt, Mime:text/plain; charset=utf-8 Compressed:true, originalSize: 1422 W0824 22:18:35.698891 upload_content.go:334 uploading 0 to http://localhost:8080/389,0f084d17353afda0: upload t.txt 816 bytes to http://localhost:8080/389,0f084d17353afda0: EOF needle: 0f084d17353afda0 Size:0, DataSize:0, Name:t.txt, Mime:text/plain; charset=utf-8 Compressed:true, originalSize: 1422 W0824 22:18:36.244293 upload_content.go:334 uploading 1 to http://localhost:8080/389,0f084d17353afda0: upload t.txt 816 bytes to http://localhost:8080/389,0f084d17353afda0: EOF needle: 0f084d17353afda0 Size:0, DataSize:0, Name:t.txt, Mime:text/plain; charset=utf-8 Compressed:true, originalSize: 1422 W0824 22:18:36.968885 upload_content.go:334 uploading 2 to http://localhost:8080/389,0f084d17353afda0: upload t.txt 816 bytes to http://localhost:8080/389,0f084d17353afda0: EOF err: upload t.txt 816 bytes to http://localhost:8080/389,0f084d17353afda0: EOF uploadResult: needle: 0f084d17353afda0 Size:0, DataSize:0, Name:t.txt, Mime:text/plain Compressed:true, dataSize:816 originalSize:1422 W0824 22:18:36.970465 upload_content.go:334 uploading 0 to http://localhost:8080/389,0f084d17353afda0: upload t.txt 816 bytes to http://localhost:8080/389,0f084d17353afda0: EOF needle: 0f084d17353afda0 Size:0, DataSize:0, Name:t.txt, Mime:text/plain Compressed:true, dataSize:816 originalSize:1422 W0824 22:18:37.449260 upload_content.go:334 uploading 1 to http://localhost:8080/389,0f084d17353afda0: upload t.txt 816 bytes to http://localhost:8080/389,0f084d17353afda0: EOF needle: 0f084d17353afda0 Size:0, DataSize:0, Name:t.txt, Mime:text/plain Compressed:true, dataSize:816 originalSize:1422 W0824 22:18:38.181706 upload_content.go:334 uploading 2 to http://localhost:8080/389,0f084d17353afda0: upload t.txt 816 bytes to http://localhost:8080/389,0f084d17353afda0: EOF --- PASS: TestCreateNeedleFromRequest (2.62s) === RUN TestUploadReaderInChunksReturnsPartialResultsOnError E0824 22:18:38.218412 upload_chunked.go:281 chunked upload failed: assign volume: simulated volume assignment failure (returning 1 partial chunks for cleanup) upload_chunked_test.go:74: ✓ Got expected error: assign volume: simulated volume assignment failure upload_chunked_test.go:80: ✓ Result is not nil (partial results returned) upload_chunked_test.go:86: ✓ Result contains 1 chunks (may be 0 if all assignments failed) upload_chunked_test.go:92: ✓ Md5Hash is available for partial data upload_chunked_test.go:99: ✓ TotalSize = 24576 bytes read before failure --- PASS: TestUploadReaderInChunksReturnsPartialResultsOnError (0.05s) === RUN TestUploadReaderInChunksSuccessPath upload_chunked_test.go:142: ✓ Upload completed without error upload_chunked_test.go:148: ✓ Result is not nil upload_chunked_test.go:154: ✓ Result contains 1 file chunk(s) upload_chunked_test.go:161: ✓ TotalSize=15 matches input data upload_chunked_test.go:168: ✓ Md5Hash is available upload_chunked_test.go:183: ✓ Chunk properties validated: FileId=test-fid,1234, Offset=0, Size=15 --- PASS: TestUploadReaderInChunksSuccessPath (0.00s) === RUN TestUploadReaderInChunksContextCancellation E0824 22:18:38.243844 upload_chunked.go:281 chunked upload failed: context canceled (returning 0 partial chunks for cleanup) upload_chunked_test.go:237: ✓ Got partial result on cancellation: chunks=0 --- PASS: TestUploadReaderInChunksContextCancellation (0.01s) === RUN TestUploadChunkToHoldersRollsBackOnPartialFailure E0824 22:18:38.295901 upload_content.go:564 unmarshal http://127.0.0.1:34687/3,01abcdef?type=replicate: boom W0824 22:18:38.306531 upload_content.go:334 uploading 0 to http://127.0.0.1:34687/3,01abcdef?type=replicate: unmarshal http://127.0.0.1:34687/3,01abcdef?type=replicate: invalid character 'b' looking for beginning of value E0824 22:18:38.821384 upload_content.go:564 unmarshal http://127.0.0.1:34687/3,01abcdef?type=replicate: boom W0824 22:18:38.823040 upload_content.go:334 uploading 1 to http://127.0.0.1:34687/3,01abcdef?type=replicate: unmarshal http://127.0.0.1:34687/3,01abcdef?type=replicate: invalid character 'b' looking for beginning of value E0824 22:18:39.550554 upload_content.go:564 unmarshal http://127.0.0.1:34687/3,01abcdef?type=replicate: boom W0824 22:18:39.550933 upload_content.go:334 uploading 2 to http://127.0.0.1:34687/3,01abcdef?type=replicate: unmarshal http://127.0.0.1:34687/3,01abcdef?type=replicate: invalid character 'b' looking for beginning of value --- PASS: TestUploadChunkToHoldersRollsBackOnPartialFailure (1.33s) === RUN TestUploadReaderInChunksReaderFailure E0824 22:18:39.580108 upload_chunked.go:281 chunked upload failed: read chunk at offset 8192 (got 1808 bytes): simulated read failure (returning 1 partial chunks for cleanup) upload_chunked_test.go:374: ✓ Got partial result on read failure: chunks=1, totalSize=8192 --- PASS: TestUploadReaderInChunksReaderFailure (0.01s) === RUN TestUploadReaderInChunksTagsTruncatedBody E0824 22:18:39.589168 upload_chunked.go:281 chunked upload failed: truncated request body: read chunk at offset 8192 (got 1808 bytes): unexpected EOF (returning 1 partial chunks for cleanup) --- PASS: TestUploadReaderInChunksTagsTruncatedBody (0.01s) === RUN TestUploadReaderInChunksReassignsOnFullVolume W0824 22:18:39.904288 upload_chunked.go:421 replica fan-out cleanup: delete 1,0a0b0c0d from volume-1:8080: Delete "http://volume-1:8080/1,0a0b0c0d?type=replicate": dial tcp: lookup volume-1: no such host --- PASS: TestUploadReaderInChunksReassignsOnFullVolume (0.31s) === RUN TestUploadReaderInChunksDoesNotReassignOnClientError W0824 22:18:39.935195 upload_chunked.go:421 replica fan-out cleanup: delete 1,0a0b0c0d from volume-1:8080: Delete "http://volume-1:8080/1,0a0b0c0d?type=replicate": dial tcp: lookup volume-1: no such host E0824 22:18:39.935856 upload_chunked.go:281 chunked upload failed: upload chunk: mismatching cookie (returning 0 partial chunks for cleanup) --- PASS: TestUploadReaderInChunksDoesNotReassignOnClientError (0.03s) === RUN TestUploadReaderInChunksBoundsReassignment W0824 22:18:39.964316 upload_chunked.go:421 replica fan-out cleanup: delete 1,0a0b0c0d from volume-1:8080: Delete "http://volume-1:8080/1,0a0b0c0d?type=replicate": dial tcp: lookup volume-1: no such host W0824 22:18:40.250350 upload_chunked.go:421 replica fan-out cleanup: delete 2,0a0b0c0d from volume-2:8080: Delete "http://volume-2:8080/2,0a0b0c0d?type=replicate": dial tcp: lookup volume-2: no such host W0824 22:18:40.877895 upload_chunked.go:421 replica fan-out cleanup: delete 3,0a0b0c0d from volume-3:8080: Delete "http://volume-3:8080/3,0a0b0c0d?type=replicate": dial tcp: lookup volume-3: no such host E0824 22:18:40.878514 upload_chunked.go:281 chunked upload failed: upload chunk: volume full (returning 0 partial chunks for cleanup) --- PASS: TestUploadReaderInChunksBoundsReassignment (0.94s) === RUN TestUploadReaderInChunksReassignsAcrossHolders W0824 22:18:40.897866 upload_content.go:334 uploading 0 to http://127.0.0.1:46781/1,0a0b0c0d?type=replicate: unmarshalled error http://127.0.0.1:46781/1,0a0b0c0d?type=replicate: failed to write to local disk: Volume Size 34361499680 Exceeded 34359738368 W0824 22:18:41.566603 upload_content.go:334 uploading 1 to http://127.0.0.1:46781/1,0a0b0c0d?type=replicate: unmarshalled error http://127.0.0.1:46781/1,0a0b0c0d?type=replicate: failed to write to local disk: Volume Size 34361499680 Exceeded 34359738368 W0824 22:18:42.286383 upload_content.go:334 uploading 2 to http://127.0.0.1:46781/1,0a0b0c0d?type=replicate: unmarshalled error http://127.0.0.1:46781/1,0a0b0c0d?type=replicate: failed to write to local disk: Volume Size 34361499680 Exceeded 34359738368 --- PASS: TestUploadReaderInChunksReassignsAcrossHolders (1.44s) === RUN TestUploadReaderInChunksDoesNotMultiplyRelayAttempts W0824 22:18:42.338132 upload_content.go:334 uploading 0 to http://127.0.0.1:44627/1,0a0b0c0d: unmarshalled error http://127.0.0.1:44627/1,0a0b0c0d: failed to write to local disk: Volume Size 34361499680 Exceeded 34359738368 W0824 22:18:42.345391 upload_content.go:334 uploading 0 to http://127.0.0.1:44627/2,0a0b0c0d: unmarshalled error http://127.0.0.1:44627/2,0a0b0c0d: failed to write to local disk: Volume Size 34361499680 Exceeded 34359738368 W0824 22:18:42.351061 upload_content.go:334 uploading 0 to http://127.0.0.1:44627/3,0a0b0c0d: unmarshalled error http://127.0.0.1:44627/3,0a0b0c0d: failed to write to local disk: Volume Size 34361499680 Exceeded 34359738368 E0824 22:18:42.354704 upload_chunked.go:281 chunked upload failed: upload chunk: unmarshalled error http://127.0.0.1:44627/3,0a0b0c0d: failed to write to local disk: Volume Size 34361499680 Exceeded 34359738368 (returning 0 partial chunks for cleanup) --- PASS: TestUploadReaderInChunksDoesNotMultiplyRelayAttempts (0.04s) === RUN TestShouldReassignUpload === RUN TestShouldReassignUpload/nil === RUN TestShouldReassignUpload/no_response_(transport) === RUN TestShouldReassignUpload/500_replica_write_failed === RUN TestShouldReassignUpload/503_service_unavailable === RUN TestShouldReassignUpload/400_bad_request === RUN TestShouldReassignUpload/401_unauthorized === RUN TestShouldReassignUpload/assign_rpc_transport === RUN TestShouldReassignUpload/other_permanent --- PASS: TestShouldReassignUpload (0.00s) --- PASS: TestShouldReassignUpload/nil (0.00s) --- PASS: TestShouldReassignUpload/no_response_(transport) (0.00s) --- PASS: TestShouldReassignUpload/500_replica_write_failed (0.00s) --- PASS: TestShouldReassignUpload/503_service_unavailable (0.00s) --- PASS: TestShouldReassignUpload/400_bad_request (0.00s) --- PASS: TestShouldReassignUpload/401_unauthorized (0.00s) --- PASS: TestShouldReassignUpload/assign_rpc_transport (0.00s) --- PASS: TestShouldReassignUpload/other_permanent (0.00s) === RUN TestUploadWithRetryDataReassignsOnVolumeSizeExceeded W0824 22:18:42.358767 upload_content.go:334 uploading 0 to http://volume-a/1,first: unmarshalled error http://volume-a/1,first: failed to write to local disk: append to volume 1 size 0 actualSize 0: Volume Size 33555976 Exceeded 33554432 W0824 22:18:42.852611 upload_content.go:334 uploading 1 to http://volume-a/1,first: unmarshalled error http://volume-a/1,first: failed to write to local disk: append to volume 1 size 0 actualSize 0: Volume Size 33555976 Exceeded 33554432 W0824 22:18:43.565743 upload_content.go:334 uploading 2 to http://volume-a/1,first: unmarshalled error http://volume-a/1,first: failed to write to local disk: append to volume 1 size 0 actualSize 0: Volume Size 33555976 Exceeded 33554432 I0824 22:18:43.566022 retry.go:144 retry uploadWithRetry: err: unmarshalled error http://volume-a/1,first: failed to write to local disk: append to volume 1 size 0 actualSize 0: Volume Size 33555976 Exceeded 33554432 I0824 22:18:44.577201 retry.go:138 retry uploadWithRetry successfully --- PASS: TestUploadWithRetryDataReassignsOnVolumeSizeExceeded (2.22s) === RUN TestUploadWithRetryDataReassignsOnReplicaWriteFailure W0824 22:18:44.578101 upload_content.go:334 uploading 0 to http://volume-a/1,first: unmarshalled error http://volume-a/1,first: failed to write to replicas for volume 1: [volume-down:8080]: dial tcp: lookup volume-down: no such host W0824 22:18:45.057336 upload_content.go:334 uploading 1 to http://volume-a/1,first: unmarshalled error http://volume-a/1,first: failed to write to replicas for volume 1: [volume-down:8080]: dial tcp: lookup volume-down: no such host W0824 22:18:45.775136 upload_content.go:334 uploading 2 to http://volume-a/1,first: unmarshalled error http://volume-a/1,first: failed to write to replicas for volume 1: [volume-down:8080]: dial tcp: lookup volume-down: no such host I0824 22:18:45.775390 retry.go:144 retry uploadWithRetry: err: unmarshalled error http://volume-a/1,first: failed to write to replicas for volume 1: [volume-down:8080]: dial tcp: lookup volume-down: no such host I0824 22:18:46.784553 retry.go:138 retry uploadWithRetry successfully --- PASS: TestUploadWithRetryDataReassignsOnReplicaWriteFailure (2.21s) === RUN TestUploadWithRetryBoundsAssignVolume I0824 22:18:46.995085 upload_content.go:266 assign volume failure count:1 expected_data_size:3: context deadline exceeded --- PASS: TestUploadWithRetryBoundsAssignVolume (0.25s) === RUN TestUploadRewindsBodyOnConnectionReset === RUN TestUploadRewindsBodyOnConnectionReset/connection_reset_by_peer === RUN TestUploadRewindsBodyOnConnectionReset/use_of_closed_network_connection --- PASS: TestUploadRewindsBodyOnConnectionReset (0.03s) --- PASS: TestUploadRewindsBodyOnConnectionReset/connection_reset_by_peer (0.03s) --- PASS: TestUploadRewindsBodyOnConnectionReset/use_of_closed_network_connection (0.00s) === RUN TestReplicationUploadPreservesUncompressedNeedle === RUN TestReplicationUploadPreservesUncompressedNeedle/unknown_binary_sampling === RUN TestReplicationUploadPreservesUncompressedNeedle/receiver_MIME_detection --- PASS: TestReplicationUploadPreservesUncompressedNeedle (0.02s) --- PASS: TestReplicationUploadPreservesUncompressedNeedle/unknown_binary_sampling (0.01s) --- PASS: TestReplicationUploadPreservesUncompressedNeedle/receiver_MIME_detection (0.01s) === RUN TestUploadToDeadReplicaRetriesThreeTimes W0824 22:18:47.196007 upload_content.go:334 uploading 0 to http://dead-replica:8080/3,01: upload test.bin 5 bytes to http://dead-replica:8080/3,01: dial tcp dead-replica:8080: i/o timeout W0824 22:18:47.785770 upload_content.go:334 uploading 1 to http://dead-replica:8080/3,01: upload test.bin 5 bytes to http://dead-replica:8080/3,01: dial tcp dead-replica:8080: i/o timeout W0824 22:18:48.616140 upload_content.go:334 uploading 2 to http://dead-replica:8080/3,01: upload test.bin 5 bytes to http://dead-replica:8080/3,01: dial tcp dead-replica:8080: i/o timeout upload_content_test.go:410: dead replica stalled the caller for 1.526935985s across 3 attempts --- PASS: TestUploadToDeadReplicaRetriesThreeTimes (1.53s) === RUN TestUploadToDeadReplicaSingleAttempt W0824 22:18:48.725118 upload_content.go:334 uploading 0 to http://dead-replica:8080/3,01: upload test.bin 5 bytes to http://dead-replica:8080/3,01: dial tcp dead-replica:8080: i/o timeout --- PASS: TestUploadToDeadReplicaSingleAttempt (0.11s) === RUN TestUploadRetryStopsOnContextCancel --- PASS: TestUploadRetryStopsOnContextCancel (0.00s) PASS ok github.com/seaweedfs/seaweedfs/weed/operation 39.751s === RUN TestMoveEcShardsSequence --- PASS: TestMoveEcShardsSequence (0.10s) === RUN TestMoveEcShardsVerifyFailureKeepsSource --- PASS: TestMoveEcShardsVerifyFailureKeepsSource (0.00s) === RUN TestMoveEcShardsZeroSizedDestinationKeepsSource --- PASS: TestMoveEcShardsZeroSizedDestinationKeepsSource (0.00s) === RUN TestMoveEcShardsRejectsSameServer --- PASS: TestMoveEcShardsRejectsSameServer (0.00s) === RUN TestRemoveEcShards --- PASS: TestRemoveEcShards (0.00s) === RUN TestCopyAndMountEcShardsSameAddressMountsOnly --- PASS: TestCopyAndMountEcShardsSameAddressMountsOnly (0.00s) === RUN TestSameServer --- PASS: TestSameServer (0.00s) === RUN TestCheckDialable --- PASS: TestCheckDialable (0.00s) === RUN TestEmbeddedSourceAddressValidated --- PASS: TestEmbeddedSourceAddressValidated (0.00s) === RUN TestNewMoverRejectsMalformedAddressWithoutDialing --- PASS: TestNewMoverRejectsMalformedAddressWithoutDialing (0.00s) === RUN TestLiveMoveVolumeSequence --- PASS: TestLiveMoveVolumeSequence (0.00s) === RUN TestLiveMoveVolumeVerifyMismatchKeepsSource --- PASS: TestLiveMoveVolumeVerifyMismatchKeepsSource (0.00s) === RUN TestLiveMoveVolumeAbortKeepsSourceReadonlyWhenTargetCleanupFails W0824 22:18:22.971511 volume_move.go:124 volume 7 is left readonly on src:8080: failed to delete the incomplete copy on dst:8080: target unreachable; delete that copy, then re-run the move --- PASS: TestLiveMoveVolumeAbortKeepsSourceReadonlyWhenTargetCleanupFails (0.01s) === RUN TestLiveMoveVolumeAlreadyReadonlyStaysReadonly --- PASS: TestLiveMoveVolumeAlreadyReadonlyStaysReadonly (0.07s) === RUN TestLiveMoveVolumePreFrozenAbortCleansTarget --- PASS: TestLiveMoveVolumePreFrozenAbortCleansTarget (0.00s) === RUN TestLiveMoveVolumeCopyErrorCleansMountedTarget --- PASS: TestLiveMoveVolumeCopyErrorCleansMountedTarget (0.00s) === RUN TestLiveMoveVolumeCopyErrorUnknownTargetKeepsSourceReadonly W0824 22:18:23.052935 volume_move.go:111 volume 7 is left readonly on src:8080: a copy may exist on dst:8080 but its origin cannot be determined; delete one side explicitly, then re-run the move --- PASS: TestLiveMoveVolumeCopyErrorUnknownTargetKeepsSourceReadonly (0.00s) === RUN TestLiveMoveVolumeCopyErrorLeavesPreexistingTarget W0824 22:18:23.055493 volume_move.go:111 volume 7 is left readonly on src:8080: a copy may exist on dst:8080 but its origin cannot be determined; delete one side explicitly, then re-run the move --- PASS: TestLiveMoveVolumeCopyErrorLeavesPreexistingTarget (0.00s) === RUN TestLiveMoveVolumeTailErrorAborts --- PASS: TestLiveMoveVolumeTailErrorAborts (0.00s) === RUN TestLiveMoveVolumeTailErrorToleratedForReadonlySource W0824 22:18:23.065663 volume_move.go:187 tail volume 7 from src:8080 to dst:8080: tail failed --- PASS: TestLiveMoveVolumeTailErrorToleratedForReadonlySource (0.01s) === RUN TestLiveMoveVolumeTailFailureAbortsWhenSourceStillChanging W0824 22:18:23.073157 volume_move.go:187 tail volume 7 from src:8080 to dst:8080: tail failed --- PASS: TestLiveMoveVolumeTailFailureAbortsWhenSourceStillChanging (0.01s) === RUN TestLiveMoveVolumeSourceDeleteFailureKeepsBothReadonly W0824 22:18:23.088346 volume_move.go:83 volume 7 is left readonly on src:8080: the source delete failed with the verified copy on dst:8080 mounted --- PASS: TestLiveMoveVolumeSourceDeleteFailureKeepsBothReadonly (0.00s) === RUN TestLiveMoveVolumeRefusesRecopyOverCompleteTarget --- PASS: TestLiveMoveVolumeRefusesRecopyOverCompleteTarget (0.00s) === RUN TestLiveMoveVolumeReadonlySourceUnknownTargetRefused --- PASS: TestLiveMoveVolumeReadonlySourceUnknownTargetRefused (0.00s) === RUN TestLiveMoveVolumePreFrozenSourceMissingRustTargetProceeds --- PASS: TestLiveMoveVolumePreFrozenSourceMissingRustTargetProceeds (0.00s) === RUN TestLiveMoveVolumeReadonlySourceExistingTargetRefused --- PASS: TestLiveMoveVolumeReadonlySourceExistingTargetRefused (0.00s) === RUN TestCopyVolumeLeavesReadonlySourceUntouched --- PASS: TestCopyVolumeLeavesReadonlySourceUntouched (0.00s) === RUN TestLiveMoveVolumeTargetAheadCommits --- PASS: TestLiveMoveVolumeTargetAheadCommits (0.00s) === RUN TestLiveMoveVolumeRejectsSameServer --- PASS: TestLiveMoveVolumeRejectsSameServer (0.00s) === RUN TestLiveMoveVolumeReadonlyMarkFailureRestores --- PASS: TestLiveMoveVolumeReadonlyMarkFailureRestores (0.00s) === RUN TestReplicateVolumePropagatesThroughput --- PASS: TestReplicateVolumePropagatesThroughput (0.00s) === RUN TestCopyVolumeRestoreWritable --- PASS: TestCopyVolumeRestoreWritable (0.00s) PASS ok github.com/seaweedfs/seaweedfs/weed/operation/volume_move 14.132s === RUN TestReadLogFileRefsMergeOrder --- PASS: TestReadLogFileRefsMergeOrder (0.38s) === RUN TestReadLogFileRefsPathFilter filer_pb_direct_read_test.go:209: Total events: 200, matching /data/filer00/: 100 --- PASS: TestReadLogFileRefsPathFilter (0.03s) === RUN TestDirectReadVsServerSideThroughput === RUN TestDirectReadVsServerSideThroughput/server_side_sequential filer_pb_direct_read_test.go:255: server-side: 6300 events 9.349s 674 events/sec (21 files sequential + 20µs send/event) === RUN TestDirectReadVsServerSideThroughput/client_direct_read_parallel_streaming filer_pb_direct_read_test.go:276: direct-read: 6300 events 181ms 34751 events/sec (3 filers parallel + streaming, no gRPC) === NAME TestDirectReadVsServerSideThroughput filer_pb_direct_read_test.go:281: Speedup: 51.6x (parallel + streaming + no gRPC vs server-side sequential) --- PASS: TestDirectReadVsServerSideThroughput (9.75s) --- PASS: TestDirectReadVsServerSideThroughput/server_side_sequential (9.35s) --- PASS: TestDirectReadVsServerSideThroughput/client_direct_read_parallel_streaming (0.18s) === RUN TestReadLogFileRefsMultiFilerGenuineErrorAborts --- PASS: TestReadLogFileRefsMultiFilerGenuineErrorAborts (0.02s) === RUN TestReadLogFileRefsMultiFilerNotFoundSkips I0824 22:18:33.932962 filer_pb_direct_read.go:228 skip log file filer=filer01 ts=1787606313931345868 after 0 entries: create reader: volume not found: filer01:1787606313931345868 --- PASS: TestReadLogFileRefsMultiFilerNotFoundSkips (0.05s) === RUN TestReadLogFileRefsSingleFilerOrder --- PASS: TestReadLogFileRefsSingleFilerOrder (0.01s) === RUN TestReadLogFileRefsSingleFilerProcessErrorStops --- PASS: TestReadLogFileRefsSingleFilerProcessErrorStops (0.02s) === RUN TestReadLogFileRefsCorruptSizePrefix --- PASS: TestReadLogFileRefsCorruptSizePrefix (0.00s) === RUN TestReadLogFileRefsAbortDoesNotJoinWedgedReader --- PASS: TestReadLogFileRefsAbortDoesNotJoinWedgedReader (0.00s) === RUN TestReadLogFileRefsSingleFilerNotFoundSkips I0824 22:18:34.026442 filer_pb_direct_read.go:228 skip log file filer=filer00 ts=1787606374023223616 after 0 entries: create reader: volume not found: filer00:1787606374023223616 --- PASS: TestReadLogFileRefsSingleFilerNotFoundSkips (0.00s) === RUN TestProcessOneLogEntrySkipsInvalidUTF8 E0824 22:18:34.027503 filer_pb_direct_read.go:242 unmarshal log entry: string field contains invalid UTF-8 --- PASS: TestProcessOneLogEntrySkipsInvalidUTF8 (0.00s) === RUN TestFilerSyncOffsetStaysFreshOnFilteredMarker filer_pb_tail_test.go:107: gauge timeline: [{src:offset ts:1000000000} {src:heartbeat ts:1787609914032312853} {src:heartbeat ts:1787609915032312853}] --- PASS: TestFilerSyncOffsetStaysFreshOnFilteredMarker (0.02s) === RUN TestFilerSyncBatchedFreshnessSignalDoesNotCrash --- PASS: TestFilerSyncBatchedFreshnessSignalDoesNotCrash (0.00s) === RUN TestShouldInvalidateConnection_MarshalErrorIsPerRequest --- PASS: TestShouldInvalidateConnection_MarshalErrorIsPerRequest (0.00s) === RUN TestShouldInvalidateConnection_CallerContextExpiryIsPerRequest --- PASS: TestShouldInvalidateConnection_CallerContextExpiryIsPerRequest (0.00s) === RUN TestShouldInvalidateConnection_StaleChannelStillInvalidates --- PASS: TestShouldInvalidateConnection_StaleChannelStillInvalidates (0.00s) === RUN TestShouldInvalidateConnection_ResourceExhaustedIsPerRequest --- PASS: TestShouldInvalidateConnection_ResourceExhaustedIsPerRequest (0.00s) === RUN TestShouldInvalidateConnection_GenuineInternalStillInvalidates --- PASS: TestShouldInvalidateConnection_GenuineInternalStillInvalidates (0.00s) === RUN TestShouldInvalidateConnection_TransportErrorsStillInvalidate --- PASS: TestShouldInvalidateConnection_TransportErrorsStillInvalidate (0.00s) === RUN TestIsClientSideMarshalError_RequiresGrpcStatus --- PASS: TestIsClientSideMarshalError_RequiresGrpcStatus (0.00s) === RUN TestResolveLocalGrpcSocket_RemotePortCollision === RUN TestResolveLocalGrpcSocket_RemotePortCollision/local_advertised_host_routes_to_socket === RUN TestResolveLocalGrpcSocket_RemotePortCollision/loopback_v4_routes_to_socket === RUN TestResolveLocalGrpcSocket_RemotePortCollision/localhost_routes_to_socket === RUN TestResolveLocalGrpcSocket_RemotePortCollision/loopback_v6_routes_to_socket === RUN TestResolveLocalGrpcSocket_RemotePortCollision/empty_host_(bare_port)_routes_to_socket === RUN TestResolveLocalGrpcSocket_RemotePortCollision/remote_host_with_same_port_stays_on_TCP === RUN TestResolveLocalGrpcSocket_RemotePortCollision/unrelated_host_with_same_port_stays_on_TCP === RUN TestResolveLocalGrpcSocket_RemotePortCollision/unregistered_port_stays_on_TCP === RUN TestResolveLocalGrpcSocket_RemotePortCollision/malformed_address_stays_on_TCP --- PASS: TestResolveLocalGrpcSocket_RemotePortCollision (0.00s) --- PASS: TestResolveLocalGrpcSocket_RemotePortCollision/local_advertised_host_routes_to_socket (0.00s) --- PASS: TestResolveLocalGrpcSocket_RemotePortCollision/loopback_v4_routes_to_socket (0.00s) --- PASS: TestResolveLocalGrpcSocket_RemotePortCollision/localhost_routes_to_socket (0.00s) --- PASS: TestResolveLocalGrpcSocket_RemotePortCollision/loopback_v6_routes_to_socket (0.00s) --- PASS: TestResolveLocalGrpcSocket_RemotePortCollision/empty_host_(bare_port)_routes_to_socket (0.00s) --- PASS: TestResolveLocalGrpcSocket_RemotePortCollision/remote_host_with_same_port_stays_on_TCP (0.00s) --- PASS: TestResolveLocalGrpcSocket_RemotePortCollision/unrelated_host_with_same_port_stays_on_TCP (0.00s) --- PASS: TestResolveLocalGrpcSocket_RemotePortCollision/unregistered_port_stays_on_TCP (0.00s) --- PASS: TestResolveLocalGrpcSocket_RemotePortCollision/malformed_address_stays_on_TCP (0.00s) === RUN TestJsonpMarshalUnmarshal marshalled: { "backendType": "aws", "backendId": "", "key": "", "offset": "0", "fileSize": "12", "modifiedTime": "0", "extension": "" } unmarshalled: backend_type:"aws" backend_id:"temp" file_size:12 --- PASS: TestJsonpMarshalUnmarshal (0.01s) === RUN TestServerAddresses_ToAddressMapOrSrv_shouldRemovePrefix --- PASS: TestServerAddresses_ToAddressMapOrSrv_shouldRemovePrefix (0.00s) === RUN TestServerAddresses_ToAddressMapOrSrv_shouldHandleIPPortList --- PASS: TestServerAddresses_ToAddressMapOrSrv_shouldHandleIPPortList (0.00s) === RUN TestServerAddress_ToHost === RUN TestServerAddress_ToHost/hostname_with_port === RUN TestServerAddress_ToHost/IPv4_with_port === RUN TestServerAddress_ToHost/IPv6_with_port === RUN TestServerAddress_ToHost/hostname_without_port === RUN TestServerAddress_ToHost/hostname_with_port.grpcPort === RUN TestServerAddress_ToHost/IPv4_with_port.grpcPort === RUN TestServerAddress_ToHost/IPv6_with_port.grpcPort === RUN TestServerAddress_ToHost/bracketed_IPv6_without_port --- PASS: TestServerAddress_ToHost (0.00s) --- PASS: TestServerAddress_ToHost/hostname_with_port (0.00s) --- PASS: TestServerAddress_ToHost/IPv4_with_port (0.00s) --- PASS: TestServerAddress_ToHost/IPv6_with_port (0.00s) --- PASS: TestServerAddress_ToHost/hostname_without_port (0.00s) --- PASS: TestServerAddress_ToHost/hostname_with_port.grpcPort (0.00s) --- PASS: TestServerAddress_ToHost/IPv4_with_port.grpcPort (0.00s) --- PASS: TestServerAddress_ToHost/IPv6_with_port.grpcPort (0.00s) --- PASS: TestServerAddress_ToHost/bracketed_IPv6_without_port (0.00s) === RUN TestIPv6ServerAddressFormatting === RUN TestIPv6ServerAddressFormatting/unbracketed_IPv6 === RUN TestIPv6ServerAddressFormatting/bracketed_IPv6 --- PASS: TestIPv6ServerAddressFormatting (0.00s) --- PASS: TestIPv6ServerAddressFormatting/unbracketed_IPv6 (0.00s) --- PASS: TestIPv6ServerAddressFormatting/bracketed_IPv6 (0.00s) === RUN TestReceiveVolumeListIsTheSameEitherWay === RUN TestReceiveVolumeListIsTheSameEitherWay/streaming=true === RUN TestReceiveVolumeListIsTheSameEitherWay/streaming=false --- PASS: TestReceiveVolumeListIsTheSameEitherWay (0.16s) --- PASS: TestReceiveVolumeListIsTheSameEitherWay/streaming=true (0.12s) --- PASS: TestReceiveVolumeListIsTheSameEitherWay/streaming=false (0.04s) === RUN TestCollectVolumeListRebuildsTheListing === RUN TestCollectVolumeListRebuildsTheListing/streaming=true === RUN TestCollectVolumeListRebuildsTheListing/streaming=false --- PASS: TestCollectVolumeListRebuildsTheListing (0.10s) --- PASS: TestCollectVolumeListRebuildsTheListing/streaming=true (0.03s) --- PASS: TestCollectVolumeListRebuildsTheListing/streaming=false (0.07s) === RUN TestCollectVolumeListSurvivesADiskAddedMidStream --- PASS: TestCollectVolumeListSurvivesADiskAddedMidStream (0.02s) === RUN TestReceiveVolumeListDoesNotRestartAStreamThatBegan --- PASS: TestReceiveVolumeListDoesNotRestartAStreamThatBegan (0.02s) === RUN TestReceiveVolumeListDoesNotRestartOnACallersError --- PASS: TestReceiveVolumeListDoesNotRestartOnACallersError (0.02s) PASS ok github.com/seaweedfs/seaweedfs/weed/pb 25.366s === RUN TestReadDirAllEntriesWithSnapshotCarriesSnapshotAcrossPages --- PASS: TestReadDirAllEntriesWithSnapshotCarriesSnapshotAcrossPages (0.29s) === RUN TestReadDirAllEntriesWithSnapshotEmptyDirectory --- PASS: TestReadDirAllEntriesWithSnapshotEmptyDirectory (0.00s) === RUN TestScanMetadataEventSkeletonMatchesFullDecode --- PASS: TestScanMetadataEventSkeletonMatchesFullDecode (0.12s) === RUN TestScanMetadataEventSkeletonFallsBack --- PASS: TestScanMetadataEventSkeletonFallsBack (0.00s) === RUN TestScanMetadataEventSkeletonEmptyPayload --- PASS: TestScanMetadataEventSkeletonEmptyPayload (0.00s) === RUN TestIsDirectoryKeyObject === RUN TestIsDirectoryKeyObject/plain_directory === RUN TestIsDirectoryKeyObject/directory_marker_with_mime === RUN TestIsDirectoryKeyObject/directory_promoted_from_file_keeps_chunks === RUN TestIsDirectoryKeyObject/directory_promoted_from_small_file_keeps_content === RUN TestIsDirectoryKeyObject/directory_promoted_from_remote-tiered_file === RUN TestIsDirectoryKeyObject/directory_with_chunks_and_nil_attributes === RUN TestIsDirectoryKeyObject/regular_file_with_chunks === RUN TestIsDirectoryKeyObject/remote_mount_directory_has_no_remote_size --- PASS: TestIsDirectoryKeyObject (0.00s) --- PASS: TestIsDirectoryKeyObject/plain_directory (0.00s) --- PASS: TestIsDirectoryKeyObject/directory_marker_with_mime (0.00s) --- PASS: TestIsDirectoryKeyObject/directory_promoted_from_file_keeps_chunks (0.00s) --- PASS: TestIsDirectoryKeyObject/directory_promoted_from_small_file_keeps_content (0.00s) --- PASS: TestIsDirectoryKeyObject/directory_promoted_from_remote-tiered_file (0.00s) --- PASS: TestIsDirectoryKeyObject/directory_with_chunks_and_nil_attributes (0.00s) --- PASS: TestIsDirectoryKeyObject/regular_file_with_chunks (0.00s) --- PASS: TestIsDirectoryKeyObject/remote_mount_directory_has_no_remote_size (0.00s) === RUN TestVTWireCompatibility --- PASS: TestVTWireCompatibility (0.18s) === RUN TestUTF8Validation --- PASS: TestUTF8Validation (0.00s) === RUN TestVtProtoPreservesLogTsNs --- PASS: TestVtProtoPreservesLogTsNs (0.04s) PASS ok github.com/seaweedfs/seaweedfs/weed/pb/filer_pb 16.088s ? github.com/seaweedfs/seaweedfs/weed/pb/iam_pb [no test files] === RUN TestDiskInfoSplitByPhysicalDisk_includesEmptyDiskFromPhysicalDisks --- PASS: TestDiskInfoSplitByPhysicalDisk_includesEmptyDiskFromPhysicalDisks (0.05s) === RUN TestDiskInfoSplitByPhysicalDisk_clampsNegativeFreeOnOverAllocation --- PASS: TestDiskInfoSplitByPhysicalDisk_clampsNegativeFreeOnOverAllocation (0.00s) === RUN TestDiskInfoSplitByPhysicalDisk_collapsesOnSingleDisk --- PASS: TestDiskInfoSplitByPhysicalDisk_collapsesOnSingleDisk (0.00s) === RUN TestDiskInfoSplitByPhysicalDisk_splitsByVolumeDiskId --- PASS: TestDiskInfoSplitByPhysicalDisk_splitsByVolumeDiskId (0.00s) === RUN TestDiskInfoSplitByPhysicalDisk_preservesAggregateCapacityWithRemainder --- PASS: TestDiskInfoSplitByPhysicalDisk_preservesAggregateCapacityWithRemainder (0.00s) === RUN TestDiskInfoSplitByPhysicalDisk_countsActiveAndRemoteExactly --- PASS: TestDiskInfoSplitByPhysicalDisk_countsActiveAndRemoteExactly (0.00s) === RUN TestDiskInfoSplitByPhysicalDisk_splitsByEcShardDiskId --- PASS: TestDiskInfoSplitByPhysicalDisk_splitsByEcShardDiskId (0.00s) === RUN TestDiskInfoSplitByPhysicalDisk_normalizesZeroToOuterDiskId --- PASS: TestDiskInfoSplitByPhysicalDisk_normalizesZeroToOuterDiskId (0.00s) === RUN TestDiskInfoSplitByPhysicalDisk_keepsRealDiskZeroWhenMixed --- PASS: TestDiskInfoSplitByPhysicalDisk_keepsRealDiskZeroWhenMixed (0.00s) === RUN TestDiskInfoSplitByPhysicalDisk_nilSafe --- PASS: TestDiskInfoSplitByPhysicalDisk_nilSafe (0.00s) === RUN TestDiskInfoSplitByPhysicalDisk_emptyDiskReturnsSelf --- PASS: TestDiskInfoSplitByPhysicalDisk_emptyDiskReturnsSelf (0.05s) PASS ok github.com/seaweedfs/seaweedfs/weed/pb/master_pb 9.491s ? github.com/seaweedfs/seaweedfs/weed/pb/message_fbs [no test files] ? github.com/seaweedfs/seaweedfs/weed/pb/mount_pb [no test files] ? github.com/seaweedfs/seaweedfs/weed/pb/mount_peer_pb [no test files] === RUN TestPublishRecordResponseSerialization --- PASS: TestPublishRecordResponseSerialization (0.12s) === RUN TestSubscribeRecordResponseSerialization --- PASS: TestSubscribeRecordResponseSerialization (0.01s) === RUN TestPublishRecordResponseBackwardCompatibility --- PASS: TestPublishRecordResponseBackwardCompatibility (0.00s) PASS ok github.com/seaweedfs/seaweedfs/weed/pb/mq_agent_pb 7.691s ? github.com/seaweedfs/seaweedfs/weed/pb/mq_pb [no test files] ? github.com/seaweedfs/seaweedfs/weed/pb/plugin_pb [no test files] ? github.com/seaweedfs/seaweedfs/weed/pb/remote_pb [no test files] ? github.com/seaweedfs/seaweedfs/weed/pb/s3_lifecycle_pb [no test files] ? github.com/seaweedfs/seaweedfs/weed/pb/s3_pb [no test files] === RUN TestOffsetTypeEnums === RUN TestOffsetTypeEnums/EXACT_OFFSET === RUN TestOffsetTypeEnums/RESET_TO_OFFSET --- PASS: TestOffsetTypeEnums (0.00s) --- PASS: TestOffsetTypeEnums/EXACT_OFFSET (0.00s) --- PASS: TestOffsetTypeEnums/RESET_TO_OFFSET (0.00s) === RUN TestPartitionOffsetSerialization --- PASS: TestPartitionOffsetSerialization (0.11s) === RUN TestPartitionOffsetBackwardCompatibility --- PASS: TestPartitionOffsetBackwardCompatibility (0.00s) PASS ok github.com/seaweedfs/seaweedfs/weed/pb/schema_pb 1.992s ? github.com/seaweedfs/seaweedfs/weed/pb/volume_server_pb [no test files] ? github.com/seaweedfs/seaweedfs/weed/pb/worker_pb [no test files] === RUN TestPickTargetPrefersTheEmptiestByFreeBytes --- PASS: TestPickTargetPrefersTheEmptiestByFreeBytes (0.00s) === RUN TestPickTargetFallsBackToSlotsWithoutReportedBytes --- PASS: TestPickTargetFallsBackToSlotsWithoutReportedBytes (0.00s) === RUN TestPickTargetSkipsSourceExcludedAndFull --- PASS: TestPickTargetSkipsSourceExcludedAndFull (0.04s) === RUN TestPickTargetReturnsNilWhenNothingFits --- PASS: TestPickTargetReturnsNilWhenNothingFits (0.00s) === RUN TestPickTargetHonoursAnchorDataCenter --- PASS: TestPickTargetHonoursAnchorDataCenter (0.00s) === RUN TestPickTargetPrefersTheSourceRackOverAnEmptierStranger --- PASS: TestPickTargetPrefersTheSourceRackOverAnEmptierStranger (0.00s) === RUN TestPickTargetLeavesTheRackWhenItCannotHoldTheVolume --- PASS: TestPickTargetLeavesTheRackWhenItCannotHoldTheVolume (0.00s) === RUN TestPickTargetPrefersSameDataCenterOverAnother --- PASS: TestPickTargetPrefersSameDataCenterOverAnother (0.00s) === RUN TestPickTargetFallsBackToCapacityForAnUnknownSource --- PASS: TestPickTargetFallsBackToCapacityForAnUnknownSource (0.00s) === RUN TestPickTargetSpendsCapacitySoABatchSpreads --- PASS: TestPickTargetSpendsCapacitySoABatchSpreads (0.00s) === RUN TestPickTargetStopsWhenTheSnapshotIsSpent --- PASS: TestPickTargetStopsWhenTheSnapshotIsSpent (0.00s) === RUN TestPickTargetUsesOneMetricWhenReportingIsMixed --- PASS: TestPickTargetUsesOneMetricWhenReportingIsMixed (0.00s) === RUN TestPickTargetReservesTheVolumeSize --- PASS: TestPickTargetReservesTheVolumeSize (0.00s) === RUN TestPickTargetHonoursTheCallerPredicate --- PASS: TestPickTargetHonoursTheCallerPredicate (0.00s) === RUN TestPickTargetPredicateSeesRackAndDataCenter --- PASS: TestPickTargetPredicateSeesRackAndDataCenter (0.00s) PASS ok github.com/seaweedfs/seaweedfs/weed/placement 11.824s === RUN TestAdminScriptFilerAddressKeepsGrpcPortOffConvention --- PASS: TestAdminScriptFilerAddressKeepsGrpcPortOffConvention (0.07s) === RUN TestAdminScriptShellOptionsRequireMasters --- PASS: TestAdminScriptShellOptionsRequireMasters (0.02s) === RUN TestAdminScriptDescriptorDefaults --- PASS: TestAdminScriptDescriptorDefaults (0.00s) === RUN TestAdminScriptDetectSkipsByRunInterval --- PASS: TestAdminScriptDetectSkipsByRunInterval (0.02s) === RUN TestAdminScriptDetectCreatesProposalWhenIntervalElapsed --- PASS: TestAdminScriptDetectCreatesProposalWhenIntervalElapsed (0.00s) === RUN TestBuildVolumeMetricsEmptyFilter --- PASS: TestBuildVolumeMetricsEmptyFilter (0.06s) === RUN TestBuildVolumeMetricsAllCollections --- PASS: TestBuildVolumeMetricsAllCollections (0.00s) === RUN TestBuildVolumeMetricsEachCollection --- PASS: TestBuildVolumeMetricsEachCollection (0.00s) === RUN TestBuildVolumeMetricsRegexFilter --- PASS: TestBuildVolumeMetricsRegexFilter (0.00s) === RUN TestBuildVolumeMetricsInvalidRegex --- PASS: TestBuildVolumeMetricsInvalidRegex (0.00s) === RUN TestWorkerBuildHelloUsesConfiguredConcurrency --- PASS: TestWorkerBuildHelloUsesConfiguredConcurrency (0.13s) === RUN TestWorkerBuildHelloIncludesMultipleCapabilities --- PASS: TestWorkerBuildHelloIncludesMultipleCapabilities (0.00s) === RUN TestWorkerCancelWorkByTargetID --- PASS: TestWorkerCancelWorkByTargetID (0.00s) === RUN TestWorkerHandleCancelRequestAck --- PASS: TestWorkerHandleCancelRequestAck (0.00s) === RUN TestWorkerSchemaRequestRequiresJobTypeWhenMultipleHandlers --- PASS: TestWorkerSchemaRequestRequiresJobTypeWhenMultipleHandlers (0.00s) === RUN TestWorkerHandleDetectionQueuesWhenAtCapacity --- PASS: TestWorkerHandleDetectionQueuesWhenAtCapacity (0.11s) === RUN TestWorkerHeartbeatReflectsActiveDetectionLoad --- PASS: TestWorkerHeartbeatReflectsActiveDetectionLoad (0.00s) === RUN TestWorkerHeartbeatReflectsActiveExecutionLoad --- PASS: TestWorkerHeartbeatReflectsActiveExecutionLoad (0.00s) PASS ok github.com/seaweedfs/seaweedfs/weed/plugin/worker 18.366s ? github.com/seaweedfs/seaweedfs/weed/plugin/worker/handlers [no test files] === RUN TestAliasTimestampIntegration === RUN TestAliasTimestampIntegration/AliasWithLargeTimestamps === RUN TestAliasTimestampIntegration/AliasWithLargeTimestamps/Timestamp_1 === RUN TestAliasTimestampIntegration/AliasWithLargeTimestamps/Timestamp_2 === RUN TestAliasTimestampIntegration/AliasWithLargeTimestamps/Timestamp_3 === RUN TestAliasTimestampIntegration/AliasWithTimestampRangeQueries === RUN TestAliasTimestampIntegration/AliasWithTimestampPrecisionEdgeCases === RUN TestAliasTimestampIntegration/MultipleAliasesWithTimestamps === RUN TestAliasTimestampIntegration/CompatibilityWithExistingTimestampFixes === RUN TestAliasTimestampIntegration/CompatibilityWithExistingTimestampFixes/ts_=_1756947416566456262 === RUN TestAliasTimestampIntegration/CompatibilityWithExistingTimestampFixes/ts_=_1756947416566456263 === RUN TestAliasTimestampIntegration/CompatibilityWithExistingTimestampFixes/ts_>_1756947416566456261 === RUN TestAliasTimestampIntegration/CompatibilityWithExistingTimestampFixes/ts_>_1756947416566456262 === RUN TestAliasTimestampIntegration/CompatibilityWithExistingTimestampFixes/ts_>=_1756947416566456262 === RUN TestAliasTimestampIntegration/CompatibilityWithExistingTimestampFixes/ts_>=_1756947416566456263 === RUN TestAliasTimestampIntegration/CompatibilityWithExistingTimestampFixes/ts_<_1756947416566456263 === RUN TestAliasTimestampIntegration/CompatibilityWithExistingTimestampFixes/ts_<_1756947416566456262 === RUN TestAliasTimestampIntegration/CompatibilityWithExistingTimestampFixes/ts_<=_1756947416566456262 === RUN TestAliasTimestampIntegration/CompatibilityWithExistingTimestampFixes/ts_<=_1756947416566456261 === RUN TestAliasTimestampIntegration/ProductionScenarioReproduction --- PASS: TestAliasTimestampIntegration (0.49s) --- PASS: TestAliasTimestampIntegration/AliasWithLargeTimestamps (0.36s) --- PASS: TestAliasTimestampIntegration/AliasWithLargeTimestamps/Timestamp_1 (0.31s) --- PASS: TestAliasTimestampIntegration/AliasWithLargeTimestamps/Timestamp_2 (0.00s) --- PASS: TestAliasTimestampIntegration/AliasWithLargeTimestamps/Timestamp_3 (0.00s) --- PASS: TestAliasTimestampIntegration/AliasWithTimestampRangeQueries (0.00s) --- PASS: TestAliasTimestampIntegration/AliasWithTimestampPrecisionEdgeCases (0.00s) --- PASS: TestAliasTimestampIntegration/MultipleAliasesWithTimestamps (0.00s) --- PASS: TestAliasTimestampIntegration/CompatibilityWithExistingTimestampFixes (0.02s) --- PASS: TestAliasTimestampIntegration/CompatibilityWithExistingTimestampFixes/ts_=_1756947416566456262 (0.00s) --- PASS: TestAliasTimestampIntegration/CompatibilityWithExistingTimestampFixes/ts_=_1756947416566456263 (0.00s) --- PASS: TestAliasTimestampIntegration/CompatibilityWithExistingTimestampFixes/ts_>_1756947416566456261 (0.00s) --- PASS: TestAliasTimestampIntegration/CompatibilityWithExistingTimestampFixes/ts_>_1756947416566456262 (0.00s) --- PASS: TestAliasTimestampIntegration/CompatibilityWithExistingTimestampFixes/ts_>=_1756947416566456262 (0.00s) --- PASS: TestAliasTimestampIntegration/CompatibilityWithExistingTimestampFixes/ts_>=_1756947416566456263 (0.00s) --- PASS: TestAliasTimestampIntegration/CompatibilityWithExistingTimestampFixes/ts_<_1756947416566456263 (0.00s) --- PASS: TestAliasTimestampIntegration/CompatibilityWithExistingTimestampFixes/ts_<_1756947416566456262 (0.00s) --- PASS: TestAliasTimestampIntegration/CompatibilityWithExistingTimestampFixes/ts_<=_1756947416566456262 (0.00s) --- PASS: TestAliasTimestampIntegration/CompatibilityWithExistingTimestampFixes/ts_<=_1756947416566456261 (0.00s) --- PASS: TestAliasTimestampIntegration/ProductionScenarioReproduction (0.00s) === RUN TestArithmeticOperations === RUN TestArithmeticOperations/Add_two_integers === RUN TestArithmeticOperations/Add_integer_and_float === RUN TestArithmeticOperations/Subtract_two_integers === RUN TestArithmeticOperations/Multiply_two_integers === RUN TestArithmeticOperations/Multiply_with_float === RUN TestArithmeticOperations/Divide_two_integers === RUN TestArithmeticOperations/Division_by_zero === RUN TestArithmeticOperations/Modulo_operation === RUN TestArithmeticOperations/Modulo_by_zero === RUN TestArithmeticOperations/Add_string_number_to_integer === RUN TestArithmeticOperations/Invalid_string_conversion === RUN TestArithmeticOperations/Add_boolean_to_integer === RUN TestArithmeticOperations/Add_with_null_left_operand === RUN TestArithmeticOperations/Add_with_null_right_operand --- PASS: TestArithmeticOperations (0.00s) --- PASS: TestArithmeticOperations/Add_two_integers (0.00s) --- PASS: TestArithmeticOperations/Add_integer_and_float (0.00s) --- PASS: TestArithmeticOperations/Subtract_two_integers (0.00s) --- PASS: TestArithmeticOperations/Multiply_two_integers (0.00s) --- PASS: TestArithmeticOperations/Multiply_with_float (0.00s) --- PASS: TestArithmeticOperations/Divide_two_integers (0.00s) --- PASS: TestArithmeticOperations/Division_by_zero (0.00s) --- PASS: TestArithmeticOperations/Modulo_operation (0.00s) --- PASS: TestArithmeticOperations/Modulo_by_zero (0.00s) --- PASS: TestArithmeticOperations/Add_string_number_to_integer (0.00s) --- PASS: TestArithmeticOperations/Invalid_string_conversion (0.00s) --- PASS: TestArithmeticOperations/Add_boolean_to_integer (0.00s) --- PASS: TestArithmeticOperations/Add_with_null_left_operand (0.00s) --- PASS: TestArithmeticOperations/Add_with_null_right_operand (0.00s) === RUN TestIndividualArithmeticFunctions --- PASS: TestIndividualArithmeticFunctions (0.00s) === RUN TestMathematicalFunctions === RUN TestMathematicalFunctions/ROUND_function_tests === RUN TestMathematicalFunctions/ROUND_function_tests/Round_float_to_integer === RUN TestMathematicalFunctions/ROUND_function_tests/Round_integer_stays_integer === RUN TestMathematicalFunctions/ROUND_function_tests/Round_with_precision_2 === RUN TestMathematicalFunctions/ROUND_function_tests/Round_negative_number === RUN TestMathematicalFunctions/ROUND_function_tests/Round_null_value === RUN TestMathematicalFunctions/CEIL_function_tests === RUN TestMathematicalFunctions/CEIL_function_tests/Ceil_positive_decimal === RUN TestMathematicalFunctions/CEIL_function_tests/Ceil_negative_decimal === RUN TestMathematicalFunctions/CEIL_function_tests/Ceil_integer === RUN TestMathematicalFunctions/CEIL_function_tests/Ceil_null_value === RUN TestMathematicalFunctions/FLOOR_function_tests === RUN TestMathematicalFunctions/FLOOR_function_tests/Floor_positive_decimal === RUN TestMathematicalFunctions/FLOOR_function_tests/Floor_negative_decimal === RUN TestMathematicalFunctions/FLOOR_function_tests/Floor_integer === RUN TestMathematicalFunctions/FLOOR_function_tests/Floor_null_value === RUN TestMathematicalFunctions/ABS_function_tests === RUN TestMathematicalFunctions/ABS_function_tests/Abs_positive_integer === RUN TestMathematicalFunctions/ABS_function_tests/Abs_negative_integer === RUN TestMathematicalFunctions/ABS_function_tests/Abs_positive_double === RUN TestMathematicalFunctions/ABS_function_tests/Abs_negative_double === RUN TestMathematicalFunctions/ABS_function_tests/Abs_positive_float === RUN TestMathematicalFunctions/ABS_function_tests/Abs_negative_float === RUN TestMathematicalFunctions/ABS_function_tests/Abs_zero === RUN TestMathematicalFunctions/ABS_function_tests/Abs_null_value --- PASS: TestMathematicalFunctions (0.02s) --- PASS: TestMathematicalFunctions/ROUND_function_tests (0.01s) --- PASS: TestMathematicalFunctions/ROUND_function_tests/Round_float_to_integer (0.00s) --- PASS: TestMathematicalFunctions/ROUND_function_tests/Round_integer_stays_integer (0.00s) --- PASS: TestMathematicalFunctions/ROUND_function_tests/Round_with_precision_2 (0.00s) --- PASS: TestMathematicalFunctions/ROUND_function_tests/Round_negative_number (0.00s) --- PASS: TestMathematicalFunctions/ROUND_function_tests/Round_null_value (0.00s) --- PASS: TestMathematicalFunctions/CEIL_function_tests (0.00s) --- PASS: TestMathematicalFunctions/CEIL_function_tests/Ceil_positive_decimal (0.00s) --- PASS: TestMathematicalFunctions/CEIL_function_tests/Ceil_negative_decimal (0.00s) --- PASS: TestMathematicalFunctions/CEIL_function_tests/Ceil_integer (0.00s) --- PASS: TestMathematicalFunctions/CEIL_function_tests/Ceil_null_value (0.00s) --- PASS: TestMathematicalFunctions/FLOOR_function_tests (0.00s) --- PASS: TestMathematicalFunctions/FLOOR_function_tests/Floor_positive_decimal (0.00s) --- PASS: TestMathematicalFunctions/FLOOR_function_tests/Floor_negative_decimal (0.00s) --- PASS: TestMathematicalFunctions/FLOOR_function_tests/Floor_integer (0.00s) --- PASS: TestMathematicalFunctions/FLOOR_function_tests/Floor_null_value (0.00s) --- PASS: TestMathematicalFunctions/ABS_function_tests (0.00s) --- PASS: TestMathematicalFunctions/ABS_function_tests/Abs_positive_integer (0.00s) --- PASS: TestMathematicalFunctions/ABS_function_tests/Abs_negative_integer (0.00s) --- PASS: TestMathematicalFunctions/ABS_function_tests/Abs_positive_double (0.00s) --- PASS: TestMathematicalFunctions/ABS_function_tests/Abs_negative_double (0.00s) --- PASS: TestMathematicalFunctions/ABS_function_tests/Abs_positive_float (0.00s) --- PASS: TestMathematicalFunctions/ABS_function_tests/Abs_negative_float (0.00s) --- PASS: TestMathematicalFunctions/ABS_function_tests/Abs_zero (0.00s) --- PASS: TestMathematicalFunctions/ABS_function_tests/Abs_null_value (0.00s) === RUN TestSQLEngine_ArithmeticOnlyQueryExecution === RUN TestSQLEngine_ArithmeticOnlyQueryExecution/Basic_arithmetic_only_query arithmetic_only_execution_test.go:75: SUCCESS: SELECT id+user_id, id*2 FROM user_events LIMIT 3 returned 3 rows with calculated values === RUN TestSQLEngine_ArithmeticOnlyQueryExecution/With_LIMIT_and_OFFSET_-_original_user_issue arithmetic_only_execution_test.go:75: SUCCESS: SELECT id+user_id, id*2 FROM user_events LIMIT 2 OFFSET 1 returned 2 rows with calculated values === RUN TestSQLEngine_ArithmeticOnlyQueryExecution/Multiple_arithmetic_expressions arithmetic_only_execution_test.go:75: SUCCESS: SELECT user_id+100, id-1000 FROM user_events LIMIT 1 returned 1 rows with calculated values --- PASS: TestSQLEngine_ArithmeticOnlyQueryExecution (0.01s) --- PASS: TestSQLEngine_ArithmeticOnlyQueryExecution/Basic_arithmetic_only_query (0.00s) --- PASS: TestSQLEngine_ArithmeticOnlyQueryExecution/With_LIMIT_and_OFFSET_-_original_user_issue (0.00s) --- PASS: TestSQLEngine_ArithmeticOnlyQueryExecution/Multiple_arithmetic_expressions (0.00s) === RUN TestSQLEngine_ArithmeticOnlyQueryBugReproduction arithmetic_only_execution_test.go:140: SUCCESS: Arithmetic-only query with OFFSET works correctly! arithmetic_only_execution_test.go:141: Query: SELECT id+user_id, id*amount, id*2 FROM user_events LIMIT 10 OFFSET 5 arithmetic_only_execution_test.go:142: Returned 5 rows with correct calculations --- PASS: TestSQLEngine_ArithmeticOnlyQueryBugReproduction (0.00s) === RUN TestArithmeticExpressionParsing === RUN TestArithmeticExpressionParsing/simple_addition === RUN TestArithmeticExpressionParsing/simple_subtraction === RUN TestArithmeticExpressionParsing/multiplication_with_spaces === RUN TestArithmeticExpressionParsing/string_concatenation === RUN TestArithmeticExpressionParsing/string_concatenation_with_spaces === RUN TestArithmeticExpressionParsing/not_arithmetic --- PASS: TestArithmeticExpressionParsing (0.03s) --- PASS: TestArithmeticExpressionParsing/simple_addition (0.02s) --- PASS: TestArithmeticExpressionParsing/simple_subtraction (0.00s) --- PASS: TestArithmeticExpressionParsing/multiplication_with_spaces (0.00s) --- PASS: TestArithmeticExpressionParsing/string_concatenation (0.00s) --- PASS: TestArithmeticExpressionParsing/string_concatenation_with_spaces (0.00s) --- PASS: TestArithmeticExpressionParsing/not_arithmetic (0.00s) === RUN TestArithmeticExpressionEvaluation === RUN TestArithmeticExpressionEvaluation/integer_addition === RUN TestArithmeticExpressionEvaluation/integer_subtraction === RUN TestArithmeticExpressionEvaluation/mixed_types_multiplication === RUN TestArithmeticExpressionEvaluation/string_concatenation === RUN TestArithmeticExpressionEvaluation/string_concatenation_with_spaces --- PASS: TestArithmeticExpressionEvaluation (0.01s) --- PASS: TestArithmeticExpressionEvaluation/integer_addition (0.00s) --- PASS: TestArithmeticExpressionEvaluation/integer_subtraction (0.00s) --- PASS: TestArithmeticExpressionEvaluation/mixed_types_multiplication (0.00s) --- PASS: TestArithmeticExpressionEvaluation/string_concatenation (0.00s) --- PASS: TestArithmeticExpressionEvaluation/string_concatenation_with_spaces (0.00s) === RUN TestSelectArithmeticExpression --- PASS: TestSelectArithmeticExpression (0.00s) === RUN TestArithmeticWithFunctions === RUN TestArithmeticWithFunctions/Simple_function_arithmetic arithmetic_with_functions_test.go:75: PASS Basic function call with addition: SELECT LENGTH('hello') + 10 FROM user_events LIMIT 1 → 15 === RUN TestArithmeticWithFunctions/Nested_functions_with_arithmetic arithmetic_with_functions_test.go:75: PASS Complex nested functions with arithmetic operation (user's original failing query): SELECT length(trim(' hello world ')) + 12 FROM user_events LIMIT 1 → 23 === RUN TestArithmeticWithFunctions/Function_subtraction arithmetic_with_functions_test.go:75: PASS Function call with subtraction: SELECT LENGTH('programming') - 5 FROM user_events LIMIT 1 → 6 === RUN TestArithmeticWithFunctions/Function_multiplication arithmetic_with_functions_test.go:75: PASS Function call with multiplication: SELECT LENGTH('test') * 3 FROM user_events LIMIT 1 → 12 === RUN TestArithmeticWithFunctions/Multiple_nested_functions arithmetic_with_functions_test.go:75: PASS Triple nested functions: SELECT LENGTH(UPPER(TRIM(' hello '))) FROM user_events LIMIT 1 → 5 --- PASS: TestArithmeticWithFunctions (0.02s) --- PASS: TestArithmeticWithFunctions/Simple_function_arithmetic (0.00s) --- PASS: TestArithmeticWithFunctions/Nested_functions_with_arithmetic (0.00s) --- PASS: TestArithmeticWithFunctions/Function_subtraction (0.00s) --- PASS: TestArithmeticWithFunctions/Function_multiplication (0.01s) --- PASS: TestArithmeticWithFunctions/Multiple_nested_functions (0.00s) === RUN TestConvertMQSchemaToTableInfo_NoSchema === RUN TestConvertMQSchemaToTableInfo_NoSchema/nil_schema === RUN TestConvertMQSchemaToTableInfo_NoSchema/schema_with_nil_RecordType --- PASS: TestConvertMQSchemaToTableInfo_NoSchema (0.02s) --- PASS: TestConvertMQSchemaToTableInfo_NoSchema/nil_schema (0.00s) --- PASS: TestConvertMQSchemaToTableInfo_NoSchema/schema_with_nil_RecordType (0.00s) === RUN TestCockroachDBParserSuccess === RUN TestCockroachDBParserSuccess/Basic_Function cockroach_parser_success_test.go:93: SUCCESS: Simple function call → 5 === RUN TestCockroachDBParserSuccess/Function_Arithmetic cockroach_parser_success_test.go:93: SUCCESS: Function with arithmetic operation (original user issue) → 15 === RUN TestCockroachDBParserSuccess/User_Original_Query cockroach_parser_success_test.go:93: SUCCESS: User's exact original failing query - now fixed! → 23 === RUN TestCockroachDBParserSuccess/String_Concatenation cockroach_parser_success_test.go:93: SUCCESS: Basic string concatenation → helloworld === RUN TestCockroachDBParserSuccess/Function_With_Concat cockroach_parser_success_test.go:93: SUCCESS: Function with string concatenation argument → 10 === RUN TestCockroachDBParserSuccess/Multiple_Arithmetic cockroach_parser_success_test.go:93: SUCCESS: Function with multiplication → 12 === RUN TestCockroachDBParserSuccess/Nested_Functions cockroach_parser_success_test.go:93: SUCCESS: Nested function calls → 5 === RUN TestCockroachDBParserSuccess/Column_Alias cockroach_parser_success_test.go:93: SUCCESS: Column alias functionality (AS keyword) → 4 === NAME TestCockroachDBParserSuccess cockroach_parser_success_test.go:101: CockroachDB Parser Integration: 8/8 tests passed! --- PASS: TestCockroachDBParserSuccess (0.02s) --- PASS: TestCockroachDBParserSuccess/Basic_Function (0.00s) --- PASS: TestCockroachDBParserSuccess/Function_Arithmetic (0.00s) --- PASS: TestCockroachDBParserSuccess/User_Original_Query (0.00s) --- PASS: TestCockroachDBParserSuccess/String_Concatenation (0.00s) --- PASS: TestCockroachDBParserSuccess/Function_With_Concat (0.00s) --- PASS: TestCockroachDBParserSuccess/Multiple_Arithmetic (0.00s) --- PASS: TestCockroachDBParserSuccess/Nested_Functions (0.00s) --- PASS: TestCockroachDBParserSuccess/Column_Alias (0.00s) === RUN TestCompleteSQLFixes === RUN TestCompleteSQLFixes/OriginalFailingProductionQueries === RUN TestCompleteSQLFixes/OriginalFailingProductionQueries/OriginalFailingQuery1 === RUN TestCompleteSQLFixes/OriginalFailingProductionQueries/OriginalFailingQuery2 === RUN TestCompleteSQLFixes/OriginalFailingProductionQueries/CurrentDataQuery === RUN TestCompleteSQLFixes/AllFixesWorkTogether === RUN TestCompleteSQLFixes/BackwardCompatibilityVerified === RUN TestCompleteSQLFixes/PerformanceAndStability === RUN TestCompleteSQLFixes/EdgeCasesAndErrorHandling --- PASS: TestCompleteSQLFixes (0.01s) --- PASS: TestCompleteSQLFixes/OriginalFailingProductionQueries (0.00s) --- PASS: TestCompleteSQLFixes/OriginalFailingProductionQueries/OriginalFailingQuery1 (0.00s) --- PASS: TestCompleteSQLFixes/OriginalFailingProductionQueries/OriginalFailingQuery2 (0.00s) --- PASS: TestCompleteSQLFixes/OriginalFailingProductionQueries/CurrentDataQuery (0.00s) --- PASS: TestCompleteSQLFixes/AllFixesWorkTogether (0.00s) --- PASS: TestCompleteSQLFixes/BackwardCompatibilityVerified (0.00s) --- PASS: TestCompleteSQLFixes/PerformanceAndStability (0.00s) --- PASS: TestCompleteSQLFixes/EdgeCasesAndErrorHandling (0.00s) === RUN TestSQLFixesSummary === RUN TestSQLFixesSummary/Summary complete_sql_fixes_test.go:251: ALL SQL FIXES VERIFIED: complete_sql_fixes_test.go:252: Timestamp precision for large int64 values complete_sql_fixes_test.go:253: SQL alias resolution in WHERE clauses complete_sql_fixes_test.go:254: Scan boundary fixes for equality queries complete_sql_fixes_test.go:255: Range query fixes for equal boundaries complete_sql_fixes_test.go:256: Hybrid scanner time range handling complete_sql_fixes_test.go:257: Backward compatibility maintained complete_sql_fixes_test.go:258: Production stability verified --- PASS: TestSQLFixesSummary (0.00s) --- PASS: TestSQLFixesSummary/Summary (0.00s) === RUN TestComprehensiveSQLSuite === RUN TestComprehensiveSQLSuite/Basic_Select_All comprehensive_sql_test.go:319: PASS: Success: Basic select all columns === RUN TestComprehensiveSQLSuite/Basic_Select_Column comprehensive_sql_test.go:319: PASS: Success: Basic select single column === RUN TestComprehensiveSQLSuite/Basic_Select_Multiple_Columns comprehensive_sql_test.go:319: PASS: Success: Basic select multiple columns === RUN TestComprehensiveSQLSuite/Arithmetic_Multiply_FIXED comprehensive_sql_test.go:319: PASS: Success: FIXED: Arithmetic multiplication works === RUN TestComprehensiveSQLSuite/Arithmetic_Add comprehensive_sql_test.go:319: PASS: Success: Arithmetic addition works === RUN TestComprehensiveSQLSuite/Arithmetic_Subtract comprehensive_sql_test.go:319: PASS: Success: Arithmetic subtraction works === RUN TestComprehensiveSQLSuite/Arithmetic_Divide comprehensive_sql_test.go:319: PASS: Success: Arithmetic division works === RUN TestComprehensiveSQLSuite/Arithmetic_Complex comprehensive_sql_test.go:319: PASS: Success: Complex arithmetic expression works === RUN TestComprehensiveSQLSuite/String_Concatenation comprehensive_sql_test.go:319: PASS: Success: String concatenation === RUN TestComprehensiveSQLSuite/String_Column_Concat comprehensive_sql_test.go:319: PASS: Success: Column string concatenation === RUN TestComprehensiveSQLSuite/Function_LENGTH comprehensive_sql_test.go:319: PASS: Success: LENGTH function with literal === RUN TestComprehensiveSQLSuite/Function_LENGTH_Column comprehensive_sql_test.go:319: PASS: Success: LENGTH function with column === RUN TestComprehensiveSQLSuite/Function_UPPER comprehensive_sql_test.go:319: PASS: Success: UPPER function === RUN TestComprehensiveSQLSuite/Function_Nested comprehensive_sql_test.go:319: PASS: Success: Nested functions === RUN TestComprehensiveSQLSuite/Function_Arithmetic comprehensive_sql_test.go:319: PASS: Success: Function with arithmetic === RUN TestComprehensiveSQLSuite/Function_Arithmetic_Complex comprehensive_sql_test.go:319: PASS: Success: Function with complex arithmetic === RUN TestComprehensiveSQLSuite/Table_Simple comprehensive_sql_test.go:319: PASS: Success: Simple table reference === RUN TestComprehensiveSQLSuite/Table_With_Database comprehensive_sql_test.go:319: PASS: Success: Table with database qualifier === RUN TestComprehensiveSQLSuite/Table_Quoted comprehensive_sql_test.go:319: PASS: Success: Quoted table name === RUN TestComprehensiveSQLSuite/Where_Simple comprehensive_sql_test.go:319: PASS: Success: Simple WHERE clause === RUN TestComprehensiveSQLSuite/Where_String comprehensive_sql_test.go:319: PASS: Success: WHERE clause with string === RUN TestComprehensiveSQLSuite/Limit_Only comprehensive_sql_test.go:319: PASS: Success: LIMIT clause only === RUN TestComprehensiveSQLSuite/Limit_Offset comprehensive_sql_test.go:319: PASS: Success: LIMIT with OFFSET === RUN TestComprehensiveSQLSuite/DateTime_CURRENT_DATE comprehensive_sql_test.go:319: PASS: Success: CURRENT_DATE function === RUN TestComprehensiveSQLSuite/DateTime_NOW comprehensive_sql_test.go:319: PASS: Success: NOW() function === RUN TestComprehensiveSQLSuite/DateTime_EXTRACT comprehensive_sql_test.go:319: PASS: Success: EXTRACT function === RUN TestComprehensiveSQLSuite/Empty_String comprehensive_sql_test.go:319: PASS: Success: Empty string literal === RUN TestComprehensiveSQLSuite/Multiple_Spaces comprehensive_sql_test.go:319: PASS: Success: Query with multiple spaces === RUN TestComprehensiveSQLSuite/Mixed_Case comprehensive_sql_test.go:319: PASS: Success: Mixed case SQL === RUN TestComprehensiveSQLSuite/Show_Databases comprehensive_sql_test.go:319: PASS: Success: SHOW DATABASES statement === RUN TestComprehensiveSQLSuite/Show_Tables comprehensive_sql_test.go:319: PASS: Success: SHOW TABLES statement === NAME TestComprehensiveSQLSuite comprehensive_sql_test.go:327: ================================================================================ comprehensive_sql_test.go:328: COMPREHENSIVE SQL TEST SUITE SUMMARY comprehensive_sql_test.go:329: ================================================================================ comprehensive_sql_test.go:330: Total Tests: 31 comprehensive_sql_test.go:331: Successful: 31 comprehensive_sql_test.go:332: Panics: 0 comprehensive_sql_test.go:333: Errors: 0 comprehensive_sql_test.go:334: ================================================================================ --- PASS: TestComprehensiveSQLSuite (0.07s) --- PASS: TestComprehensiveSQLSuite/Basic_Select_All (0.00s) --- PASS: TestComprehensiveSQLSuite/Basic_Select_Column (0.00s) --- PASS: TestComprehensiveSQLSuite/Basic_Select_Multiple_Columns (0.00s) --- PASS: TestComprehensiveSQLSuite/Arithmetic_Multiply_FIXED (0.00s) --- PASS: TestComprehensiveSQLSuite/Arithmetic_Add (0.00s) --- PASS: TestComprehensiveSQLSuite/Arithmetic_Subtract (0.00s) --- PASS: TestComprehensiveSQLSuite/Arithmetic_Divide (0.00s) --- PASS: TestComprehensiveSQLSuite/Arithmetic_Complex (0.00s) --- PASS: TestComprehensiveSQLSuite/String_Concatenation (0.00s) --- PASS: TestComprehensiveSQLSuite/String_Column_Concat (0.00s) --- PASS: TestComprehensiveSQLSuite/Function_LENGTH (0.00s) --- PASS: TestComprehensiveSQLSuite/Function_LENGTH_Column (0.00s) --- PASS: TestComprehensiveSQLSuite/Function_UPPER (0.00s) --- PASS: TestComprehensiveSQLSuite/Function_Nested (0.00s) --- PASS: TestComprehensiveSQLSuite/Function_Arithmetic (0.00s) --- PASS: TestComprehensiveSQLSuite/Function_Arithmetic_Complex (0.00s) --- PASS: TestComprehensiveSQLSuite/Table_Simple (0.00s) --- PASS: TestComprehensiveSQLSuite/Table_With_Database (0.00s) --- PASS: TestComprehensiveSQLSuite/Table_Quoted (0.00s) --- PASS: TestComprehensiveSQLSuite/Where_Simple (0.00s) --- PASS: TestComprehensiveSQLSuite/Where_String (0.00s) --- PASS: TestComprehensiveSQLSuite/Limit_Only (0.00s) --- PASS: TestComprehensiveSQLSuite/Limit_Offset (0.00s) --- PASS: TestComprehensiveSQLSuite/DateTime_CURRENT_DATE (0.00s) --- PASS: TestComprehensiveSQLSuite/DateTime_NOW (0.00s) --- PASS: TestComprehensiveSQLSuite/DateTime_EXTRACT (0.00s) --- PASS: TestComprehensiveSQLSuite/Empty_String (0.00s) --- PASS: TestComprehensiveSQLSuite/Multiple_Spaces (0.00s) --- PASS: TestComprehensiveSQLSuite/Mixed_Case (0.00s) --- PASS: TestComprehensiveSQLSuite/Show_Databases (0.00s) --- PASS: TestComprehensiveSQLSuite/Show_Tables (0.00s) === RUN TestDateTimeFunctions === RUN TestDateTimeFunctions/CURRENT_DATE_function_tests === RUN TestDateTimeFunctions/CURRENT_TIMESTAMP_function_tests === RUN TestDateTimeFunctions/NOW_function_tests === RUN TestDateTimeFunctions/CURRENT_TIME_function_tests --- PASS: TestDateTimeFunctions (0.00s) --- PASS: TestDateTimeFunctions/CURRENT_DATE_function_tests (0.00s) --- PASS: TestDateTimeFunctions/CURRENT_TIMESTAMP_function_tests (0.00s) --- PASS: TestDateTimeFunctions/NOW_function_tests (0.00s) --- PASS: TestDateTimeFunctions/CURRENT_TIME_function_tests (0.00s) === RUN TestExtractFunction === RUN TestExtractFunction/Extract_YEAR === RUN TestExtractFunction/Extract_MONTH === RUN TestExtractFunction/Extract_DAY === RUN TestExtractFunction/Extract_HOUR === RUN TestExtractFunction/Extract_MINUTE === RUN TestExtractFunction/Extract_SECOND === RUN TestExtractFunction/Extract_QUARTER_from_June === RUN TestExtractFunction/Extract_from_string_date === RUN TestExtractFunction/Extract_from_Unix_timestamp === RUN TestExtractFunction/Extract_from_null_value === RUN TestExtractFunction/Extract_invalid_part === RUN TestExtractFunction/Extract_from_invalid_string --- PASS: TestExtractFunction (0.01s) --- PASS: TestExtractFunction/Extract_YEAR (0.00s) --- PASS: TestExtractFunction/Extract_MONTH (0.00s) --- PASS: TestExtractFunction/Extract_DAY (0.00s) --- PASS: TestExtractFunction/Extract_HOUR (0.00s) --- PASS: TestExtractFunction/Extract_MINUTE (0.00s) --- PASS: TestExtractFunction/Extract_SECOND (0.00s) --- PASS: TestExtractFunction/Extract_QUARTER_from_June (0.00s) --- PASS: TestExtractFunction/Extract_from_string_date (0.00s) --- PASS: TestExtractFunction/Extract_from_Unix_timestamp (0.00s) --- PASS: TestExtractFunction/Extract_from_null_value (0.00s) --- PASS: TestExtractFunction/Extract_invalid_part (0.00s) --- PASS: TestExtractFunction/Extract_from_invalid_string (0.00s) === RUN TestDateTruncFunction === RUN TestDateTruncFunction/Truncate_to_second === RUN TestDateTruncFunction/Truncate_to_minute === RUN TestDateTruncFunction/Truncate_to_hour === RUN TestDateTruncFunction/Truncate_to_day === RUN TestDateTruncFunction/Truncate_to_month === RUN TestDateTruncFunction/Truncate_to_quarter === RUN TestDateTruncFunction/Truncate_to_year === RUN TestDateTruncFunction/Truncate_with_plural_precision === RUN TestDateTruncFunction/Truncate_from_string_date === RUN TestDateTruncFunction/Truncate_null_value === RUN TestDateTruncFunction/Invalid_precision --- PASS: TestDateTruncFunction (0.01s) --- PASS: TestDateTruncFunction/Truncate_to_second (0.00s) --- PASS: TestDateTruncFunction/Truncate_to_minute (0.00s) --- PASS: TestDateTruncFunction/Truncate_to_hour (0.00s) --- PASS: TestDateTruncFunction/Truncate_to_day (0.00s) --- PASS: TestDateTruncFunction/Truncate_to_month (0.00s) --- PASS: TestDateTruncFunction/Truncate_to_quarter (0.00s) --- PASS: TestDateTruncFunction/Truncate_to_year (0.00s) --- PASS: TestDateTruncFunction/Truncate_with_plural_precision (0.00s) --- PASS: TestDateTruncFunction/Truncate_from_string_date (0.00s) --- PASS: TestDateTruncFunction/Truncate_null_value (0.00s) --- PASS: TestDateTruncFunction/Invalid_precision (0.00s) === RUN TestDateTimeConstantsInSQL === RUN TestDateTimeConstantsInSQL/CURRENT_TIME_in_SQL_query datetime_functions_test.go:468: CURRENT_TIME returned valid time: 22:18:32 === RUN TestDateTimeConstantsInSQL/CURRENT_DATE_in_SQL_query datetime_functions_test.go:494: CURRENT_DATE returned: 2026-08-24 --- PASS: TestDateTimeConstantsInSQL (0.01s) --- PASS: TestDateTimeConstantsInSQL/CURRENT_TIME_in_SQL_query (0.01s) --- PASS: TestDateTimeConstantsInSQL/CURRENT_DATE_in_SQL_query (0.00s) === RUN TestFunctionArgumentCountHandling === RUN TestFunctionArgumentCountHandling/Zero-argument_function_should_fail_appropriately === RUN TestFunctionArgumentCountHandling/Single-argument_function_should_still_work === RUN TestFunctionArgumentCountHandling/Any_zero-argument_function_should_fail === RUN TestFunctionArgumentCountHandling/Wrong_argument_count_for_single-arg_function_should_fail --- PASS: TestFunctionArgumentCountHandling (0.01s) --- PASS: TestFunctionArgumentCountHandling/Zero-argument_function_should_fail_appropriately (0.00s) --- PASS: TestFunctionArgumentCountHandling/Single-argument_function_should_still_work (0.00s) --- PASS: TestFunctionArgumentCountHandling/Any_zero-argument_function_should_fail (0.00s) --- PASS: TestFunctionArgumentCountHandling/Wrong_argument_count_for_single-arg_function_should_fail (0.00s) === RUN TestExtractFunctionSQL === RUN TestExtractFunctionSQL/Extract_YEAR_from_current_date === RUN TestExtractFunctionSQL/Extract_MONTH_from_current_date === RUN TestExtractFunctionSQL/Extract_DAY_from_current_date === RUN TestExtractFunctionSQL/Extract_HOUR_from_current_timestamp === RUN TestExtractFunctionSQL/Extract_MINUTE_from_current_timestamp === RUN TestExtractFunctionSQL/Extract_QUARTER_from_current_date === RUN TestExtractFunctionSQL/Multiple_EXTRACT_functions === RUN TestExtractFunctionSQL/EXTRACT_with_invalid_date_part === RUN TestExtractFunctionSQL/EXTRACT_with_wrong_number_of_arguments === RUN TestExtractFunctionSQL/EXTRACT_with_too_many_arguments --- PASS: TestExtractFunctionSQL (0.05s) --- PASS: TestExtractFunctionSQL/Extract_YEAR_from_current_date (0.00s) --- PASS: TestExtractFunctionSQL/Extract_MONTH_from_current_date (0.00s) --- PASS: TestExtractFunctionSQL/Extract_DAY_from_current_date (0.02s) --- PASS: TestExtractFunctionSQL/Extract_HOUR_from_current_timestamp (0.00s) --- PASS: TestExtractFunctionSQL/Extract_MINUTE_from_current_timestamp (0.00s) --- PASS: TestExtractFunctionSQL/Extract_QUARTER_from_current_date (0.00s) --- PASS: TestExtractFunctionSQL/Multiple_EXTRACT_functions (0.00s) --- PASS: TestExtractFunctionSQL/EXTRACT_with_invalid_date_part (0.00s) --- PASS: TestExtractFunctionSQL/EXTRACT_with_wrong_number_of_arguments (0.00s) --- PASS: TestExtractFunctionSQL/EXTRACT_with_too_many_arguments (0.00s) === RUN TestDateTruncFunctionSQL === RUN TestDateTruncFunctionSQL/DATE_TRUNC_to_day === RUN TestDateTruncFunctionSQL/DATE_TRUNC_to_hour === RUN TestDateTruncFunctionSQL/DATE_TRUNC_to_month === RUN TestDateTruncFunctionSQL/DATE_TRUNC_with_invalid_precision === RUN TestDateTruncFunctionSQL/DATE_TRUNC_with_wrong_number_of_arguments --- PASS: TestDateTruncFunctionSQL (0.01s) --- PASS: TestDateTruncFunctionSQL/DATE_TRUNC_to_day (0.00s) --- PASS: TestDateTruncFunctionSQL/DATE_TRUNC_to_hour (0.00s) --- PASS: TestDateTruncFunctionSQL/DATE_TRUNC_to_month (0.00s) --- PASS: TestDateTruncFunctionSQL/DATE_TRUNC_with_invalid_precision (0.00s) --- PASS: TestDateTruncFunctionSQL/DATE_TRUNC_with_wrong_number_of_arguments (0.00s) === RUN TestFastPathPredicateValidation === RUN TestFastPathPredicateValidation/No_WHERE_clause fast_path_predicate_validation_test.go:142: No WHERE clause: onlyTimePredicates=true, startTimeNs=0, stopTimeNs=0 === RUN TestFastPathPredicateValidation/Time-only_predicate_(greater_than) fast_path_predicate_validation_test.go:142: Time-only predicate (greater than): onlyTimePredicates=true, startTimeNs=1640995200000000000, stopTimeNs=0 === RUN TestFastPathPredicateValidation/Time-only_predicate_(less_than) fast_path_predicate_validation_test.go:142: Time-only predicate (less than): onlyTimePredicates=true, startTimeNs=0, stopTimeNs=1640995200000000000 === RUN TestFastPathPredicateValidation/Time-only_predicate_(range_with_AND) fast_path_predicate_validation_test.go:142: Time-only predicate (range with AND): onlyTimePredicates=true, startTimeNs=1640995200000000000, stopTimeNs=1641081600000000000 === RUN TestFastPathPredicateValidation/Mixed_predicate_(time_+_non-time) fast_path_predicate_validation_test.go:142: Mixed predicate (time + non-time): onlyTimePredicates=false, startTimeNs=1640995200000000000, stopTimeNs=0 === RUN TestFastPathPredicateValidation/Non-time_predicate_only fast_path_predicate_validation_test.go:142: Non-time predicate only: onlyTimePredicates=false, startTimeNs=0, stopTimeNs=0 === RUN TestFastPathPredicateValidation/Multiple_non-time_predicates fast_path_predicate_validation_test.go:142: Multiple non-time predicates: onlyTimePredicates=false, startTimeNs=0, stopTimeNs=0 === RUN TestFastPathPredicateValidation/OR_with_time_predicate_(unsafe) fast_path_predicate_validation_test.go:142: OR with time predicate (unsafe): onlyTimePredicates=false, startTimeNs=0, stopTimeNs=0 === RUN TestFastPathPredicateValidation/OR_with_only_time_predicates_(still_unsafe) fast_path_predicate_validation_test.go:142: OR with only time predicates (still unsafe): onlyTimePredicates=false, startTimeNs=0, stopTimeNs=0 === RUN TestFastPathPredicateValidation/String_column_comparison fast_path_predicate_validation_test.go:142: String column comparison: onlyTimePredicates=false, startTimeNs=0, stopTimeNs=0 === RUN TestFastPathPredicateValidation/Numeric_column_comparison fast_path_predicate_validation_test.go:142: Numeric column comparison: onlyTimePredicates=false, startTimeNs=0, stopTimeNs=0 === RUN TestFastPathPredicateValidation/Internal_timestamp_column fast_path_predicate_validation_test.go:142: Internal timestamp column: onlyTimePredicates=true, startTimeNs=1640995200000000000, stopTimeNs=0 --- PASS: TestFastPathPredicateValidation (0.03s) --- PASS: TestFastPathPredicateValidation/No_WHERE_clause (0.00s) --- PASS: TestFastPathPredicateValidation/Time-only_predicate_(greater_than) (0.00s) --- PASS: TestFastPathPredicateValidation/Time-only_predicate_(less_than) (0.00s) --- PASS: TestFastPathPredicateValidation/Time-only_predicate_(range_with_AND) (0.00s) --- PASS: TestFastPathPredicateValidation/Mixed_predicate_(time_+_non-time) (0.00s) --- PASS: TestFastPathPredicateValidation/Non-time_predicate_only (0.00s) --- PASS: TestFastPathPredicateValidation/Multiple_non-time_predicates (0.00s) --- PASS: TestFastPathPredicateValidation/OR_with_time_predicate_(unsafe) (0.00s) --- PASS: TestFastPathPredicateValidation/OR_with_only_time_predicates_(still_unsafe) (0.00s) --- PASS: TestFastPathPredicateValidation/String_column_comparison (0.00s) --- PASS: TestFastPathPredicateValidation/Numeric_column_comparison (0.00s) --- PASS: TestFastPathPredicateValidation/Internal_timestamp_column (0.00s) === RUN TestFastPathAggregationSafety === RUN TestFastPathAggregationSafety/No_WHERE_-_should_use_fast_path fast_path_predicate_validation_test.go:215: No WHERE - should use fast path: canAttemptFastPath=true (onlyTimePredicates=true, startTimeNs=0, stopTimeNs=0) === RUN TestFastPathAggregationSafety/Time-only_WHERE_-_should_use_fast_path fast_path_predicate_validation_test.go:215: Time-only WHERE - should use fast path: canAttemptFastPath=true (onlyTimePredicates=true, startTimeNs=1640995200000000000, stopTimeNs=0) === RUN TestFastPathAggregationSafety/Mixed_WHERE_-_should_NOT_use_fast_path fast_path_predicate_validation_test.go:215: Mixed WHERE - should NOT use fast path: canAttemptFastPath=false (onlyTimePredicates=false, startTimeNs=1640995200000000000, stopTimeNs=0) === RUN TestFastPathAggregationSafety/Non-time_WHERE_-_should_NOT_use_fast_path fast_path_predicate_validation_test.go:215: Non-time WHERE - should NOT use fast path: canAttemptFastPath=false (onlyTimePredicates=false, startTimeNs=0, stopTimeNs=0) === RUN TestFastPathAggregationSafety/OR_expression_-_should_NOT_use_fast_path fast_path_predicate_validation_test.go:215: OR expression - should NOT use fast path: canAttemptFastPath=false (onlyTimePredicates=false, startTimeNs=0, stopTimeNs=0) --- PASS: TestFastPathAggregationSafety (0.01s) --- PASS: TestFastPathAggregationSafety/No_WHERE_-_should_use_fast_path (0.00s) --- PASS: TestFastPathAggregationSafety/Time-only_WHERE_-_should_use_fast_path (0.00s) --- PASS: TestFastPathAggregationSafety/Mixed_WHERE_-_should_NOT_use_fast_path (0.00s) --- PASS: TestFastPathAggregationSafety/Non-time_WHERE_-_should_NOT_use_fast_path (0.00s) --- PASS: TestFastPathAggregationSafety/OR_expression_-_should_NOT_use_fast_path (0.00s) === RUN TestTimestampColumnDetection === RUN TestTimestampColumnDetection/_ts fast_path_predicate_validation_test.go:269: Column '_ts': isTimestamp=true === RUN TestTimestampColumnDetection/_ts_ns fast_path_predicate_validation_test.go:269: Column '_ts_ns': isTimestamp=true === RUN TestTimestampColumnDetection/user_id fast_path_predicate_validation_test.go:269: Column 'user_id': isTimestamp=false === RUN TestTimestampColumnDetection/id fast_path_predicate_validation_test.go:269: Column 'id': isTimestamp=false === RUN TestTimestampColumnDetection/status fast_path_predicate_validation_test.go:269: Column 'status': isTimestamp=false === RUN TestTimestampColumnDetection/event_type fast_path_predicate_validation_test.go:269: Column 'event_type': isTimestamp=false --- PASS: TestTimestampColumnDetection (0.00s) --- PASS: TestTimestampColumnDetection/_ts (0.00s) --- PASS: TestTimestampColumnDetection/_ts_ns (0.00s) --- PASS: TestTimestampColumnDetection/user_id (0.00s) --- PASS: TestTimestampColumnDetection/id (0.00s) --- PASS: TestTimestampColumnDetection/status (0.00s) --- PASS: TestTimestampColumnDetection/event_type (0.00s) === RUN TestSQLEngine_HybridSelectBasic hybrid_test.go:67: Found live_log data source from unflushed messages --- PASS: TestSQLEngine_HybridSelectBasic (0.00s) === RUN TestSQLEngine_HybridSelectWithLimit --- PASS: TestSQLEngine_HybridSelectWithLimit (0.00s) === RUN TestSQLEngine_HybridSelectDifferentTables hybrid_test.go:129: Table user_events: 1 columns, 10 rows with hybrid data sources hybrid_test.go:129: Table system_logs: 1 columns, 4 rows with hybrid data sources --- PASS: TestSQLEngine_HybridSelectDifferentTables (0.00s) === RUN TestSQLEngine_HybridDataSource hybrid_test.go:178: Found live event: live_login from live_log hybrid_test.go:178: Found live event: live_action from live_log hybrid_test.go:183: Found archived event: archived_login from parquet_archive hybrid_test.go:183: Found archived event: archived_logout from parquet_archive --- PASS: TestSQLEngine_HybridDataSource (0.00s) === RUN TestSQLEngine_HybridSystemLogs hybrid_test.go:240: Live system log: level=INFO hybrid_test.go:240: Live system log: level=WARN hybrid_test.go:248: Archived system log: level=ERROR hybrid_test.go:248: Archived system log: level=INFO --- PASS: TestSQLEngine_HybridSystemLogs (0.00s) === RUN TestSQLEngine_HybridSelectWithTimeImplications hybrid_test.go:304: Hybrid query results: 4 live messages, 6 archived messages --- PASS: TestSQLEngine_HybridSelectWithTimeImplications (0.00s) === RUN TestMockBrokerClient_BasicFunctionality --- PASS: TestMockBrokerClient_BasicFunctionality (0.00s) === RUN TestMockBrokerClient_FailureScenarios --- PASS: TestMockBrokerClient_FailureScenarios (0.00s) === RUN TestMockBrokerClient_TopicManagement --- PASS: TestMockBrokerClient_TopicManagement (0.00s) === RUN TestSQLEngineWithMockBrokerClient_ErrorHandling mock_test.go:147: ExecuteSQL returned error (acceptable): topic default.nonexistent_topic not found or no schema available: mock broker failure: mock broker unavailable --- PASS: TestSQLEngineWithMockBrokerClient_ErrorHandling (0.00s) === RUN TestNoSchemaError --- PASS: TestNoSchemaError (0.00s) === RUN TestParseSQL_OFFSET_EdgeCases === RUN TestParseSQL_OFFSET_EdgeCases/Valid_LIMIT_OFFSET_with_WHERE === RUN TestParseSQL_OFFSET_EdgeCases/LIMIT_OFFSET_with_mixed_case === RUN TestParseSQL_OFFSET_EdgeCases/LIMIT_OFFSET_with_extra_spaces --- PASS: TestParseSQL_OFFSET_EdgeCases (0.00s) --- PASS: TestParseSQL_OFFSET_EdgeCases/Valid_LIMIT_OFFSET_with_WHERE (0.00s) --- PASS: TestParseSQL_OFFSET_EdgeCases/LIMIT_OFFSET_with_mixed_case (0.00s) --- PASS: TestParseSQL_OFFSET_EdgeCases/LIMIT_OFFSET_with_extra_spaces (0.00s) === RUN TestSQLEngine_OFFSET_EdgeCases === RUN TestSQLEngine_OFFSET_EdgeCases/OFFSET_larger_than_result_set === RUN TestSQLEngine_OFFSET_EdgeCases/OFFSET_with_LIMIT_0 === RUN TestSQLEngine_OFFSET_EdgeCases/High_OFFSET_with_small_LIMIT --- PASS: TestSQLEngine_OFFSET_EdgeCases (0.01s) --- PASS: TestSQLEngine_OFFSET_EdgeCases/OFFSET_larger_than_result_set (0.00s) --- PASS: TestSQLEngine_OFFSET_EdgeCases/OFFSET_with_LIMIT_0 (0.00s) --- PASS: TestSQLEngine_OFFSET_EdgeCases/High_OFFSET_with_small_LIMIT (0.00s) === RUN TestSQLEngine_OFFSET_ErrorCases === RUN TestSQLEngine_OFFSET_ErrorCases/Negative_OFFSET_value offset_test.go:149: Parser accepts negative OFFSET, execution should validate === RUN TestSQLEngine_OFFSET_ErrorCases/Very_large_OFFSET_value --- PASS: TestSQLEngine_OFFSET_ErrorCases (0.00s) --- PASS: TestSQLEngine_OFFSET_ErrorCases/Negative_OFFSET_value (0.00s) --- PASS: TestSQLEngine_OFFSET_ErrorCases/Very_large_OFFSET_value (0.00s) === RUN TestSQLEngine_OFFSET_Consistency === RUN TestSQLEngine_OFFSET_Consistency/OFFSET_0 === RUN TestSQLEngine_OFFSET_Consistency/OFFSET_1 === RUN TestSQLEngine_OFFSET_Consistency/OFFSET_2 === RUN TestSQLEngine_OFFSET_Consistency/OFFSET_3 === RUN TestSQLEngine_OFFSET_Consistency/OFFSET_4 === RUN TestSQLEngine_OFFSET_Consistency/OFFSET_5 === RUN TestSQLEngine_OFFSET_Consistency/OFFSET_6 === RUN TestSQLEngine_OFFSET_Consistency/OFFSET_7 === RUN TestSQLEngine_OFFSET_Consistency/OFFSET_8 === RUN TestSQLEngine_OFFSET_Consistency/OFFSET_9 --- PASS: TestSQLEngine_OFFSET_Consistency (0.02s) --- PASS: TestSQLEngine_OFFSET_Consistency/OFFSET_0 (0.00s) --- PASS: TestSQLEngine_OFFSET_Consistency/OFFSET_1 (0.00s) --- PASS: TestSQLEngine_OFFSET_Consistency/OFFSET_2 (0.00s) --- PASS: TestSQLEngine_OFFSET_Consistency/OFFSET_3 (0.00s) --- PASS: TestSQLEngine_OFFSET_Consistency/OFFSET_4 (0.00s) --- PASS: TestSQLEngine_OFFSET_Consistency/OFFSET_5 (0.00s) --- PASS: TestSQLEngine_OFFSET_Consistency/OFFSET_6 (0.00s) --- PASS: TestSQLEngine_OFFSET_Consistency/OFFSET_7 (0.00s) --- PASS: TestSQLEngine_OFFSET_Consistency/OFFSET_8 (0.00s) --- PASS: TestSQLEngine_OFFSET_Consistency/OFFSET_9 (0.00s) === RUN TestSQLEngine_LIMIT_OFFSET_BugFix === RUN TestSQLEngine_LIMIT_OFFSET_BugFix/LIMIT_10_OFFSET_5_returns_correct_count offset_test.go:240: LIMIT 10 OFFSET 5 returned 5 rows (within limit) === RUN TestSQLEngine_LIMIT_OFFSET_BugFix/LIMIT_5_OFFSET_0 offset_test.go:287: LIMIT 5 OFFSET 0: returned 5 rows (within limit 5) === RUN TestSQLEngine_LIMIT_OFFSET_BugFix/LIMIT_5_OFFSET_2 offset_test.go:287: LIMIT 5 OFFSET 2: returned 5 rows (within limit 5) === RUN TestSQLEngine_LIMIT_OFFSET_BugFix/LIMIT_8_OFFSET_3 offset_test.go:287: LIMIT 8 OFFSET 3: returned 7 rows (within limit 8) === RUN TestSQLEngine_LIMIT_OFFSET_BugFix/LIMIT_15_OFFSET_1 offset_test.go:287: LIMIT 15 OFFSET 1: returned 9 rows (within limit 15) === RUN TestSQLEngine_LIMIT_OFFSET_BugFix/LIMIT_3_OFFSET_7 offset_test.go:287: LIMIT 3 OFFSET 7: returned 3 rows (within limit 3) === RUN TestSQLEngine_LIMIT_OFFSET_BugFix/LIMIT_12_OFFSET_4 offset_test.go:287: LIMIT 12 OFFSET 4: returned 6 rows (within limit 12) --- PASS: TestSQLEngine_LIMIT_OFFSET_BugFix (0.02s) --- PASS: TestSQLEngine_LIMIT_OFFSET_BugFix/LIMIT_10_OFFSET_5_returns_correct_count (0.00s) --- PASS: TestSQLEngine_LIMIT_OFFSET_BugFix/LIMIT_5_OFFSET_0 (0.00s) --- PASS: TestSQLEngine_LIMIT_OFFSET_BugFix/LIMIT_5_OFFSET_2 (0.00s) --- PASS: TestSQLEngine_LIMIT_OFFSET_BugFix/LIMIT_8_OFFSET_3 (0.00s) --- PASS: TestSQLEngine_LIMIT_OFFSET_BugFix/LIMIT_15_OFFSET_1 (0.00s) --- PASS: TestSQLEngine_LIMIT_OFFSET_BugFix/LIMIT_3_OFFSET_7 (0.00s) --- PASS: TestSQLEngine_LIMIT_OFFSET_BugFix/LIMIT_12_OFFSET_4 (0.00s) === RUN TestSQLEngine_OFFSET_DataCollectionBuffer === RUN TestSQLEngine_OFFSET_DataCollectionBuffer/Large_OFFSET_with_small_LIMIT === RUN TestSQLEngine_OFFSET_DataCollectionBuffer/Medium_OFFSET_with_medium_LIMIT === RUN TestSQLEngine_OFFSET_DataCollectionBuffer/Progressive_OFFSET_test offset_test.go:351: OFFSET 0: returned 3 rows offset_test.go:351: OFFSET 1: returned 3 rows offset_test.go:351: OFFSET 2: returned 3 rows offset_test.go:351: OFFSET 3: returned 3 rows offset_test.go:351: OFFSET 4: returned 3 rows offset_test.go:351: OFFSET 5: returned 3 rows --- PASS: TestSQLEngine_OFFSET_DataCollectionBuffer (0.03s) --- PASS: TestSQLEngine_OFFSET_DataCollectionBuffer/Large_OFFSET_with_small_LIMIT (0.00s) --- PASS: TestSQLEngine_OFFSET_DataCollectionBuffer/Medium_OFFSET_with_medium_LIMIT (0.00s) --- PASS: TestSQLEngine_OFFSET_DataCollectionBuffer/Progressive_OFFSET_test (0.02s) === RUN TestSQLEngine_LIMIT_OFFSET_ArithmeticExpressions === RUN TestSQLEngine_LIMIT_OFFSET_ArithmeticExpressions/Arithmetic_expressions_with_LIMIT_OFFSET offset_test.go:391: LIMIT 10: returned 10 rows offset_test.go:392: LIMIT 10 OFFSET 5: returned 5 rows offset_test.go:399: Insufficient data (10 rows) to fully test LIMIT 10 OFFSET 5 scenario offset_test.go:409: Row 0: id=417224, user_id=7810, id+user_id=425034 offset_test.go:409: Row 1: id=424297, user_id=8897, id+user_id=433194 offset_test.go:409: Row 2: id=431189, user_id=3400, id+user_id=434589 offset_test.go:409: Row 3: id=413249, user_id=5175, id+user_id=418424 offset_test.go:409: Row 4: id=120612, user_id=5429, id+user_id=126041 === RUN TestSQLEngine_LIMIT_OFFSET_ArithmeticExpressions/Multiplication_expressions offset_test.go:441: Row 0: id=82460, id*2=164920 ✓ offset_test.go:441: Row 1: id=841256, id*2=1682512 ✓ offset_test.go:441: Row 2: id=55537, id*2=111074 ✓ --- PASS: TestSQLEngine_LIMIT_OFFSET_ArithmeticExpressions (0.01s) --- PASS: TestSQLEngine_LIMIT_OFFSET_ArithmeticExpressions/Arithmetic_expressions_with_LIMIT_OFFSET (0.00s) --- PASS: TestSQLEngine_LIMIT_OFFSET_ArithmeticExpressions/Multiplication_expressions (0.00s) === RUN TestSQLEngine_OFFSET_WithAggregation === RUN TestSQLEngine_OFFSET_WithAggregation/COUNT_with_OFFSET === RUN TestSQLEngine_OFFSET_WithAggregation/COUNT_with_OFFSET_1 --- PASS: TestSQLEngine_OFFSET_WithAggregation (0.01s) --- PASS: TestSQLEngine_OFFSET_WithAggregation/COUNT_with_OFFSET (0.00s) --- PASS: TestSQLEngine_OFFSET_WithAggregation/COUNT_with_OFFSET_1 (0.00s) === RUN TestBasicParsing === RUN TestBasicParsing/Query_1 parsing_debug_test.go:20: Testing SQL: SELECT * FROM user_events parsing_debug_test.go:28: Parsed statement type: *engine.SelectStatement parsing_debug_test.go:31: SelectStatement details: parsing_debug_test.go:32: SelectExprs count: 1 parsing_debug_test.go:33: From count: 1 parsing_debug_test.go:34: WHERE clause exists: false parsing_debug_test.go:39: WHERE clause is NIL - this is the bug! === RUN TestBasicParsing/Query_2 parsing_debug_test.go:20: Testing SQL: SELECT id FROM user_events parsing_debug_test.go:28: Parsed statement type: *engine.SelectStatement parsing_debug_test.go:31: SelectStatement details: parsing_debug_test.go:32: SelectExprs count: 1 parsing_debug_test.go:33: From count: 1 parsing_debug_test.go:34: WHERE clause exists: false parsing_debug_test.go:39: WHERE clause is NIL - this is the bug! === RUN TestBasicParsing/Query_3 parsing_debug_test.go:20: Testing SQL: SELECT id FROM user_events WHERE id = 123 parsing_debug_test.go:28: Parsed statement type: *engine.SelectStatement parsing_debug_test.go:31: SelectStatement details: parsing_debug_test.go:32: SelectExprs count: 1 parsing_debug_test.go:33: From count: 1 parsing_debug_test.go:34: WHERE clause exists: true parsing_debug_test.go:37: WHERE expression type: *engine.ComparisonExpr === RUN TestBasicParsing/Query_4 parsing_debug_test.go:20: Testing SQL: SELECT id FROM user_events WHERE id > 123 parsing_debug_test.go:28: Parsed statement type: *engine.SelectStatement parsing_debug_test.go:31: SelectStatement details: parsing_debug_test.go:32: SelectExprs count: 1 parsing_debug_test.go:33: From count: 1 parsing_debug_test.go:34: WHERE clause exists: true parsing_debug_test.go:37: WHERE expression type: *engine.ComparisonExpr === RUN TestBasicParsing/Query_5 parsing_debug_test.go:20: Testing SQL: SELECT id FROM user_events WHERE status = 'active' parsing_debug_test.go:28: Parsed statement type: *engine.SelectStatement parsing_debug_test.go:31: SelectStatement details: parsing_debug_test.go:32: SelectExprs count: 1 parsing_debug_test.go:33: From count: 1 parsing_debug_test.go:34: WHERE clause exists: true parsing_debug_test.go:37: WHERE expression type: *engine.ComparisonExpr --- PASS: TestBasicParsing (0.03s) --- PASS: TestBasicParsing/Query_1 (0.00s) --- PASS: TestBasicParsing/Query_2 (0.00s) --- PASS: TestBasicParsing/Query_3 (0.01s) --- PASS: TestBasicParsing/Query_4 (0.01s) --- PASS: TestBasicParsing/Query_5 (0.01s) === RUN TestCockroachParserDirectly parsing_debug_test.go:53: Testing CockroachDB parser directly with: SELECT id FROM user_events WHERE id > 123 parsing_debug_test.go:61: Our ParseSQL returned: *engine.SelectStatement parsing_debug_test.go:68: Our ParseSQL extracted WHERE clause: *engine.ComparisonExpr --- PASS: TestCockroachParserDirectly (0.05s) === RUN TestParseMethodComparison parsing_debug_test.go:77: Comparing parsing methods for: SELECT id FROM user_events WHERE id > 123 parsing_debug_test.go:81: Global ParseSQL: *engine.SelectStatement, error: parsing_debug_test.go:84: WHERE clause: true parsing_debug_test.go:92: ExecuteSQL error (helps identify parsing path): --- PASS: TestParseMethodComparison (0.00s) === RUN TestPostgreSQLOnlySupport === RUN TestPostgreSQLOnlySupport/MySQL_Backticks_Table postgresql_only_test.go:89: CORRECTLY REJECTED: MySQL backticks for table names should be rejected === RUN TestPostgreSQLOnlySupport/MySQL_Backticks_Column postgresql_only_test.go:89: CORRECTLY REJECTED: MySQL backticks for column names should be rejected === RUN TestPostgreSQLOnlySupport/PostgreSQL_Double_Quotes_OK postgresql_only_test.go:103: CORRECTLY ACCEPTED: PostgreSQL double quotes for identifiers should work === RUN TestPostgreSQLOnlySupport/PostgreSQL_EXTRACT_OK postgresql_only_test.go:103: CORRECTLY ACCEPTED: PostgreSQL EXTRACT function should work === RUN TestPostgreSQLOnlySupport/Single_Quotes_String_Literal_OK postgresql_only_test.go:103: CORRECTLY ACCEPTED: Single quotes for string literals should work === NAME TestPostgreSQLOnlySupport postgresql_only_test.go:109: PostgreSQL-only compliance: 5/5 tests passed --- PASS: TestPostgreSQLOnlySupport (0.01s) --- PASS: TestPostgreSQLOnlySupport/MySQL_Backticks_Table (0.00s) --- PASS: TestPostgreSQLOnlySupport/MySQL_Backticks_Column (0.00s) --- PASS: TestPostgreSQLOnlySupport/PostgreSQL_Double_Quotes_OK (0.00s) --- PASS: TestPostgreSQLOnlySupport/PostgreSQL_EXTRACT_OK (0.00s) --- PASS: TestPostgreSQLOnlySupport/Single_Quotes_String_Literal_OK (0.00s) === RUN TestParseSQL_COUNT_Functions === RUN TestParseSQL_COUNT_Functions/COUNT(*)_basic === RUN TestParseSQL_COUNT_Functions/COUNT(column_name) === RUN TestParseSQL_COUNT_Functions/Multiple_aggregate_functions --- PASS: TestParseSQL_COUNT_Functions (0.00s) --- PASS: TestParseSQL_COUNT_Functions/COUNT(*)_basic (0.00s) --- PASS: TestParseSQL_COUNT_Functions/COUNT(column_name) (0.00s) --- PASS: TestParseSQL_COUNT_Functions/Multiple_aggregate_functions (0.00s) === RUN TestParseSQL_SELECT_Expressions === RUN TestParseSQL_SELECT_Expressions/SELECT_*_FROM_table === RUN TestParseSQL_SELECT_Expressions/SELECT_column_FROM_table === RUN TestParseSQL_SELECT_Expressions/SELECT_multiple_columns --- PASS: TestParseSQL_SELECT_Expressions (0.00s) --- PASS: TestParseSQL_SELECT_Expressions/SELECT_*_FROM_table (0.00s) --- PASS: TestParseSQL_SELECT_Expressions/SELECT_column_FROM_table (0.00s) --- PASS: TestParseSQL_SELECT_Expressions/SELECT_multiple_columns (0.00s) === RUN TestParseSQL_WHERE_Clauses === RUN TestParseSQL_WHERE_Clauses/WHERE_with_simple_comparison === RUN TestParseSQL_WHERE_Clauses/WHERE_with_AND_condition === RUN TestParseSQL_WHERE_Clauses/WHERE_with_OR_condition --- PASS: TestParseSQL_WHERE_Clauses (0.00s) --- PASS: TestParseSQL_WHERE_Clauses/WHERE_with_simple_comparison (0.00s) --- PASS: TestParseSQL_WHERE_Clauses/WHERE_with_AND_condition (0.00s) --- PASS: TestParseSQL_WHERE_Clauses/WHERE_with_OR_condition (0.00s) === RUN TestParseSQL_LIMIT_Clauses === RUN TestParseSQL_LIMIT_Clauses/LIMIT_with_number === RUN TestParseSQL_LIMIT_Clauses/LIMIT_with_OFFSET === RUN TestParseSQL_LIMIT_Clauses/LIMIT_with_OFFSET_zero === RUN TestParseSQL_LIMIT_Clauses/LIMIT_with_large_OFFSET --- PASS: TestParseSQL_LIMIT_Clauses (0.00s) --- PASS: TestParseSQL_LIMIT_Clauses/LIMIT_with_number (0.00s) --- PASS: TestParseSQL_LIMIT_Clauses/LIMIT_with_OFFSET (0.00s) --- PASS: TestParseSQL_LIMIT_Clauses/LIMIT_with_OFFSET_zero (0.00s) --- PASS: TestParseSQL_LIMIT_Clauses/LIMIT_with_large_OFFSET (0.00s) === RUN TestParseSQL_SHOW_Statements === RUN TestParseSQL_SHOW_Statements/SHOW_DATABASES === RUN TestParseSQL_SHOW_Statements/SHOW_TABLES === RUN TestParseSQL_SHOW_Statements/SHOW_TABLES_FROM_database --- PASS: TestParseSQL_SHOW_Statements (0.00s) --- PASS: TestParseSQL_SHOW_Statements/SHOW_DATABASES (0.00s) --- PASS: TestParseSQL_SHOW_Statements/SHOW_TABLES (0.00s) --- PASS: TestParseSQL_SHOW_Statements/SHOW_TABLES_FROM_database (0.00s) === RUN TestRealNamespaceDiscovery real_namespace_test.go:24: Discovered 0 namespaces (no fallback data): real_namespace_test.go:26: (No namespaces found - requires real SeaweedFS MQ cluster) --- PASS: TestRealNamespaceDiscovery (0.08s) === RUN TestRealTopicDiscovery real_namespace_test.go:53: Discovered 0 topics in 'default' namespace (no fallback data): real_namespace_test.go:55: (No topics found - requires real SeaweedFS MQ cluster with 'default' namespace) --- PASS: TestRealTopicDiscovery (0.02s) === RUN TestNamespaceDiscoveryNoFallback real_namespace_test.go:79: ListNamespaces failed as expected: failed to get filer client: failed to discover filer: failed to list filers from master: rpc error: code = Unavailable desc = connection error: desc = "transport: Error while dialing: dial tcp [::1]:18888: connect: connection refused" real_namespace_test.go:99: No fallback behavior - returns empty lists when filer unavailable --- PASS: TestNamespaceDiscoveryNoFallback (0.03s) === RUN TestRealWorldWhereClauseFailure real_world_where_clause_test.go:44: TESTING REAL-WORLD WHERE CLAUSE SCENARIOS real_world_where_clause_test.go:45: ============================================ === RUN TestRealWorldWhereClauseFailure/Where_ID_Greater_Than_Large_Number real_world_where_clause_test.go:65: Query: SELECT id FROM user_events WHERE id > 10000000 real_world_where_clause_test.go:66: Total rows returned: 0 real_world_where_clause_test.go:131: No rows returned - this could be correct if no data matches === RUN TestRealWorldWhereClauseFailure/Where_ID_Greater_Than_Small_Number real_world_where_clause_test.go:65: Query: SELECT id FROM user_events WHERE id > 100000 real_world_where_clause_test.go:66: Total rows returned: 7 real_world_where_clause_test.go:69: Sample IDs returned: real_world_where_clause_test.go:78: Row 1: id = 841256 real_world_where_clause_test.go:78: Row 2: id = 686003 real_world_where_clause_test.go:78: Row 3: id = 417224 real_world_where_clause_test.go:78: Row 4: id = 424297 real_world_where_clause_test.go:78: Row 5: id = 431189 real_world_where_clause_test.go:120: Analysis: real_world_where_clause_test.go:121: Rows matching WHERE condition: 7 real_world_where_clause_test.go:122: Rows NOT matching WHERE condition: 0 real_world_where_clause_test.go:128: PASS: WHERE id > 100000 should filter results - All returned rows match the WHERE condition === RUN TestRealWorldWhereClauseFailure/Where_ID_Less_Than real_world_where_clause_test.go:65: Query: SELECT id FROM user_events WHERE id < 100000 real_world_where_clause_test.go:66: Total rows returned: 3 real_world_where_clause_test.go:69: Sample IDs returned: real_world_where_clause_test.go:78: Row 1: id = 82460 real_world_where_clause_test.go:78: Row 2: id = 55537 real_world_where_clause_test.go:78: Row 3: id = 65143 real_world_where_clause_test.go:120: Analysis: real_world_where_clause_test.go:121: Rows matching WHERE condition: 3 real_world_where_clause_test.go:122: Rows NOT matching WHERE condition: 0 real_world_where_clause_test.go:128: PASS: WHERE id < 100000 should filter results - All returned rows match the WHERE condition --- PASS: TestRealWorldWhereClauseFailure (0.01s) --- PASS: TestRealWorldWhereClauseFailure/Where_ID_Greater_Than_Large_Number (0.00s) --- PASS: TestRealWorldWhereClauseFailure/Where_ID_Greater_Than_Small_Number (0.00s) --- PASS: TestRealWorldWhereClauseFailure/Where_ID_Less_Than (0.00s) === RUN TestWhereClauseWithLimitOffset real_world_where_clause_test.go:144: Testing exact failing query: SELECT id FROM user_events WHERE id > 10000000 LIMIT 10 OFFSET 5 real_world_where_clause_test.go:159: Returned 0 rows (LIMIT 10 worked) real_world_where_clause_test.go:181: WHERE clause working correctly --- PASS: TestWhereClauseWithLimitOffset (0.00s) === RUN TestWhatShouldHaveBeenTested real_world_where_clause_test.go:189: THE TEST THAT SHOULD HAVE CAUGHT THE WHERE CLAUSE ISSUE real_world_where_clause_test.go:190: ======================================================== real_world_where_clause_test.go:199: All rows: 10 real_world_where_clause_test.go:200: WHERE id > 999999999: 0 rows real_world_where_clause_test.go:213: WHERE 1 = 0 (impossible): 0 rows --- PASS: TestWhatShouldHaveBeenTested (0.00s) === RUN TestSchemaAwareParsing === RUN TestSchemaAwareParsing/JSON_Message_Parsing schema_parsing_test.go:74: JSON parsing correctly converted types: int32=1234, string='login', double=75.5, bool=true === RUN TestSchemaAwareParsing/Raw_Data_Type_Conversion schema_parsing_test.go:118: Raw data type conversions working correctly === RUN TestSchemaAwareParsing/Invalid_JSON_Graceful_Handling schema_parsing_test.go:129: Invalid JSON handled gracefully with error --- PASS: TestSchemaAwareParsing (0.00s) --- PASS: TestSchemaAwareParsing/JSON_Message_Parsing (0.00s) --- PASS: TestSchemaAwareParsing/Raw_Data_Type_Conversion (0.00s) --- PASS: TestSchemaAwareParsing/Invalid_JSON_Graceful_Handling (0.00s) === RUN TestSchemaAwareParsingIntegration schema_parsing_test.go:160: Schema-aware parsing integrates correctly with SQL engine --- PASS: TestSchemaAwareParsingIntegration (0.00s) === RUN TestSQLEngine_SelectBasic --- PASS: TestSQLEngine_SelectBasic (0.00s) === RUN TestSQLEngine_SelectWithLimit --- PASS: TestSQLEngine_SelectWithLimit (0.00s) === RUN TestSQLEngine_SelectSpecificColumns --- PASS: TestSQLEngine_SelectSpecificColumns (0.00s) === RUN TestSQLEngine_SelectFromNonExistentTable select_test.go:83: Skipping non-existent table test - table name parsing issue needs investigation --- SKIP: TestSQLEngine_SelectFromNonExistentTable (0.02s) === RUN TestSQLEngine_SelectWithOffset --- PASS: TestSQLEngine_SelectWithOffset (0.00s) === RUN TestSQLEngine_SelectWithLimitAndOffset --- PASS: TestSQLEngine_SelectWithLimitAndOffset (0.00s) === RUN TestSQLEngine_SelectWithOffsetExceedsRows --- PASS: TestSQLEngine_SelectWithOffsetExceedsRows (0.00s) === RUN TestSQLEngine_SelectWithOffsetZero --- PASS: TestSQLEngine_SelectWithOffsetZero (0.00s) === RUN TestSQLEngine_SelectDifferentTables select_test.go:211: Table user_events: 4 columns, 10 rows select_test.go:211: Table system_logs: 3 columns, 4 rows --- PASS: TestSQLEngine_SelectDifferentTables (0.00s) === RUN TestSQLAliasResolution === RUN TestSQLAliasResolution/ResolveColumnAlias === RUN TestSQLAliasResolution/SingleAliasInWhere === RUN TestSQLAliasResolution/MultipleAliasesInWhere === RUN TestSQLAliasResolution/RangeQueryWithAliases === RUN TestSQLAliasResolution/MixedAliasAndDirectColumn === RUN TestSQLAliasResolution/AliasCompatibilityWithTimestampFixes === RUN TestSQLAliasResolution/EdgeCasesAndErrorHandling === RUN TestSQLAliasResolution/ComparisonOperators === RUN TestSQLAliasResolution/ComparisonOperators/=_1000 === RUN TestSQLAliasResolution/ComparisonOperators/=_999 === RUN TestSQLAliasResolution/ComparisonOperators/>_999 === RUN TestSQLAliasResolution/ComparisonOperators/>_1000 === RUN TestSQLAliasResolution/ComparisonOperators/>=_1000 === RUN TestSQLAliasResolution/ComparisonOperators/>=_1001 === RUN TestSQLAliasResolution/ComparisonOperators/<_1001 === RUN TestSQLAliasResolution/ComparisonOperators/<_1000 === RUN TestSQLAliasResolution/ComparisonOperators/<=_1000 === RUN TestSQLAliasResolution/ComparisonOperators/<=_999 === RUN TestSQLAliasResolution/BackwardCompatibility --- PASS: TestSQLAliasResolution (0.02s) --- PASS: TestSQLAliasResolution/ResolveColumnAlias (0.00s) --- PASS: TestSQLAliasResolution/SingleAliasInWhere (0.00s) --- PASS: TestSQLAliasResolution/MultipleAliasesInWhere (0.00s) --- PASS: TestSQLAliasResolution/RangeQueryWithAliases (0.00s) --- PASS: TestSQLAliasResolution/MixedAliasAndDirectColumn (0.00s) --- PASS: TestSQLAliasResolution/AliasCompatibilityWithTimestampFixes (0.00s) --- PASS: TestSQLAliasResolution/EdgeCasesAndErrorHandling (0.00s) --- PASS: TestSQLAliasResolution/ComparisonOperators (0.01s) --- PASS: TestSQLAliasResolution/ComparisonOperators/=_1000 (0.00s) --- PASS: TestSQLAliasResolution/ComparisonOperators/=_999 (0.00s) --- PASS: TestSQLAliasResolution/ComparisonOperators/>_999 (0.00s) --- PASS: TestSQLAliasResolution/ComparisonOperators/>_1000 (0.00s) --- PASS: TestSQLAliasResolution/ComparisonOperators/>=_1000 (0.00s) --- PASS: TestSQLAliasResolution/ComparisonOperators/>=_1001 (0.00s) --- PASS: TestSQLAliasResolution/ComparisonOperators/<_1001 (0.00s) --- PASS: TestSQLAliasResolution/ComparisonOperators/<_1000 (0.00s) --- PASS: TestSQLAliasResolution/ComparisonOperators/<=_1000 (0.00s) --- PASS: TestSQLAliasResolution/ComparisonOperators/<=_999 (0.00s) --- PASS: TestSQLAliasResolution/BackwardCompatibility (0.00s) === RUN TestAliasIntegrationWithProductionScenarios === RUN TestAliasIntegrationWithProductionScenarios/OriginalFailingQuery === RUN TestAliasIntegrationWithProductionScenarios/ComplexProductionQuery === RUN TestAliasIntegrationWithProductionScenarios/PerformanceRegression --- PASS: TestAliasIntegrationWithProductionScenarios (0.00s) --- PASS: TestAliasIntegrationWithProductionScenarios/OriginalFailingQuery (0.00s) --- PASS: TestAliasIntegrationWithProductionScenarios/ComplexProductionQuery (0.00s) --- PASS: TestAliasIntegrationWithProductionScenarios/PerformanceRegression (0.00s) === RUN TestSQLFeatureDiagnostic sql_feature_diagnostic_test.go:14: SEAWEEDFS SQL ENGINE FEATURE DIAGNOSTIC sql_feature_diagnostic_test.go:15: ================================================================================ sql_feature_diagnostic_test.go:18: 1. TESTING LIMIT FUNCTIONALITY: sql_feature_diagnostic_test.go:35: LIMIT 0: PASS - Got 0 rows sql_feature_diagnostic_test.go:35: LIMIT 1: PASS - Got 1 rows sql_feature_diagnostic_test.go:35: LIMIT 3: PASS - Got 3 rows sql_feature_diagnostic_test.go:35: LIMIT 5: PASS - Got 5 rows sql_feature_diagnostic_test.go:35: LIMIT 10: PASS - Got 10 rows sql_feature_diagnostic_test.go:35: LIMIT 100: PASS - Got 10 rows sql_feature_diagnostic_test.go:43: 2. TESTING OFFSET FUNCTIONALITY: sql_feature_diagnostic_test.go:58: OFFSET 0: PASS - Got 3 rows sql_feature_diagnostic_test.go:58: OFFSET 1: PASS - Got 3 rows sql_feature_diagnostic_test.go:58: OFFSET 2: PASS - Got 3 rows sql_feature_diagnostic_test.go:58: OFFSET 5: PASS - Got 3 rows sql_feature_diagnostic_test.go:56: OFFSET 10: PASS - Beyond data range, got 0 rows sql_feature_diagnostic_test.go:56: OFFSET 100: PASS - Beyond data range, got 0 rows sql_feature_diagnostic_test.go:64: 3. TESTING WHERE CLAUSE FUNCTIONALITY: sql_feature_diagnostic_test.go:90: Specific ID match: PASS - WHERE clause working, got 1 rows sql_feature_diagnostic_test.go:90: Greater than comparison: PASS - WHERE clause working, got 7 rows sql_feature_diagnostic_test.go:90: String equality: PASS - WHERE clause working, got 1 rows sql_feature_diagnostic_test.go:90: Non-existent ID: PASS - WHERE clause working, got 0 rows sql_feature_diagnostic_test.go:90: Always false condition: PASS - WHERE clause working, got 0 rows sql_feature_diagnostic_test.go:96: 4. TESTING COMBINED LIMIT + OFFSET + WHERE: sql_feature_diagnostic_test.go:106: Combined query: Got 2 rows (LIMIT=2 part works, WHERE filtering unknown) sql_feature_diagnostic_test.go:110: ================================================================================ sql_feature_diagnostic_test.go:111: FEATURE SUMMARY: sql_feature_diagnostic_test.go:112: LIMIT: FULLY WORKING - Correctly limits result rows sql_feature_diagnostic_test.go:113: OFFSET: FULLY WORKING - Correctly skips rows sql_feature_diagnostic_test.go:114: WHERE: FULLY WORKING - All comparison operators working sql_feature_diagnostic_test.go:115: SELECT: WORKING - Supports *, columns, functions, arithmetic sql_feature_diagnostic_test.go:116: Functions: WORKING - String and datetime functions work sql_feature_diagnostic_test.go:117: Arithmetic: WORKING - +, -, *, / operations work sql_feature_diagnostic_test.go:118: ================================================================================ --- PASS: TestSQLFeatureDiagnostic (0.02s) === RUN TestSQLWhereClauseIssue sql_feature_diagnostic_test.go:125: DEMONSTRATING WHERE CLAUSE ISSUE: sql_feature_diagnostic_test.go:130: Total rows in test data: 10 sql_feature_diagnostic_test.go:134: First row ID: 82460 sql_feature_diagnostic_test.go:144: WHERE id = 82460 returned 1 rows sql_feature_diagnostic_test.go:152: WHERE clause working correctly sql_feature_diagnostic_test.go:162: WHERE 1 = 0 returned 0 rows sql_feature_diagnostic_test.go:167: Impossible WHERE condition correctly returns no rows --- PASS: TestSQLWhereClauseIssue (0.00s) === RUN TestSQLFilteringLimitOffset === RUN TestSQLFilteringLimitOffset/Where_Equals_Integer sql_filtering_limit_offset_test.go:295: PASS: WHERE with equals operator (integer) - Correct row count: 1 === RUN TestSQLFilteringLimitOffset/Where_Equals_String sql_filtering_limit_offset_test.go:298: PASS: WHERE with equals operator (string) - Row count: 1 (not validated) === RUN TestSQLFilteringLimitOffset/Where_Not_Equals sql_filtering_limit_offset_test.go:298: PASS: WHERE with not equals operator - Row count: 2 (not validated) === RUN TestSQLFilteringLimitOffset/Where_Greater_Than sql_filtering_limit_offset_test.go:298: PASS: WHERE with greater than operator - Row count: 7 (not validated) === RUN TestSQLFilteringLimitOffset/Where_Less_Than sql_filtering_limit_offset_test.go:298: PASS: WHERE with less than operator - Row count: 3 (not validated) === RUN TestSQLFilteringLimitOffset/Where_Greater_Equal sql_filtering_limit_offset_test.go:298: PASS: WHERE with greater than or equal operator - Row count: 8 (not validated) === RUN TestSQLFilteringLimitOffset/Where_Less_Equal sql_filtering_limit_offset_test.go:298: PASS: WHERE with less than or equal operator - Row count: 3 (not validated) === RUN TestSQLFilteringLimitOffset/Where_Column_Comparison sql_filtering_limit_offset_test.go:295: PASS: WHERE filtering with specific columns selected - Correct row count: 1 === RUN TestSQLFilteringLimitOffset/Where_With_Function sql_filtering_limit_offset_test.go:298: PASS: WHERE with function in SELECT - Row count: 1 (not validated) === RUN TestSQLFilteringLimitOffset/Where_With_Arithmetic sql_filtering_limit_offset_test.go:295: PASS: WHERE with arithmetic in SELECT - Correct row count: 1 === RUN TestSQLFilteringLimitOffset/Limit_1 sql_filtering_limit_offset_test.go:295: PASS: LIMIT 1 row - Correct row count: 1 === RUN TestSQLFilteringLimitOffset/Limit_5 sql_filtering_limit_offset_test.go:295: PASS: LIMIT 5 rows - Correct row count: 5 === RUN TestSQLFilteringLimitOffset/Limit_0 sql_filtering_limit_offset_test.go:295: PASS: LIMIT 0 rows (should return no results) - Correct row count: 0 === RUN TestSQLFilteringLimitOffset/Limit_Large sql_filtering_limit_offset_test.go:298: PASS: LIMIT with large number - Row count: 10 (not validated) === RUN TestSQLFilteringLimitOffset/Limit_With_Columns sql_filtering_limit_offset_test.go:295: PASS: LIMIT with specific columns - Correct row count: 3 === RUN TestSQLFilteringLimitOffset/Limit_With_Functions sql_filtering_limit_offset_test.go:295: PASS: LIMIT with functions - Correct row count: 2 === RUN TestSQLFilteringLimitOffset/Offset_0 sql_filtering_limit_offset_test.go:295: PASS: OFFSET 0 (same as no offset) - Correct row count: 5 === RUN TestSQLFilteringLimitOffset/Offset_1 sql_filtering_limit_offset_test.go:295: PASS: OFFSET 1 row - Correct row count: 3 === RUN TestSQLFilteringLimitOffset/Offset_5 sql_filtering_limit_offset_test.go:295: PASS: OFFSET 5 rows - Correct row count: 2 === RUN TestSQLFilteringLimitOffset/Offset_Large sql_filtering_limit_offset_test.go:298: PASS: OFFSET with large number - Row count: 0 (not validated) === RUN TestSQLFilteringLimitOffset/Limit_Offset_Pagination_Page1 sql_filtering_limit_offset_test.go:295: PASS: Pagination: Page 1 (LIMIT 3, OFFSET 0) - Correct row count: 3 === RUN TestSQLFilteringLimitOffset/Limit_Offset_Pagination_Page2 sql_filtering_limit_offset_test.go:295: PASS: Pagination: Page 2 (LIMIT 3, OFFSET 3) - Correct row count: 3 === RUN TestSQLFilteringLimitOffset/Limit_Offset_Pagination_Page3 sql_filtering_limit_offset_test.go:295: PASS: Pagination: Page 3 (LIMIT 3, OFFSET 6) - Correct row count: 3 === RUN TestSQLFilteringLimitOffset/Where_Limit sql_filtering_limit_offset_test.go:298: PASS: WHERE clause with LIMIT - Row count: 1 (not validated) === RUN TestSQLFilteringLimitOffset/Where_Limit_Offset sql_filtering_limit_offset_test.go:298: PASS: WHERE clause with LIMIT and OFFSET - Row count: 0 (not validated) === RUN TestSQLFilteringLimitOffset/Where_Complex_Limit sql_filtering_limit_offset_test.go:298: PASS: Complex WHERE with functions and arithmetic, plus LIMIT - Row count: 3 (not validated) === RUN TestSQLFilteringLimitOffset/Where_No_Match sql_filtering_limit_offset_test.go:295: PASS: WHERE clause that matches no rows - Correct row count: 0 === RUN TestSQLFilteringLimitOffset/Limit_Offset_Beyond_Data sql_filtering_limit_offset_test.go:295: PASS: OFFSET beyond available data - Correct row count: 0 === RUN TestSQLFilteringLimitOffset/Where_Empty_String sql_filtering_limit_offset_test.go:298: PASS: WHERE with empty string value - Row count: 0 (not validated) === RUN TestSQLFilteringLimitOffset/Small_Result_Set sql_filtering_limit_offset_test.go:295: PASS: Optimized query: specific WHERE + LIMIT 1 - Correct row count: 1 === RUN TestSQLFilteringLimitOffset/Batch_Processing sql_filtering_limit_offset_test.go:298: PASS: Batch processing pattern: moderate LIMIT - Row count: 10 (not validated) === NAME TestSQLFilteringLimitOffset sql_filtering_limit_offset_test.go:307: ================================================================================ sql_filtering_limit_offset_test.go:308: SQL FILTERING, LIMIT & OFFSET TEST SUITE SUMMARY sql_filtering_limit_offset_test.go:309: ================================================================================ sql_filtering_limit_offset_test.go:310: Total Tests: 31 sql_filtering_limit_offset_test.go:311: Successful: 31 sql_filtering_limit_offset_test.go:312: Errors: 0 sql_filtering_limit_offset_test.go:313: Row Count Mismatches: 0 sql_filtering_limit_offset_test.go:314: ================================================================================ --- PASS: TestSQLFilteringLimitOffset (0.06s) --- PASS: TestSQLFilteringLimitOffset/Where_Equals_Integer (0.00s) --- PASS: TestSQLFilteringLimitOffset/Where_Equals_String (0.00s) --- PASS: TestSQLFilteringLimitOffset/Where_Not_Equals (0.00s) --- PASS: TestSQLFilteringLimitOffset/Where_Greater_Than (0.00s) --- PASS: TestSQLFilteringLimitOffset/Where_Less_Than (0.00s) --- PASS: TestSQLFilteringLimitOffset/Where_Greater_Equal (0.00s) --- PASS: TestSQLFilteringLimitOffset/Where_Less_Equal (0.00s) --- PASS: TestSQLFilteringLimitOffset/Where_Column_Comparison (0.00s) --- PASS: TestSQLFilteringLimitOffset/Where_With_Function (0.00s) --- PASS: TestSQLFilteringLimitOffset/Where_With_Arithmetic (0.00s) --- PASS: TestSQLFilteringLimitOffset/Limit_1 (0.00s) --- PASS: TestSQLFilteringLimitOffset/Limit_5 (0.00s) --- PASS: TestSQLFilteringLimitOffset/Limit_0 (0.00s) --- PASS: TestSQLFilteringLimitOffset/Limit_Large (0.00s) --- PASS: TestSQLFilteringLimitOffset/Limit_With_Columns (0.00s) --- PASS: TestSQLFilteringLimitOffset/Limit_With_Functions (0.00s) --- PASS: TestSQLFilteringLimitOffset/Offset_0 (0.00s) --- PASS: TestSQLFilteringLimitOffset/Offset_1 (0.00s) --- PASS: TestSQLFilteringLimitOffset/Offset_5 (0.00s) --- PASS: TestSQLFilteringLimitOffset/Offset_Large (0.00s) --- PASS: TestSQLFilteringLimitOffset/Limit_Offset_Pagination_Page1 (0.00s) --- PASS: TestSQLFilteringLimitOffset/Limit_Offset_Pagination_Page2 (0.00s) --- PASS: TestSQLFilteringLimitOffset/Limit_Offset_Pagination_Page3 (0.00s) --- PASS: TestSQLFilteringLimitOffset/Where_Limit (0.00s) --- PASS: TestSQLFilteringLimitOffset/Where_Limit_Offset (0.00s) --- PASS: TestSQLFilteringLimitOffset/Where_Complex_Limit (0.00s) --- PASS: TestSQLFilteringLimitOffset/Where_No_Match (0.00s) --- PASS: TestSQLFilteringLimitOffset/Limit_Offset_Beyond_Data (0.00s) --- PASS: TestSQLFilteringLimitOffset/Where_Empty_String (0.00s) --- PASS: TestSQLFilteringLimitOffset/Small_Result_Set (0.00s) --- PASS: TestSQLFilteringLimitOffset/Batch_Processing (0.00s) === RUN TestSQLFilteringAccuracy sql_filtering_limit_offset_test.go:335: Testing SQL filtering accuracy with specific data verification sql_filtering_limit_offset_test.go:350: PASS: Exact ID filtering works correctly sql_filtering_limit_offset_test.go:363: PASS: LIMIT 3 returns exactly 3 rows sql_filtering_limit_offset_test.go:380: PASS: OFFSET 1 correctly skips first row --- PASS: TestSQLFilteringAccuracy (0.00s) === RUN TestSQLFilteringEdgeCases === RUN TestSQLFilteringEdgeCases/Zero_Limit sql_filtering_limit_offset_test.go:441: PASS: LIMIT 0 should return empty result set - Rows: 0 === RUN TestSQLFilteringEdgeCases/Large_Offset sql_filtering_limit_offset_test.go:441: PASS: Very large OFFSET should handle gracefully - Rows: 0 === RUN TestSQLFilteringEdgeCases/Where_False_Condition sql_filtering_limit_offset_test.go:431: UNEXPECTED SUCCESS: WHERE with always-false condition (may indicate feature is implemented) === RUN TestSQLFilteringEdgeCases/Complex_Where sql_filtering_limit_offset_test.go:431: UNEXPECTED SUCCESS: Complex WHERE with AND condition (may indicate feature is implemented) --- PASS: TestSQLFilteringEdgeCases (0.01s) --- PASS: TestSQLFilteringEdgeCases/Zero_Limit (0.00s) --- PASS: TestSQLFilteringEdgeCases/Large_Offset (0.00s) --- PASS: TestSQLFilteringEdgeCases/Where_False_Condition (0.00s) --- PASS: TestSQLFilteringEdgeCases/Complex_Where (0.00s) === RUN TestSQLEngine_StringConcatenationWithLiterals === RUN TestSQLEngine_StringConcatenationWithLiterals/Simple_concatenation_with_literals string_concatenation_test.go:108: Query: SELECT 'test' || action || 'end' FROM user_events LIMIT 1 string_concatenation_test.go:109: Columns: ['test'||action||'end'] string_concatenation_test.go:115: Row 0: [testloginend] === RUN TestSQLEngine_StringConcatenationWithLiterals/User's_original_complex_concatenation string_concatenation_test.go:92: Expected column 0 to be ''test'||action||'xxx'||action||'~~~'||status', got ''test'||action||'xxx'||action||' ~~~ '||status' string_concatenation_test.go:108: Query: SELECT 'test' || action || 'xxx' || action || ' ~~~ ' || status FROM user_events LIMIT 1 string_concatenation_test.go:109: Columns: ['test'||action||'xxx'||action||' ~~~ '||status] string_concatenation_test.go:115: Row 0: [testloginxxxlogin ~~~ active] === RUN TestSQLEngine_StringConcatenationWithLiterals/Mixed_columns_and_literals string_concatenation_test.go:108: Query: SELECT status || '=' || action, 'prefix:' || user_type FROM user_events LIMIT 1 string_concatenation_test.go:109: Columns: [status||'='||action 'prefix:'||user_type] string_concatenation_test.go:115: Row 0: [active=login prefix:premium] === RUN TestSQLEngine_StringConcatenationWithLiterals/Concatenation_with_spaces_in_literals string_concatenation_test.go:92: Expected column 0 to be ''['||status||']'', got '' [ '||status||' ] '' string_concatenation_test.go:108: Query: SELECT ' [ ' || status || ' ] ' FROM user_events LIMIT 2 string_concatenation_test.go:109: Columns: [' [ '||status||' ] '] string_concatenation_test.go:115: Row 0: [ [ active ] ] string_concatenation_test.go:115: Row 1: [ [ pending ] ] --- PASS: TestSQLEngine_StringConcatenationWithLiterals (0.01s) --- PASS: TestSQLEngine_StringConcatenationWithLiterals/Simple_concatenation_with_literals (0.00s) --- PASS: TestSQLEngine_StringConcatenationWithLiterals/User's_original_complex_concatenation (0.00s) --- PASS: TestSQLEngine_StringConcatenationWithLiterals/Mixed_columns_and_literals (0.00s) --- PASS: TestSQLEngine_StringConcatenationWithLiterals/Concatenation_with_spaces_in_literals (0.00s) === RUN TestSQLEngine_StringConcatenationBugReproduction string_concatenation_test.go:180: SUCCESS: Complex string concatenation works correctly! string_concatenation_test.go:181: Query: SELECT UPPER(status), id*2, 'test' || action || 'xxx' || action || ' ~~~ ' || status FROM user_events LIMIT 2 string_concatenation_test.go:188: Row 0: [ACTIVE 164920 testloginxxxlogin ~~~ active] string_concatenation_test.go:188: Row 1: [PENDING 1682512 testclickxxxclick ~~~ pending] --- PASS: TestSQLEngine_StringConcatenationBugReproduction (0.00s) === RUN TestStringFunctions === RUN TestStringFunctions/LENGTH_function_tests === RUN TestStringFunctions/LENGTH_function_tests/Length_of_string === RUN TestStringFunctions/LENGTH_function_tests/Length_of_empty_string === RUN TestStringFunctions/LENGTH_function_tests/Length_of_number === RUN TestStringFunctions/LENGTH_function_tests/Length_of_null_value === RUN TestStringFunctions/UPPER/LOWER_function_tests === RUN TestStringFunctions/TRIM_function_tests === RUN TestStringFunctions/TRIM_function_tests/TRIM_whitespace === RUN TestStringFunctions/TRIM_function_tests/LTRIM_whitespace === RUN TestStringFunctions/TRIM_function_tests/RTRIM_whitespace === RUN TestStringFunctions/TRIM_function_tests/TRIM_with_tabs_and_newlines === RUN TestStringFunctions/SUBSTRING_function_tests === RUN TestStringFunctions/CONCAT_function_tests === RUN TestStringFunctions/REPLACE_function_tests === RUN TestStringFunctions/POSITION_function_tests === RUN TestStringFunctions/LEFT/RIGHT_function_tests === RUN TestStringFunctions/REVERSE_function_tests --- PASS: TestStringFunctions (0.01s) --- PASS: TestStringFunctions/LENGTH_function_tests (0.00s) --- PASS: TestStringFunctions/LENGTH_function_tests/Length_of_string (0.00s) --- PASS: TestStringFunctions/LENGTH_function_tests/Length_of_empty_string (0.00s) --- PASS: TestStringFunctions/LENGTH_function_tests/Length_of_number (0.00s) --- PASS: TestStringFunctions/LENGTH_function_tests/Length_of_null_value (0.00s) --- PASS: TestStringFunctions/UPPER/LOWER_function_tests (0.00s) --- PASS: TestStringFunctions/TRIM_function_tests (0.00s) --- PASS: TestStringFunctions/TRIM_function_tests/TRIM_whitespace (0.00s) --- PASS: TestStringFunctions/TRIM_function_tests/LTRIM_whitespace (0.00s) --- PASS: TestStringFunctions/TRIM_function_tests/RTRIM_whitespace (0.00s) --- PASS: TestStringFunctions/TRIM_function_tests/TRIM_with_tabs_and_newlines (0.00s) --- PASS: TestStringFunctions/SUBSTRING_function_tests (0.00s) --- PASS: TestStringFunctions/CONCAT_function_tests (0.00s) --- PASS: TestStringFunctions/REPLACE_function_tests (0.00s) --- PASS: TestStringFunctions/POSITION_function_tests (0.00s) --- PASS: TestStringFunctions/LEFT/RIGHT_function_tests (0.00s) --- PASS: TestStringFunctions/REVERSE_function_tests (0.00s) === RUN TestStringFunctionsSQL === RUN TestStringFunctionsSQL/UPPER_function === RUN TestStringFunctionsSQL/LOWER_function === RUN TestStringFunctionsSQL/LENGTH_function === RUN TestStringFunctionsSQL/TRIM_function === RUN TestStringFunctionsSQL/LTRIM_function === RUN TestStringFunctionsSQL/RTRIM_function === RUN TestStringFunctionsSQL/Multiple_string_functions === RUN TestStringFunctionsSQL/String_function_with_wrong_argument_count === RUN TestStringFunctionsSQL/String_function_with_no_arguments --- PASS: TestStringFunctionsSQL (0.02s) --- PASS: TestStringFunctionsSQL/UPPER_function (0.00s) --- PASS: TestStringFunctionsSQL/LOWER_function (0.00s) --- PASS: TestStringFunctionsSQL/LENGTH_function (0.00s) --- PASS: TestStringFunctionsSQL/TRIM_function (0.00s) --- PASS: TestStringFunctionsSQL/LTRIM_function (0.00s) --- PASS: TestStringFunctionsSQL/RTRIM_function (0.00s) --- PASS: TestStringFunctionsSQL/Multiple_string_functions (0.00s) --- PASS: TestStringFunctionsSQL/String_function_with_wrong_argument_count (0.00s) --- PASS: TestStringFunctionsSQL/String_function_with_no_arguments (0.00s) === RUN TestSQLEngine_StringFunctionsAndLiterals === RUN TestSQLEngine_StringFunctionsAndLiterals/String_functions_-_UPPER_and_LENGTH string_literal_function_test.go:48: Status: 'active', UPPER: 'ACTIVE', LENGTH: '6' string_literal_function_test.go:160: Query: SELECT status, UPPER(status), LENGTH(status) FROM user_events LIMIT 3 string_literal_function_test.go:161: Columns: [status UPPER(status) LENGTH(status)] string_literal_function_test.go:167: Row 0: [active ACTIVE 6] string_literal_function_test.go:167: Row 1: [pending PENDING 7] string_literal_function_test.go:167: Row 2: [ ] === RUN TestSQLEngine_StringFunctionsAndLiterals/String_literal_in_SELECT string_literal_function_test.go:160: Query: SELECT id, user_id, 'good' FROM user_events LIMIT 2 string_literal_function_test.go:161: Columns: [id user_id 'good'] string_literal_function_test.go:167: Row 0: [82460 9465 good] string_literal_function_test.go:167: Row 1: [841256 2336 good] === RUN TestSQLEngine_StringFunctionsAndLiterals/Mixed:_columns,_functions,_arithmetic,_and_literals string_literal_function_test.go:160: Query: SELECT id, UPPER(status), id*2, 'test' FROM user_events LIMIT 2 string_literal_function_test.go:161: Columns: [id UPPER(status) id*2 'test'] string_literal_function_test.go:167: Row 0: [82460 ACTIVE 164920 test] string_literal_function_test.go:167: Row 1: [841256 PENDING 1682512 test] === RUN TestSQLEngine_StringFunctionsAndLiterals/User's_original_failing_query_-_fixed string_literal_function_test.go:160: Query: SELECT status, action, user_type, UPPER(action), LENGTH(action) FROM user_events LIMIT 2 string_literal_function_test.go:161: Columns: [status action user_type UPPER(action) LENGTH(action)] string_literal_function_test.go:167: Row 0: [active login premium LOGIN 5] string_literal_function_test.go:167: Row 1: [pending click standard CLICK 5] --- PASS: TestSQLEngine_StringFunctionsAndLiterals (0.02s) --- PASS: TestSQLEngine_StringFunctionsAndLiterals/String_functions_-_UPPER_and_LENGTH (0.00s) --- PASS: TestSQLEngine_StringFunctionsAndLiterals/String_literal_in_SELECT (0.01s) --- PASS: TestSQLEngine_StringFunctionsAndLiterals/Mixed:_columns,_functions,_arithmetic,_and_literals (0.00s) --- PASS: TestSQLEngine_StringFunctionsAndLiterals/User's_original_failing_query_-_fixed (0.00s) === RUN TestSQLEngine_StringFunctionErrorHandling string_literal_function_test.go:186: UPPER function works correctly string_literal_function_test.go:197: LENGTH function works correctly --- PASS: TestSQLEngine_StringFunctionErrorHandling (0.00s) === RUN TestTimestampIntegrationScenarios === RUN TestTimestampIntegrationScenarios/EndToEndTimestampEquality === RUN TestTimestampIntegrationScenarios/EndToEndTimestampEquality/original_failing_1 === RUN TestTimestampIntegrationScenarios/EndToEndTimestampEquality/original_failing_2 === RUN TestTimestampIntegrationScenarios/EndToEndTimestampEquality/current_data === RUN TestTimestampIntegrationScenarios/ComplexRangeQueries === RUN TestTimestampIntegrationScenarios/ComplexRangeQueries/RangeWithDifferentBounds === RUN TestTimestampIntegrationScenarios/ComplexRangeQueries/RangeWithSameBounds === RUN TestTimestampIntegrationScenarios/ComplexRangeQueries/OpenEndedRange === RUN TestTimestampIntegrationScenarios/ProductionScenarioReproduction --- PASS: TestTimestampIntegrationScenarios (0.01s) --- PASS: TestTimestampIntegrationScenarios/EndToEndTimestampEquality (0.01s) --- PASS: TestTimestampIntegrationScenarios/EndToEndTimestampEquality/original_failing_1 (0.00s) --- PASS: TestTimestampIntegrationScenarios/EndToEndTimestampEquality/original_failing_2 (0.00s) --- PASS: TestTimestampIntegrationScenarios/EndToEndTimestampEquality/current_data (0.00s) --- PASS: TestTimestampIntegrationScenarios/ComplexRangeQueries (0.00s) --- PASS: TestTimestampIntegrationScenarios/ComplexRangeQueries/RangeWithDifferentBounds (0.00s) --- PASS: TestTimestampIntegrationScenarios/ComplexRangeQueries/RangeWithSameBounds (0.00s) --- PASS: TestTimestampIntegrationScenarios/ComplexRangeQueries/OpenEndedRange (0.00s) --- PASS: TestTimestampIntegrationScenarios/ProductionScenarioReproduction (0.00s) === RUN TestRegressionPrevention === RUN TestRegressionPrevention/SmallTimestamps === RUN TestRegressionPrevention/NonTimestampColumns === RUN TestRegressionPrevention/StringComparisons --- PASS: TestRegressionPrevention (0.00s) --- PASS: TestRegressionPrevention/SmallTimestamps (0.00s) --- PASS: TestRegressionPrevention/NonTimestampColumns (0.00s) --- PASS: TestRegressionPrevention/StringComparisons (0.00s) === RUN TestTimestampQueryFixes === RUN TestTimestampQueryFixes/Fix1_PrecisionLoss === RUN TestTimestampQueryFixes/Fix2_TimeFilterExtraction === RUN TestTimestampQueryFixes/Fix3_RangeBoundaryFix === RUN TestTimestampQueryFixes/Fix4_DifferentRangeBoundaries === RUN TestTimestampQueryFixes/Fix5_PredicateAccuracy === RUN TestTimestampQueryFixes/Fix6_ComparisonOperators === RUN TestTimestampQueryFixes/Fix7_EdgeCases --- PASS: TestTimestampQueryFixes (0.01s) --- PASS: TestTimestampQueryFixes/Fix1_PrecisionLoss (0.00s) --- PASS: TestTimestampQueryFixes/Fix2_TimeFilterExtraction (0.00s) --- PASS: TestTimestampQueryFixes/Fix3_RangeBoundaryFix (0.00s) --- PASS: TestTimestampQueryFixes/Fix4_DifferentRangeBoundaries (0.00s) --- PASS: TestTimestampQueryFixes/Fix5_PredicateAccuracy (0.00s) --- PASS: TestTimestampQueryFixes/Fix6_ComparisonOperators (0.00s) --- PASS: TestTimestampQueryFixes/Fix7_EdgeCases (0.00s) === RUN TestOriginalFailingQueries === RUN TestOriginalFailingQueries/OriginalQuery1 === RUN TestOriginalFailingQueries/OriginalQuery2 === RUN TestOriginalFailingQueries/CurrentDataQuery --- PASS: TestOriginalFailingQueries (0.01s) --- PASS: TestOriginalFailingQueries/OriginalQuery1 (0.00s) --- PASS: TestOriginalFailingQueries/OriginalQuery2 (0.00s) --- PASS: TestOriginalFailingQueries/CurrentDataQuery (0.00s) === RUN TestWhereParsing === RUN TestWhereParsing/Simple_Equals where_clause_debug_test.go:78: PASS: WHERE clause parsed successfully for: Simple equality WHERE clause where_clause_debug_test.go:79: WHERE expression type: *engine.ComparisonExpr === RUN TestWhereParsing/Greater_Than where_clause_debug_test.go:78: PASS: WHERE clause parsed successfully for: Greater than WHERE clause where_clause_debug_test.go:79: WHERE expression type: *engine.ComparisonExpr === RUN TestWhereParsing/String_Equals where_clause_debug_test.go:78: PASS: WHERE clause parsed successfully for: String equality WHERE clause where_clause_debug_test.go:79: WHERE expression type: *engine.ComparisonExpr === RUN TestWhereParsing/Impossible_Condition where_clause_debug_test.go:78: PASS: WHERE clause parsed successfully for: Impossible WHERE condition (should parse but return no rows) where_clause_debug_test.go:79: WHERE expression type: *engine.ComparisonExpr --- PASS: TestWhereParsing (0.01s) --- PASS: TestWhereParsing/Simple_Equals (0.00s) --- PASS: TestWhereParsing/Greater_Than (0.00s) --- PASS: TestWhereParsing/String_Equals (0.00s) --- PASS: TestWhereParsing/Impossible_Condition (0.00s) === RUN TestPredicateBuilding === RUN TestPredicateBuilding/Simple_Equals_Match where_clause_debug_test.go:172: PASS: Simple equality - should match - Predicate worked correctly (match=true) === RUN TestPredicateBuilding/Simple_Equals_NoMatch where_clause_debug_test.go:172: PASS: Simple equality - should not match - Predicate worked correctly (match=false) === RUN TestPredicateBuilding/Greater_Than_Match where_clause_debug_test.go:172: PASS: Greater than - should match - Predicate worked correctly (match=true) === RUN TestPredicateBuilding/Greater_Than_NoMatch where_clause_debug_test.go:172: PASS: Greater than - should not match - Predicate worked correctly (match=false) === RUN TestPredicateBuilding/String_Equals_Match where_clause_debug_test.go:172: PASS: String equality - should match - Predicate worked correctly (match=true) === RUN TestPredicateBuilding/String_Equals_NoMatch where_clause_debug_test.go:172: PASS: String equality - should not match - Predicate worked correctly (match=false) === RUN TestPredicateBuilding/Impossible_Condition where_clause_debug_test.go:172: PASS: Impossible condition - should never match - Predicate worked correctly (match=false) --- PASS: TestPredicateBuilding (0.01s) --- PASS: TestPredicateBuilding/Simple_Equals_Match (0.00s) --- PASS: TestPredicateBuilding/Simple_Equals_NoMatch (0.00s) --- PASS: TestPredicateBuilding/Greater_Than_Match (0.00s) --- PASS: TestPredicateBuilding/Greater_Than_NoMatch (0.00s) --- PASS: TestPredicateBuilding/String_Equals_Match (0.00s) --- PASS: TestPredicateBuilding/String_Equals_NoMatch (0.00s) --- PASS: TestPredicateBuilding/Impossible_Condition (0.00s) === RUN TestWhereClauseEndToEnd where_clause_debug_test.go:185: END-TO-END WHERE CLAUSE VALIDATION where_clause_debug_test.go:186: =================================== where_clause_debug_test.go:194: Baseline (no WHERE): 10 rows where_clause_debug_test.go:202: WHERE 1 = 0: 0 rows where_clause_debug_test.go:210: Impossible WHERE condition correctly returns 0 rows where_clause_debug_test.go:222: WHERE id = 82460: 1 rows where_clause_debug_test.go:227: Specific ID WHERE clause working correctly where_clause_debug_test.go:239: WHERE id > 10000000: 0 rows where_clause_debug_test.go:256: WHERE id > 10000000 correctly filtered results --- PASS: TestWhereClauseEndToEnd (0.01s) === RUN TestSpecificWhereClauseBug where_clause_debug_test.go:298: REPRODUCING EXACT WHERE CLAUSE BUG where_clause_debug_test.go:299: ================================== where_clause_debug_test.go:309: Query: SELECT id FROM user_events WHERE id > 10000000 LIMIT 10 OFFSET 5 where_clause_debug_test.go:310: Returned 0 rows: where_clause_debug_test.go:326: WHERE clause working correctly - all IDs > 10,000,000 --- PASS: TestSpecificWhereClauseBug (0.00s) === RUN TestWhereClauseValidation where_validation_test.go:13: WHERE CLAUSE VALIDATION TESTS where_validation_test.go:14: ============================== where_validation_test.go:22: Baseline data - Total rows: 10 where_validation_test.go:24: Sample IDs: 82460, 841256, 55537 where_validation_test.go:38: WHERE id = 82460: 1 rows where_validation_test.go:40: Specific ID filtering works correctly where_validation_test.go:59: Data range: min ID = 55537, max ID = 841256 where_validation_test.go:69: WHERE id > 448396: 2 rows where_validation_test.go:83: Range filtering works correctly - all returned IDs > 448396 where_validation_test.go:95: WHERE status = 'active': 1 rows where_validation_test.go:107: String filtering works correctly where_validation_test.go:111: TESTING REAL-WORLD CASE: where_validation_test.go:118: Real-world query returned: 0 rows where_validation_test.go:131: Real-world case FIXED: No violations found --- PASS: TestWhereClauseValidation (0.01s) === RUN TestWhereClauseComparisonOperators where_validation_test.go:164: Testing comparison operators with ID = 841256 where_validation_test.go:175: WHERE id = 841256: 1 rows (equals) where_validation_test.go:175: WHERE id != 841256: 9 rows (not equals) where_validation_test.go:175: WHERE id > 841256: 0 rows (greater than) where_validation_test.go:175: WHERE id < 841256: 9 rows (less than) where_validation_test.go:175: WHERE id >= 841256: 1 rows (greater or equal) where_validation_test.go:175: WHERE id <= 841256: 10 rows (less or equal) --- PASS: TestWhereClauseComparisonOperators (0.01s) PASS ok github.com/seaweedfs/seaweedfs/weed/query/engine 23.968s === RUN TestGjson { "quiz": { "sport": { "q1": { "question": "Which one is correct team name in NBA?", "options": [ "New York Bulls", "Los Angeles Kings", "Golden State Warriros", "Huston Rocket" ], "answer": "Huston Rocket" } }, "maths": { "q1": { "question": "5 + 7 = ?", "options": [ "10", "11", "12", "13" ], "answer": "12" }, "q2": { "question": "12 - 8 = ?", "options": [ "1", "2", "3", "4" ], "answer": "4" } } } } +++++++++++ 12 5 { "sport": { "q1": { "question": "Which one is correct team name in NBA?", "options": [ "New York Bulls", "Los Angeles Kings", "Golden State Warriros", "Huston Rocket" ], "answer": "Huston Rocket" } }, "maths": { "q1": { "question": "5 + 7 = ?", "options": [ "10", "11", "12", "13" ], "answer": "12" }, "q2": { "question": "12 - 8 = ?", "options": [ "1", "2", "3", "4" ], "answer": "4" } } } 0 0 ----------- { "fruit": "Apple", "size": "Large", "quiz": "Red" } +++++++++++ 51 3 Red 13 3 Apple ----------- --- PASS: TestGjson (0.00s) === RUN TestJsonQueryRow {fruit:"Bl\"ue",size:6} --- PASS: TestJsonQueryRow (0.00s) === RUN TestJsonQueryNumber {fruit:"Bl\"ue",quiz:"green"} --- PASS: TestJsonQueryNumber (0.00s) PASS ok github.com/seaweedfs/seaweedfs/weed/query/json 1.508s ? github.com/seaweedfs/seaweedfs/weed/query/sqltypes [no test files] ? github.com/seaweedfs/seaweedfs/weed/remote_storage [no test files] === RUN TestNewAzBlobClientRequiresAccountName --- PASS: TestNewAzBlobClientRequiresAccountName (0.00s) === RUN TestNewAzBlobClientRejectsMalformedAccountName --- PASS: TestNewAzBlobClientRejectsMalformedAccountName (0.03s) === RUN TestNewAzBlobClientSharedKey --- PASS: TestNewAzBlobClientSharedKey (0.08s) === RUN TestNewAzBlobClientSharedKeyRejectsMalformedKey --- PASS: TestNewAzBlobClientSharedKeyRejectsMalformedKey (0.00s) === RUN TestAzureServiceURL --- PASS: TestAzureServiceURL (0.00s) === RUN TestNewAzBlobClientEntraID --- PASS: TestNewAzBlobClientEntraID (0.03s) === RUN TestNewAzBlobClientWorkloadIdentity --- PASS: TestNewAzBlobClientWorkloadIdentity (0.00s) === RUN TestNewAzureTokenCredentialManagedIdentity --- PASS: TestNewAzureTokenCredentialManagedIdentity (0.00s) === RUN TestAzureStorageClientBasic azure_storage_client_test.go:26: Skipping Azure storage test: AZURE_STORAGE_ACCOUNT or AZURE_STORAGE_ACCESS_KEY not set --- SKIP: TestAzureStorageClientBasic (0.00s) === RUN TestToMetadata === RUN TestToMetadata/basic_metadata === RUN TestToMetadata/metadata_with_dashes === RUN TestToMetadata/non-metadata_keys_ignored === RUN TestToMetadata/keys_starting_with_digits === RUN TestToMetadata/uppercase_and_mixed_case_keys === RUN TestToMetadata/keys_with_invalid_characters === RUN TestToMetadata/collision_prevention === RUN TestToMetadata/empty_input --- PASS: TestToMetadata (0.00s) --- PASS: TestToMetadata/basic_metadata (0.00s) --- PASS: TestToMetadata/metadata_with_dashes (0.00s) --- PASS: TestToMetadata/non-metadata_keys_ignored (0.00s) --- PASS: TestToMetadata/keys_starting_with_digits (0.00s) --- PASS: TestToMetadata/uppercase_and_mixed_case_keys (0.00s) --- PASS: TestToMetadata/keys_with_invalid_characters (0.00s) --- PASS: TestToMetadata/collision_prevention (0.00s) --- PASS: TestToMetadata/empty_input (0.00s) === RUN TestAzureRemoteStorageMaker --- PASS: TestAzureRemoteStorageMaker (0.00s) === RUN TestResolveAzureAccount --- PASS: TestResolveAzureAccount (0.00s) === RUN TestAzureRemoteStorageMakerWithoutAccountKey --- PASS: TestAzureRemoteStorageMakerWithoutAccountKey (0.00s) === RUN TestAzureStorageClientErrors --- PASS: TestAzureStorageClientErrors (10.21s) === RUN TestAzureRemoteStorageClientImplementsInterface --- PASS: TestAzureRemoteStorageClientImplementsInterface (0.00s) === RUN TestAzureErrRemoteObjectNotFoundIsAccessible --- PASS: TestAzureErrRemoteObjectNotFoundIsAccessible (0.00s) PASS ok github.com/seaweedfs/seaweedfs/weed/remote_storage/azure 27.473s === RUN TestGCSRemoteStorageClientImplementsInterface --- PASS: TestGCSRemoteStorageClientImplementsInterface (0.00s) === RUN TestGCSErrRemoteObjectNotFoundIsAccessible --- PASS: TestGCSErrRemoteObjectNotFoundIsAccessible (0.02s) === RUN TestMakeWithHTTPClientAllowedTypes W0824 22:18:28.074741 gcs_storage_client.go:101 need to specify GOOGLE_CLOUD_PROJECT env variable W0824 22:18:28.104122 gcs_storage_client.go:101 need to specify GOOGLE_CLOUD_PROJECT env variable --- PASS: TestMakeWithHTTPClientAllowedTypes (0.09s) === RUN TestParseInlineCredentials --- PASS: TestParseInlineCredentials (0.02s) PASS ok github.com/seaweedfs/seaweedfs/weed/remote_storage/gcs 19.071s === RUN TestS3CompatibleEndpointCoversAllProviders --- PASS: TestS3CompatibleEndpointCoversAllProviders (0.00s) === RUN TestS3MakeUsesAnonymousCredentialsWhenKeysAreEmpty --- PASS: TestS3MakeUsesAnonymousCredentialsWhenKeysAreEmpty (0.04s) === RUN TestS3MakeUsesStaticCredentialsWhenKeysAreProvided --- PASS: TestS3MakeUsesStaticCredentialsWhenKeysAreProvided (0.04s) === RUN TestS3RemoteStorageClientImplementsInterface --- PASS: TestS3RemoteStorageClientImplementsInterface (0.00s) === RUN TestS3ErrRemoteObjectNotFoundIsAccessible --- PASS: TestS3ErrRemoteObjectNotFoundIsAccessible (0.00s) === RUN TestS3RemoveDirectoryDeletesEveryListedObject --- PASS: TestS3RemoveDirectoryDeletesEveryListedObject (0.00s) === RUN TestS3RemoveDirectoryEmptyListingSendsNoDeletes --- PASS: TestS3RemoveDirectoryEmptyListingSendsNoDeletes (0.00s) === RUN TestS3RemoveDirectoryRefusesBucketRoot W0824 22:18:26.650627 s3_storage_client.go:438 s3 bucket: skip removing directory mapped to the bucket root --- PASS: TestS3RemoveDirectoryRefusesBucketRoot (0.02s) === RUN TestS3RemoveDirectoryReturnsBatchError --- PASS: TestS3RemoveDirectoryReturnsBatchError (0.00s) === RUN TestS3RemoveDirectoryReturnsPerKeyError --- PASS: TestS3RemoveDirectoryReturnsPerKeyError (0.00s) === RUN TestS3WriteFilePassesMimeAsContentType --- PASS: TestS3WriteFilePassesMimeAsContentType (0.13s) === RUN TestS3UpdateFileMetadataSkipsTaggingWhenUnsupported --- PASS: TestS3UpdateFileMetadataSkipsTaggingWhenUnsupported (0.00s) === RUN TestS3UpdateFileMetadataSendsTaggingWhenSupported --- PASS: TestS3UpdateFileMetadataSendsTaggingWhenSupported (0.11s) === RUN TestS3WriteFileOmitsContentTypeWhenMimeMissing --- PASS: TestS3WriteFileOmitsContentTypeWhenMimeMissing (0.01s) PASS ok github.com/seaweedfs/seaweedfs/weed/remote_storage/s3 17.849s === RUN TestReplicateRenameUsesTargetKeyForNonFilerSink --- PASS: TestReplicateRenameUsesTargetKeyForNonFilerSink (0.01s) === RUN TestReplicateRenameWithoutMoverUsesCreateThenDelete --- PASS: TestReplicateRenameWithoutMoverUsesCreateThenDelete (0.00s) === RUN TestReplicateRenameUsesMoveEntryWhenSupported --- PASS: TestReplicateRenameUsesMoveEntryWhenSupported (0.00s) === RUN TestReplicateInPlaceUpdateUsesUpdateEntry --- PASS: TestReplicateInPlaceUpdateUsesUpdateEntry (0.00s) === RUN TestReplicateInPlaceUpdateFallbackCreates --- PASS: TestReplicateInPlaceUpdateFallbackCreates (0.00s) === RUN TestPathIsEqualOrUnderUsesDirectoryBoundaries === RUN TestPathIsEqualOrUnderUsesDirectoryBoundaries/equal === RUN TestPathIsEqualOrUnderUsesDirectoryBoundaries/descendant === RUN TestPathIsEqualOrUnderUsesDirectoryBoundaries/sibling_prefix === RUN TestPathIsEqualOrUnderUsesDirectoryBoundaries/root --- PASS: TestPathIsEqualOrUnderUsesDirectoryBoundaries (0.00s) --- PASS: TestPathIsEqualOrUnderUsesDirectoryBoundaries/equal (0.00s) --- PASS: TestPathIsEqualOrUnderUsesDirectoryBoundaries/descendant (0.00s) --- PASS: TestPathIsEqualOrUnderUsesDirectoryBoundaries/sibling_prefix (0.00s) --- PASS: TestPathIsEqualOrUnderUsesDirectoryBoundaries/root (0.00s) === RUN TestReplicateRenameOutToSiblingPrefixBecomesDelete --- PASS: TestReplicateRenameOutToSiblingPrefixBecomesDelete (0.00s) === RUN TestReplicateRenameFromExcludedDirBecomesCreate --- PASS: TestReplicateRenameFromExcludedDirBecomesCreate (0.00s) PASS ok github.com/seaweedfs/seaweedfs/weed/replication 25.234s === RUN TestDetectSSEType === RUN TestDetectSSEType/no_chunks_no_extended === RUN TestDetectSSEType/plaintext_chunks === RUN TestDetectSSEType/uniform_SSE-S3_chunks === RUN TestDetectSSEType/uniform_SSE-KMS_chunks === RUN TestDetectSSEType/mixed_chunk_SSE_types === RUN TestDetectSSEType/inline_SSE-S3_via_extended === RUN TestDetectSSEType/inline_SSE-KMS_via_extended === RUN TestDetectSSEType/inline_SSE-C_via_extended === RUN TestDetectSSEType/conflicting_extended_metadata === RUN TestDetectSSEType/chunks_take_precedence_over_extended --- PASS: TestDetectSSEType (0.02s) --- PASS: TestDetectSSEType/no_chunks_no_extended (0.00s) --- PASS: TestDetectSSEType/plaintext_chunks (0.00s) --- PASS: TestDetectSSEType/uniform_SSE-S3_chunks (0.00s) --- PASS: TestDetectSSEType/uniform_SSE-KMS_chunks (0.00s) --- PASS: TestDetectSSEType/mixed_chunk_SSE_types (0.00s) --- PASS: TestDetectSSEType/inline_SSE-S3_via_extended (0.00s) --- PASS: TestDetectSSEType/inline_SSE-KMS_via_extended (0.00s) --- PASS: TestDetectSSEType/inline_SSE-C_via_extended (0.00s) --- PASS: TestDetectSSEType/conflicting_extended_metadata (0.00s) --- PASS: TestDetectSSEType/chunks_take_precedence_over_extended (0.00s) === RUN TestMaybeDecryptReader_Plaintext --- PASS: TestMaybeDecryptReader_Plaintext (0.00s) === RUN TestMaybeDecryptReader_NilEntry --- PASS: TestMaybeDecryptReader_NilEntry (0.00s) === RUN TestMaybeDecryptReader_SSEC_Error --- PASS: TestMaybeDecryptReader_SSEC_Error (0.00s) === RUN TestMaybeDecryptContent_Plaintext --- PASS: TestMaybeDecryptContent_Plaintext (0.00s) === RUN TestMaybeDecryptContent_NilEntry --- PASS: TestMaybeDecryptContent_NilEntry (0.00s) === RUN TestMaybeDecryptContent_Empty --- PASS: TestMaybeDecryptContent_Empty (0.00s) === RUN TestMaybeDecryptContent_SSEC_Error --- PASS: TestMaybeDecryptContent_SSEC_Error (0.00s) === RUN TestMaybeDecryptContent_MixedExtended_Error --- PASS: TestMaybeDecryptContent_MixedExtended_Error (0.00s) === RUN TestMaybeDecryptReader_SSES3 I0824 22:18:37.903259 s3_sse_s3.go:529 SSE-S3 KeyManager: Loaded KEK from s3.sse.kek config I0824 22:18:37.936925 s3_sse_s3.go:566 SSE-S3 KeyManager: No KEK configured. SSE-S3 encryption is disabled. Set s3.sse.kek or s3.sse.key in security.toml to enable it. --- PASS: TestMaybeDecryptReader_SSES3 (0.05s) === RUN TestMaybeDecryptContent_SSES3 I0824 22:18:37.938873 s3_sse_s3.go:529 SSE-S3 KeyManager: Loaded KEK from s3.sse.kek config I0824 22:18:37.942420 s3_sse_s3.go:566 SSE-S3 KeyManager: No KEK configured. SSE-S3 encryption is disabled. Set s3.sse.kek or s3.sse.key in security.toml to enable it. --- PASS: TestMaybeDecryptContent_SSES3 (0.00s) === RUN TestMaybeDecryptReader_SSEKMS --- PASS: TestMaybeDecryptReader_SSEKMS (0.00s) === RUN TestMaybeDecryptContent_SSEKMS --- PASS: TestMaybeDecryptContent_SSEKMS (0.00s) PASS ok github.com/seaweedfs/seaweedfs/weed/replication/repl_util 28.856s ? github.com/seaweedfs/seaweedfs/weed/replication/sink [no test files] === RUN TestAzureSinkInterface --- PASS: TestAzureSinkInterface (0.00s) === RUN TestAzureSinkInitialization azure_sink_test.go:99: Skipping Azure sink test: AZURE_STORAGE_ACCOUNT or AZURE_STORAGE_ACCESS_KEY not set --- SKIP: TestAzureSinkInitialization (0.02s) === RUN TestAzureSinkInitializeFromConfig azure_sink_test.go:132: Skipping Azure sink config test: AZURE_STORAGE_ACCOUNT or AZURE_STORAGE_ACCESS_KEY not set --- SKIP: TestAzureSinkInitializeFromConfig (0.00s) === RUN TestCleanKey === RUN TestCleanKey//test/file.txt === RUN TestCleanKey/test/file.txt === RUN TestCleanKey// === RUN TestCleanKey/#00 === RUN TestCleanKey//a/b/c --- PASS: TestCleanKey (0.00s) --- PASS: TestCleanKey//test/file.txt (0.00s) --- PASS: TestCleanKey/test/file.txt (0.00s) --- PASS: TestCleanKey// (0.00s) --- PASS: TestCleanKey/#00 (0.00s) --- PASS: TestCleanKey//a/b/c (0.00s) === RUN TestAzureSinkEntryOperations azure_sink_test.go:186: Skipping Azure sink entry test: credentials not set --- SKIP: TestAzureSinkEntryOperations (0.00s) === RUN TestAzureSinkPrecondition azure_sink_test.go:281: Skipping Azure sink precondition test: credentials not set --- SKIP: TestAzureSinkPrecondition (0.01s) === RUN TestAzureSinkIdempotentCreate azure_sink_test.go:350: Skipping Azure sink idempotent create test: credentials not set --- SKIP: TestAzureSinkIdempotentCreate (0.00s) === RUN TestAzureSinkInvalidCredentials azure_sink_test.go:465: Invalid credentials correctly rejected at initialization --- SKIP: TestAzureSinkInvalidCredentials (0.02s) PASS ok github.com/seaweedfs/seaweedfs/weed/replication/sink/azuresink 25.623s ? github.com/seaweedfs/seaweedfs/weed/replication/sink/b2sink [no test files] === RUN TestTargetPathToSourcePath === RUN TestTargetPathToSourcePath/basic_mapping === RUN TestTargetPathToSourcePath/incremental_sink_is_unmappable === RUN TestTargetPathToSourcePath/trailing_slash_roots === RUN TestTargetPathToSourcePath/root_target_mapping === RUN TestTargetPathToSourcePath/target_root_itself === RUN TestTargetPathToSourcePath/outside_target_root --- PASS: TestTargetPathToSourcePath (0.00s) --- PASS: TestTargetPathToSourcePath/basic_mapping (0.00s) --- PASS: TestTargetPathToSourcePath/incremental_sink_is_unmappable (0.00s) --- PASS: TestTargetPathToSourcePath/trailing_slash_roots (0.00s) --- PASS: TestTargetPathToSourcePath/root_target_mapping (0.00s) --- PASS: TestTargetPathToSourcePath/target_root_itself (0.00s) --- PASS: TestTargetPathToSourcePath/outside_target_root (0.00s) === RUN TestValidateReplicatedChunkSize === RUN TestValidateReplicatedChunkSize/healthy === RUN TestValidateReplicatedChunkSize/legitimately_empty_file === RUN TestValidateReplicatedChunkSize/zero-byte_read_for_non-empty_source === RUN TestValidateReplicatedChunkSize/short_read === RUN TestValidateReplicatedChunkSize/over-read_(server_returned_more_than_metadata) --- PASS: TestValidateReplicatedChunkSize (0.01s) --- PASS: TestValidateReplicatedChunkSize/healthy (0.00s) --- PASS: TestValidateReplicatedChunkSize/legitimately_empty_file (0.00s) --- PASS: TestValidateReplicatedChunkSize/zero-byte_read_for_non-empty_source (0.00s) --- PASS: TestValidateReplicatedChunkSize/short_read (0.00s) --- PASS: TestValidateReplicatedChunkSize/over-read_(server_returned_more_than_metadata) (0.00s) === RUN TestFetchAndWriteRejectsZeroByteSource I0824 22:18:24.321336 fetch_write.go:406 permanent size mismatch replicating 74,047d16a94aa581 for /dst/index.bin: chunk size mismatch: read 74,047d16a94aa581 got 0 bytes, source metadata says 5171 --- PASS: TestFetchAndWriteRejectsZeroByteSource (0.37s) === RUN TestIsRetryableNetworkError === RUN TestIsRetryableNetworkError/nil === RUN TestIsRetryableNetworkError/eof === RUN TestIsRetryableNetworkError/unexpected_eof === RUN TestIsRetryableNetworkError/volume_idle_timeout_json === RUN TestIsRetryableNetworkError/volume_idle_timeout_capitalized === RUN TestIsRetryableNetworkError/connection_reset === RUN TestIsRetryableNetworkError/connection_reset_capitalized === RUN TestIsRetryableNetworkError/broken_pipe === RUN TestIsRetryableNetworkError/broken_pipe_capitalized === RUN TestIsRetryableNetworkError/net.Error_timeout === RUN TestIsRetryableNetworkError/size_mismatch_is_permanent === RUN TestIsRetryableNetworkError/unrelated_error --- PASS: TestIsRetryableNetworkError (0.00s) --- PASS: TestIsRetryableNetworkError/nil (0.00s) --- PASS: TestIsRetryableNetworkError/eof (0.00s) --- PASS: TestIsRetryableNetworkError/unexpected_eof (0.00s) --- PASS: TestIsRetryableNetworkError/volume_idle_timeout_json (0.00s) --- PASS: TestIsRetryableNetworkError/volume_idle_timeout_capitalized (0.00s) --- PASS: TestIsRetryableNetworkError/connection_reset (0.00s) --- PASS: TestIsRetryableNetworkError/connection_reset_capitalized (0.00s) --- PASS: TestIsRetryableNetworkError/broken_pipe (0.00s) --- PASS: TestIsRetryableNetworkError/broken_pipe_capitalized (0.00s) --- PASS: TestIsRetryableNetworkError/net.Error_timeout (0.00s) --- PASS: TestIsRetryableNetworkError/size_mismatch_is_permanent (0.00s) --- PASS: TestIsRetryableNetworkError/unrelated_error (0.00s) === RUN TestReplicateChunksPreservesSizeMismatchSentinel I0824 22:18:24.368812 fetch_write.go:406 permanent size mismatch replicating 74,047d16a94aa581 for /dst/index.bin: chunk size mismatch: read 74,047d16a94aa581 got 0 bytes, source metadata says 5171 --- PASS: TestReplicateChunksPreservesSizeMismatchSentinel (0.02s) === RUN TestSourceSupersedes === RUN TestSourceSupersedes/not-found_sentinel === RUN TestSourceSupersedes/not-found_wrapped === RUN TestSourceSupersedes/not-found_as_string_(gRPC) === RUN TestSourceSupersedes/nil_entry,_nil_error === RUN TestSourceSupersedes/network_error === RUN TestSourceSupersedes/source_strictly_newer === RUN TestSourceSupersedes/source_same_version === RUN TestSourceSupersedes/source_older_(out-of-order_replay) --- PASS: TestSourceSupersedes (0.01s) --- PASS: TestSourceSupersedes/not-found_sentinel (0.00s) --- PASS: TestSourceSupersedes/not-found_wrapped (0.00s) --- PASS: TestSourceSupersedes/not-found_as_string_(gRPC) (0.00s) --- PASS: TestSourceSupersedes/nil_entry,_nil_error (0.00s) --- PASS: TestSourceSupersedes/network_error (0.00s) --- PASS: TestSourceSupersedes/source_strictly_newer (0.00s) --- PASS: TestSourceSupersedes/source_same_version (0.00s) --- PASS: TestSourceSupersedes/source_older_(out-of-order_replay) (0.00s) === RUN TestSourceSupersedesEpochMtime --- PASS: TestSourceSupersedesEpochMtime (0.00s) === RUN TestManifestResolveRetryGateUnverifiableSupersessionBounded I0824 22:18:24.379107 fetch_write.go:222 resolve manifest 3,01abc for /backup/2026-07-10/buckets/x/f.pt: LookupFileId volume id 3: not found I0824 22:18:24.379434 fetch_write.go:222 resolve manifest 3,01abc for /backup/2026-07-10/buckets/x/f.pt: LookupFileId volume id 3: not found I0824 22:18:24.379668 fetch_write.go:219 resolve manifest 3,01abc for /backup/2026-07-10/buckets/x/f.pt: supersession unverifiable, propagating after 3 attempts: LookupFileId volume id 3: not found --- PASS: TestManifestResolveRetryGateUnverifiableSupersessionBounded (0.00s) === RUN TestManifestResolveRetryGateNonTransientPropagates I0824 22:18:24.382264 fetch_write.go:214 resolve manifest 3,01abc for /backup/buckets/x/f.pt: non-transient error, propagating: fail to unmarshal manifest 3,01abc: proto: cannot parse invalid wire-format data I0824 22:18:24.382594 fetch_write.go:214 resolve manifest 3,01abc for /backup/buckets/x/f.pt: non-transient error, propagating: invalid fileId abc I0824 22:18:24.382708 fetch_write.go:222 resolve manifest 3,01abc for /backup/buckets/x/f.pt: LookupFileId volume id 3: not found --- PASS: TestManifestResolveRetryGateNonTransientPropagates (0.00s) === RUN TestGetEntryMtimeNs --- PASS: TestGetEntryMtimeNs (0.00s) === RUN TestOnCorruptChunkRefusesWhenSupersessionUnconfirmed E0824 22:18:24.383710 filer_sink.go:441 refuse to replicate entry with corrupt chunk /buckets/x/config: corrupt chunk: read 0 bytes, source says 107 --- PASS: TestOnCorruptChunkRefusesWhenSupersessionUnconfirmed (0.02s) === RUN TestOnReplicateChunkErrorPropagatesWhenSupersessionUnconfirmed --- PASS: TestOnReplicateChunkErrorPropagatesWhenSupersessionUnconfirmed (0.00s) === RUN TestChooseUpdateAction === RUN TestChooseUpdateAction/nil_existing_entry === RUN TestChooseUpdateAction/destination_older,_catching_up === RUN TestChooseUpdateAction/same_mtime === RUN TestChooseUpdateAction/destination_newer_and_complete === RUN TestChooseUpdateAction/destination_newer_and_larger === RUN TestChooseUpdateAction/newer_smaller_dest,_older_larger_event_->_skip === RUN TestChooseUpdateAction/same_second,_destination_newer_and_complete === RUN TestChooseUpdateAction/same_second,_destination_older === RUN TestChooseUpdateAction/behind,_covers_less_than_oldEntry_->_repair === RUN TestChooseUpdateAction/behind,_matches_oldEntry_coverage_->_normal === RUN TestChooseUpdateAction/behind,_exceeds_oldEntry_coverage_->_normal === RUN TestChooseUpdateAction/behind,_equal_total_but_range_gap_->_repair === RUN TestChooseUpdateAction/older,_stale_backup_chunk_->_repair === RUN TestChooseUpdateAction/equal-mtime_stale_->_deferred_(normal) === RUN TestChooseUpdateAction/older,_rsync_chunks_->_normal === RUN TestChooseUpdateAction/in-sync_(fast_path)_->_normal --- PASS: TestChooseUpdateAction (0.01s) --- PASS: TestChooseUpdateAction/nil_existing_entry (0.00s) --- PASS: TestChooseUpdateAction/destination_older,_catching_up (0.00s) --- PASS: TestChooseUpdateAction/same_mtime (0.00s) --- PASS: TestChooseUpdateAction/destination_newer_and_complete (0.00s) --- PASS: TestChooseUpdateAction/destination_newer_and_larger (0.00s) --- PASS: TestChooseUpdateAction/newer_smaller_dest,_older_larger_event_->_skip (0.00s) --- PASS: TestChooseUpdateAction/same_second,_destination_newer_and_complete (0.00s) --- PASS: TestChooseUpdateAction/same_second,_destination_older (0.00s) --- PASS: TestChooseUpdateAction/behind,_covers_less_than_oldEntry_->_repair (0.00s) --- PASS: TestChooseUpdateAction/behind,_matches_oldEntry_coverage_->_normal (0.00s) --- PASS: TestChooseUpdateAction/behind,_exceeds_oldEntry_coverage_->_normal (0.00s) --- PASS: TestChooseUpdateAction/behind,_equal_total_but_range_gap_->_repair (0.00s) --- PASS: TestChooseUpdateAction/older,_stale_backup_chunk_->_repair (0.00s) --- PASS: TestChooseUpdateAction/equal-mtime_stale_->_deferred_(normal) (0.00s) --- PASS: TestChooseUpdateAction/older,_rsync_chunks_->_normal (0.00s) --- PASS: TestChooseUpdateAction/in-sync_(fast_path)_->_normal (0.00s) === RUN TestCoversReference === RUN TestCoversReference/empty_reference === RUN TestCoversReference/reference_no_chunks === RUN TestCoversReference/exact === RUN TestCoversReference/superset === RUN TestCoversReference/truncated_tail === RUN TestCoversReference/equal_total,_range_gap === RUN TestCoversReference/contiguous_parts_cover === RUN TestCoversReference/overlapping_parts_cover === RUN TestCoversReference/existing_empty,_reference_has_bytes --- PASS: TestCoversReference (0.01s) --- PASS: TestCoversReference/empty_reference (0.00s) --- PASS: TestCoversReference/reference_no_chunks (0.00s) --- PASS: TestCoversReference/exact (0.00s) --- PASS: TestCoversReference/superset (0.00s) --- PASS: TestCoversReference/truncated_tail (0.00s) --- PASS: TestCoversReference/equal_total,_range_gap (0.00s) --- PASS: TestCoversReference/contiguous_parts_cover (0.00s) --- PASS: TestCoversReference/overlapping_parts_cover (0.00s) --- PASS: TestCoversReference/existing_empty,_reference_has_bytes (0.00s) === RUN TestDestinationSatisfiesSource === RUN TestDestinationSatisfiesSource/nil_existing === RUN TestDestinationSatisfiesSource/identical_complete_->_skip === RUN TestDestinationSatisfiesSource/truncated_(no_md5)_->_replicate === RUN TestDestinationSatisfiesSource/md5-backed_truncated_->_replicate === RUN TestDestinationSatisfiesSource/md5_equal_and_complete_->_skip === RUN TestDestinationSatisfiesSource/complete,_dest_newer_->_skip === RUN TestDestinationSatisfiesSource/complete,_dest_older_->_replicate === RUN TestDestinationSatisfiesSource/older,_stale_backup_chunk_->_replicate === RUN TestDestinationSatisfiesSource/equal-mtime_stale_->_skip_(deferred) === RUN TestDestinationSatisfiesSource/rsync_complete_->_skip === RUN TestDestinationSatisfiesSource/in-sync_(fast_path)_->_skip === RUN TestDestinationSatisfiesSource/newer_smaller_dest,_older_larger_source_->_skip --- PASS: TestDestinationSatisfiesSource (0.05s) --- PASS: TestDestinationSatisfiesSource/nil_existing (0.00s) --- PASS: TestDestinationSatisfiesSource/identical_complete_->_skip (0.03s) --- PASS: TestDestinationSatisfiesSource/truncated_(no_md5)_->_replicate (0.00s) --- PASS: TestDestinationSatisfiesSource/md5-backed_truncated_->_replicate (0.00s) --- PASS: TestDestinationSatisfiesSource/md5_equal_and_complete_->_skip (0.00s) --- PASS: TestDestinationSatisfiesSource/complete,_dest_newer_->_skip (0.00s) --- PASS: TestDestinationSatisfiesSource/complete,_dest_older_->_replicate (0.00s) --- PASS: TestDestinationSatisfiesSource/older,_stale_backup_chunk_->_replicate (0.00s) --- PASS: TestDestinationSatisfiesSource/equal-mtime_stale_->_skip_(deferred) (0.00s) --- PASS: TestDestinationSatisfiesSource/rsync_complete_->_skip (0.00s) --- PASS: TestDestinationSatisfiesSource/in-sync_(fast_path)_->_skip (0.00s) --- PASS: TestDestinationSatisfiesSource/newer_smaller_dest,_older_larger_source_->_skip (0.00s) === RUN TestDestinationMatchesReference === RUN TestDestinationMatchesReference/both_empty === RUN TestDestinationMatchesReference/exact_in_order === RUN TestDestinationMatchesReference/count_mismatch === RUN TestDestinationMatchesReference/reordered === RUN TestDestinationMatchesReference/stale_identity === RUN TestDestinationMatchesReference/out-of-band_chunk === RUN TestDestinationMatchesReference/shifted_offset,_same_identity === RUN TestDestinationMatchesReference/larger_size,_same_identity --- PASS: TestDestinationMatchesReference (0.00s) --- PASS: TestDestinationMatchesReference/both_empty (0.00s) --- PASS: TestDestinationMatchesReference/exact_in_order (0.00s) --- PASS: TestDestinationMatchesReference/count_mismatch (0.00s) --- PASS: TestDestinationMatchesReference/reordered (0.00s) --- PASS: TestDestinationMatchesReference/stale_identity (0.00s) --- PASS: TestDestinationMatchesReference/out-of-band_chunk (0.00s) --- PASS: TestDestinationMatchesReference/shifted_offset,_same_identity (0.00s) --- PASS: TestDestinationMatchesReference/larger_size,_same_identity (0.00s) === RUN TestHasStaleBackupChunk === RUN TestHasStaleBackupChunk/nil_existing === RUN TestHasStaleBackupChunk/in_sync === RUN TestHasStaleBackupChunk/stale_backup_chunk === RUN TestHasStaleBackupChunk/rsync_chunks_ignored === RUN TestHasStaleBackupChunk/synced_backup_+_rsync === RUN TestHasStaleBackupChunk/stale_backup_among_rsync --- PASS: TestHasStaleBackupChunk (0.00s) --- PASS: TestHasStaleBackupChunk/nil_existing (0.00s) --- PASS: TestHasStaleBackupChunk/in_sync (0.00s) --- PASS: TestHasStaleBackupChunk/stale_backup_chunk (0.00s) --- PASS: TestHasStaleBackupChunk/rsync_chunks_ignored (0.00s) --- PASS: TestHasStaleBackupChunk/synced_backup_+_rsync (0.00s) --- PASS: TestHasStaleBackupChunk/stale_backup_among_rsync (0.00s) === RUN TestUpdatedEntryKey === RUN TestUpdatedEntryKey/content_update,_no_new_parent === RUN TestUpdatedEntryKey/rename_same_dir === RUN TestUpdatedEntryKey/rename_to_subdir === RUN TestUpdatedEntryKey/root_parent --- PASS: TestUpdatedEntryKey (0.00s) --- PASS: TestUpdatedEntryKey/content_update,_no_new_parent (0.00s) --- PASS: TestUpdatedEntryKey/rename_same_dir (0.00s) --- PASS: TestUpdatedEntryKey/rename_to_subdir (0.00s) --- PASS: TestUpdatedEntryKey/root_parent (0.00s) PASS ok github.com/seaweedfs/seaweedfs/weed/replication/sink/filersink 15.406s ? github.com/seaweedfs/seaweedfs/weed/replication/sink/gcssink [no test files] === RUN TestSanitizeFsKeyKeepsColons --- PASS: TestSanitizeFsKeyKeepsColons (0.00s) === RUN TestCreateEntry_OverwriteReadOnlyFile --- PASS: TestCreateEntry_OverwriteReadOnlyFile (0.04s) PASS ok github.com/seaweedfs/seaweedfs/weed/replication/sink/localsink 31.044s === RUN TestBuildTaggingString_ShouldStripTagPrefix --- PASS: TestBuildTaggingString_ShouldStripTagPrefix (0.02s) === RUN TestBuildTaggingString_ShouldURLEncodeValues --- PASS: TestBuildTaggingString_ShouldURLEncodeValues (0.00s) === RUN TestS3SinkCreateEntryPassesMimeAsContentType --- PASS: TestS3SinkCreateEntryPassesMimeAsContentType (0.09s) === RUN TestS3SinkCreateEntryOmitsContentTypeWhenMimeMissing --- PASS: TestS3SinkCreateEntryOmitsContentTypeWhenMimeMissing (0.01s) === RUN TestBuildTaggingString_EmptyWhenNoTags --- PASS: TestBuildTaggingString_EmptyWhenNoTags (0.00s) PASS ok github.com/seaweedfs/seaweedfs/weed/replication/sink/s3sink 31.140s === RUN TestDownloadFile_NoOffset --- PASS: TestDownloadFile_NoOffset (0.14s) === RUN TestDownloadFile_WithOffset --- PASS: TestDownloadFile_WithOffset (0.06s) === RUN TestDownloadFile_RejectsIgnoredRange --- PASS: TestDownloadFile_RejectsIgnoredRange (0.04s) === RUN TestDownloadFile_ContentDisposition --- PASS: TestDownloadFile_ContentDisposition (0.01s) === RUN TestDownloadFile_PartialReadThenResume --- PASS: TestDownloadFile_PartialReadThenResume (0.03s) === RUN TestDownloadFile_GzipPartialReadThenResume --- PASS: TestDownloadFile_GzipPartialReadThenResume (0.13s) PASS ok github.com/seaweedfs/seaweedfs/weed/replication/source 15.199s === RUN TestKafkaProgressHoldsOffsetAtFailure E0824 22:18:25.781980 notification_kafka.go:201 replicate kafka test partition 0 offset 11 failed; holding the committed offset before it --- PASS: TestKafkaProgressHoldsOffsetAtFailure (0.07s) === RUN TestKafkaProgressKeepsOldestFailure E0824 22:18:25.791604 notification_kafka.go:201 replicate kafka test partition 0 offset 20 failed; holding the committed offset before it E0824 22:18:25.791646 notification_kafka.go:201 replicate kafka test partition 0 offset 11 failed; holding the committed offset before it --- PASS: TestKafkaProgressKeepsOldestFailure (0.00s) PASS ok github.com/seaweedfs/seaweedfs/weed/replication/sub 16.695s === RUN TestAccountForUnscopedIdentity --- PASS: TestAccountForUnscopedIdentity (0.17s) === RUN TestUnscopedIdentitiesGetDistinctAccounts I0824 22:18:40.438901 config_loader.go:73 Using explicit credential store: memory --- PASS: TestUnscopedIdentitiesGetDistinctAccounts (0.17s) === RUN TestCheckAccessByOwnershipDeniesNonOwner --- PASS: TestCheckAccessByOwnershipDeniesNonOwner (0.07s) === RUN TestUnscopedIdentityAccountResolvesByName I0824 22:18:40.612551 config_loader.go:73 Using explicit credential store: memory --- PASS: TestUnscopedIdentityAccountResolvesByName (0.01s) === RUN TestUnscopedIdentityReusesConfiguredAccount I0824 22:18:40.618776 config_loader.go:73 Using explicit credential store: memory --- PASS: TestUnscopedIdentityReusesConfiguredAccount (0.00s) === RUN TestInlineIdentityAccountIsRegistered I0824 22:18:40.624611 config_loader.go:73 Using explicit credential store: memory --- PASS: TestInlineIdentityAccountIsRegistered (0.00s) === RUN TestInlineIdentityAccountIsRegisteredOnUpsert I0824 22:18:40.629768 config_loader.go:73 Using explicit credential store: memory --- PASS: TestInlineIdentityAccountIsRegisteredOnUpsert (0.02s) === RUN TestInlineAccountEmailClaimAndHandoff I0824 22:18:40.647147 config_loader.go:73 Using explicit credential store: memory --- PASS: TestInlineAccountEmailClaimAndHandoff (0.00s) === RUN TestDeclaredAccountOutranksInlineIdentityAccount I0824 22:18:40.651871 config_loader.go:73 Using explicit credential store: memory --- PASS: TestDeclaredAccountOutranksInlineIdentityAccount (0.00s) === RUN TestAuditRequesterArnForSTSSession --- PASS: TestAuditRequesterArnForSTSSession (0.03s) === RUN TestAuditRequesterArnForJWTIdentity --- PASS: TestAuditRequesterArnForJWTIdentity (0.00s) === RUN TestAuditRequesterForAssumeRole --- PASS: TestAuditRequesterForAssumeRole (0.14s) === RUN TestAuthorizeCopySource_AuthDisabled --- PASS: TestAuthorizeCopySource_AuthDisabled (0.02s) === RUN TestAuthorizeCopySource_NilIdentityDenies --- PASS: TestAuthorizeCopySource_NilIdentityDenies (0.00s) === RUN TestAuthorizeCopySource_AdminAllowed --- PASS: TestAuthorizeCopySource_AdminAllowed (0.00s) === RUN TestAuthorizeCopySource_PrefixScopedIdentity === RUN TestAuthorizeCopySource_PrefixScopedIdentity/source_within_scope_is_allowed === RUN TestAuthorizeCopySource_PrefixScopedIdentity/source_outside_scope_is_denied === RUN TestAuthorizeCopySource_PrefixScopedIdentity/source_in_different_bucket_is_denied --- PASS: TestAuthorizeCopySource_PrefixScopedIdentity (0.02s) --- PASS: TestAuthorizeCopySource_PrefixScopedIdentity/source_within_scope_is_allowed (0.02s) --- PASS: TestAuthorizeCopySource_PrefixScopedIdentity/source_outside_scope_is_denied (0.00s) --- PASS: TestAuthorizeCopySource_PrefixScopedIdentity/source_in_different_bucket_is_denied (0.00s) === RUN TestAuthorizeCopySource_IAMIntegrationGetsSourceResource --- PASS: TestAuthorizeCopySource_IAMIntegrationGetsSourceResource (0.00s) === RUN TestAuthorizeCopySource_IAMIntegrationAllow --- PASS: TestAuthorizeCopySource_IAMIntegrationAllow (0.00s) === RUN TestAuthorizeCopySource_VersionIdPropagated --- PASS: TestAuthorizeCopySource_VersionIdPropagated (0.00s) === RUN TestAuthorizeCopySource_PresignedURLSessionTokenPreserved --- PASS: TestAuthorizeCopySource_PresignedURLSessionTokenPreserved (0.02s) === RUN TestAuthorizeCopySource_PreservesCopySourceHeader --- PASS: TestAuthorizeCopySource_PreservesCopySourceHeader (0.00s) === RUN TestLoadAdvancedIAMConfigPolicyDocument --- PASS: TestLoadAdvancedIAMConfigPolicyDocument (0.02s) === RUN TestNormalizeAdvancedIAMPolicies === RUN TestNormalizeAdvancedIAMPolicies/document_object === RUN TestNormalizeAdvancedIAMPolicies/document_already_encoded_as_a_string === RUN TestNormalizeAdvancedIAMPolicies/content_wins_over_document === RUN TestNormalizeAdvancedIAMPolicies/content_only --- PASS: TestNormalizeAdvancedIAMPolicies (0.01s) --- PASS: TestNormalizeAdvancedIAMPolicies/document_object (0.00s) --- PASS: TestNormalizeAdvancedIAMPolicies/document_already_encoded_as_a_string (0.00s) --- PASS: TestNormalizeAdvancedIAMPolicies/content_wins_over_document (0.00s) --- PASS: TestNormalizeAdvancedIAMPolicies/content_only (0.00s) === RUN TestNormalizeAdvancedIAMPoliciesLeavesOtherConfigsAlone --- PASS: TestNormalizeAdvancedIAMPoliciesLeavesOtherConfigsAlone (0.00s) === RUN TestPutPolicy_SyncsToIAMManager --- PASS: TestPutPolicy_SyncsToIAMManager (0.02s) === RUN TestDeletePolicy_SyncsToIAMManager --- PASS: TestDeletePolicy_SyncsToIAMManager (0.01s) === RUN TestSetIAMIntegration_FlushesLoadedPolicies --- PASS: TestSetIAMIntegration_FlushesLoadedPolicies (0.00s) === RUN TestResyncIAMManager_ReflectsCurrentPolicies --- PASS: TestResyncIAMManager_ReflectsCurrentPolicies (0.00s) === RUN TestResync_SkipsUnparsablePolicy W0824 22:18:40.985093 iam_manager.go:474 skipping unparsable runtime policy "broken": unexpected end of JSON input --- PASS: TestResync_SkipsUnparsablePolicy (0.03s) === RUN TestResync_UnparsablePolicyStopsGranting W0824 22:18:40.998604 iam_manager.go:474 skipping unparsable runtime policy "good": invalid character 'n' looking for beginning of object key string --- PASS: TestResync_UnparsablePolicyStopsGranting (0.00s) === RUN TestIamConfigWithoutIdentitiesIsNotStatic --- PASS: TestIamConfigWithoutIdentitiesIsNotStatic (0.04s) === RUN TestConfigWithIdentitiesStillLiveReloadsDynamic --- PASS: TestConfigWithIdentitiesStillLiveReloadsDynamic (0.00s) === RUN TestUpsertIdentityKeepsOtherDynamicIdentities --- PASS: TestUpsertIdentityKeepsOtherDynamicIdentities (0.00s) === RUN TestReloadStaticConfigMarksNewIdentitiesWithoutFreezingDynamic --- PASS: TestReloadStaticConfigMarksNewIdentitiesWithoutFreezingDynamic (0.01s) === RUN TestReloadStaticConfigUpdatesExistingSecretKey --- PASS: TestReloadStaticConfigUpdatesExistingSecretKey (0.00s) === RUN TestReloadStaticConfigUpdatesServiceAccountSecret --- PASS: TestReloadStaticConfigUpdatesServiceAccountSecret (0.01s) === RUN TestFullStateMergeReconcilesPoliciesAndGroups W0824 22:18:41.088747 auth_credentials.go:661 ignoring 1 groups in static config file: groups are managed via the IAM API W0824 22:18:41.089836 auth_credentials.go:3053 IAM policy cache rebuild: skipping invalid policy "file-policy": invalid policy: failed to parse policy JSON: Statement must be an array or a single object W0824 22:18:41.090367 auth_credentials.go:3053 IAM policy cache rebuild: skipping invalid policy "file-policy": invalid policy: failed to parse policy JSON: Statement must be an array or a single object W0824 22:18:41.090507 auth_credentials.go:3053 IAM policy cache rebuild: skipping invalid policy "dynamic-policy": invalid policy: failed to parse policy JSON: Statement must be an array or a single object W0824 22:18:41.090608 auth_credentials.go:3053 IAM policy cache rebuild: skipping invalid policy "file-policy": invalid policy: failed to parse policy JSON: Statement must be an array or a single object W0824 22:18:41.090769 auth_credentials.go:3053 IAM policy cache rebuild: skipping invalid policy "dynamic-policy": invalid policy: failed to parse policy JSON: Statement must be an array or a single object W0824 22:18:41.091897 auth_credentials.go:661 ignoring 1 groups in static config file: groups are managed via the IAM API W0824 22:18:41.092391 auth_credentials.go:3053 IAM policy cache rebuild: skipping invalid policy "file-policy": invalid policy: failed to parse policy JSON: Statement must be an array or a single object W0824 22:18:41.092623 auth_credentials.go:3053 IAM policy cache rebuild: skipping invalid policy "dynamic-policy": invalid policy: failed to parse policy JSON: Statement must be an array or a single object W0824 22:18:41.092806 auth_credentials.go:3053 IAM policy cache rebuild: skipping invalid policy "file-policy": invalid policy: failed to parse policy JSON: Statement must be an array or a single object --- PASS: TestFullStateMergeReconcilesPoliciesAndGroups (0.02s) === RUN TestFailedReloadRetriesUntilSuccess E0824 22:18:41.094855 auth_credentials.go:2279 Failed to load configuration from credential manager: transient store failure E0824 22:18:41.109911 auth_credentials_subscribe.go:91 failed to reload IAM configuration after change in /etc/iam/identities: failed to load configuration from credential manager: transient store failure E0824 22:18:41.110506 auth_credentials.go:2279 Failed to load configuration from credential manager: transient store failure W0824 22:18:41.111691 auth_credentials.go:475 retrying IAM reload: failed to load configuration from credential manager: transient store failure --- PASS: TestFailedReloadRetriesUntilSuccess (0.05s) === RUN TestStaticConfigExpandsEnvCredentialRefs --- PASS: TestStaticConfigExpandsEnvCredentialRefs (0.01s) === RUN TestStaticConfigDropsUnresolvedEnvCredentialRefs W0824 22:18:41.161445 auth_credentials.go:695 identity anvAdmin: dropping credential ${SEAWEEDFS_S3_MISSING_ACCESS_KEY_ID}, it references an unset environment variable --- PASS: TestStaticConfigDropsUnresolvedEnvCredentialRefs (0.01s) === RUN TestEnvCredentialRefsOnlyApplyToStaticConfig --- PASS: TestEnvCredentialRefsOnlyApplyToStaticConfig (0.02s) === RUN TestStaticConfigKeepsLiteralDollarSigns --- PASS: TestStaticConfigKeepsLiteralDollarSigns (0.01s) === RUN TestStaticConfigDropsEmptyEnvCredentialRefs W0824 22:18:41.209408 auth_credentials.go:695 identity anvAdmin: dropping credential ${SEAWEEDFS_S3_ADMIN_ACCESS_KEY_ID}, it references an unset environment variable --- PASS: TestStaticConfigDropsEmptyEnvCredentialRefs (0.02s) === RUN TestStaticConfigDropsMalformedEnvCredentialRefs === RUN TestStaticConfigDropsMalformedEnvCredentialRefs/${MY-VAR} W0824 22:18:41.229490 auth_credentials.go:695 identity anvAdmin: dropping credential ${MY-VAR}, it references an unset environment variable === RUN TestStaticConfigDropsMalformedEnvCredentialRefs/${1VAR} W0824 22:18:41.236475 auth_credentials.go:695 identity anvAdmin: dropping credential ${1VAR}, it references an unset environment variable === RUN TestStaticConfigDropsMalformedEnvCredentialRefs/${} W0824 22:18:41.242443 auth_credentials.go:695 identity anvAdmin: dropping credential ${}, it references an unset environment variable === RUN TestStaticConfigDropsMalformedEnvCredentialRefs/${UNTERMINATED W0824 22:18:41.250450 auth_credentials.go:695 identity anvAdmin: dropping credential ${UNTERMINATED, it references an unset environment variable === RUN TestStaticConfigDropsMalformedEnvCredentialRefs/${A}${B W0824 22:18:41.259233 auth_credentials.go:695 identity anvAdmin: dropping credential ${A}${B, it references an unset environment variable --- PASS: TestStaticConfigDropsMalformedEnvCredentialRefs (0.05s) --- PASS: TestStaticConfigDropsMalformedEnvCredentialRefs/${MY-VAR} (0.02s) --- PASS: TestStaticConfigDropsMalformedEnvCredentialRefs/${1VAR} (0.01s) --- PASS: TestStaticConfigDropsMalformedEnvCredentialRefs/${} (0.01s) --- PASS: TestStaticConfigDropsMalformedEnvCredentialRefs/${UNTERMINATED (0.01s) --- PASS: TestStaticConfigDropsMalformedEnvCredentialRefs/${A}${B (0.01s) === RUN TestStaticConfigKeepsBracesComingFromTheEnvironment --- PASS: TestStaticConfigKeepsBracesComingFromTheEnvironment (0.01s) === RUN TestOnIamConfigChangeLegacyIdentityDeletionReloadsConfiguration --- PASS: TestOnIamConfigChangeLegacyIdentityDeletionReloadsConfiguration (0.00s) === RUN TestOnIamConfigChangeReloadsOnIamIdentityDirectoryChanges --- PASS: TestOnIamConfigChangeReloadsOnIamIdentityDirectoryChanges (0.00s) === RUN TestJWTStreamingUnsignedAuth I0824 22:18:41.272523 config_loader.go:73 Using explicit credential store: memory --- PASS: TestJWTStreamingUnsignedAuth (0.00s) === RUN TestJWTStreamingUnsignedChunkedReader --- PASS: TestJWTStreamingUnsignedChunkedReader (0.02s) === RUN TestPresignedPutSignedContentTypeWithRawSemicolon --- PASS: TestPresignedPutSignedContentTypeWithRawSemicolon (0.01s) === RUN TestReverseProxySignatureVerification === RUN TestReverseProxySignatureVerification/non-standard_port,_externalUrl_matches_proxy_address I0824 22:18:41.321412 auth_credentials.go:312 S3 signature verification will use external host: "127.0.0.1:42391" (from "http://127.0.0.1:42391") I0824 22:18:41.321762 config_loader.go:73 Using explicit credential store: memory === RUN TestReverseProxySignatureVerification/externalUrl_with_non-standard_port,_client_signs_against_external_host I0824 22:18:41.371922 auth_credentials.go:312 S3 signature verification will use external host: "api.example.com:9000" (from "http://api.example.com:9000") I0824 22:18:41.372056 config_loader.go:73 Using explicit credential store: memory === RUN TestReverseProxySignatureVerification/externalUrl_with_HTTPS_default_port_stripped,_client_signs_without_port I0824 22:18:41.388443 auth_credentials.go:312 S3 signature verification will use external host: "api.example.com" (from "https://api.example.com:443") I0824 22:18:41.388630 config_loader.go:73 Using explicit credential store: memory === RUN TestReverseProxySignatureVerification/externalUrl_with_HTTP_default_port_stripped,_client_signs_without_port I0824 22:18:41.417466 auth_credentials.go:312 S3 signature verification will use external host: "api.example.com" (from "http://api.example.com:80") I0824 22:18:41.417860 config_loader.go:73 Using explicit credential store: memory === RUN TestReverseProxySignatureVerification/proxy_forwards_X-Forwarded-Host_correctly,_no_externalUrl_needed I0824 22:18:41.455267 config_loader.go:73 Using explicit credential store: memory === RUN TestReverseProxySignatureVerification/proxy_without_X-Forwarded-Host,_no_externalUrl:_host_mismatch I0824 22:18:41.492065 config_loader.go:73 Using explicit credential store: memory === RUN TestReverseProxySignatureVerification/nginx_$host/$server_port_forwarding,_backend_on_same_hostname I0824 22:18:41.531677 config_loader.go:73 Using explicit credential store: memory === RUN TestReverseProxySignatureVerification/portless_X-Forwarded-Host_on_default_port,_backend_on_same_hostname I0824 22:18:41.565885 config_loader.go:73 Using explicit credential store: memory === RUN TestReverseProxySignatureVerification/proxy_without_X-Forwarded-Host,_externalUrl_saves_the_day I0824 22:18:41.593841 auth_credentials.go:312 S3 signature verification will use external host: "api.example.com:9000" (from "http://api.example.com:9000") I0824 22:18:41.594239 config_loader.go:73 Using explicit credential store: memory --- PASS: TestReverseProxySignatureVerification (0.30s) --- PASS: TestReverseProxySignatureVerification/non-standard_port,_externalUrl_matches_proxy_address (0.05s) --- PASS: TestReverseProxySignatureVerification/externalUrl_with_non-standard_port,_client_signs_against_external_host (0.02s) --- PASS: TestReverseProxySignatureVerification/externalUrl_with_HTTPS_default_port_stripped,_client_signs_without_port (0.02s) --- PASS: TestReverseProxySignatureVerification/externalUrl_with_HTTP_default_port_stripped,_client_signs_without_port (0.04s) --- PASS: TestReverseProxySignatureVerification/proxy_forwards_X-Forwarded-Host_correctly,_no_externalUrl_needed (0.04s) --- PASS: TestReverseProxySignatureVerification/proxy_without_X-Forwarded-Host,_no_externalUrl:_host_mismatch (0.04s) --- PASS: TestReverseProxySignatureVerification/nginx_$host/$server_port_forwarding,_backend_on_same_hostname (0.03s) --- PASS: TestReverseProxySignatureVerification/portless_X-Forwarded-Host_on_default_port,_backend_on_same_hostname (0.03s) --- PASS: TestReverseProxySignatureVerification/proxy_without_X-Forwarded-Host,_externalUrl_saves_the_day (0.02s) === RUN TestActionScopedToBucket --- PASS: TestActionScopedToBucket (0.00s) === RUN TestReproIssue7912 I0824 22:18:41.613634 config_loader.go:73 Using explicit credential store: memory === RUN TestReproIssue7912/Unknown_access_key W0824 22:18:41.641884 auth_signature_v4.go:255 InvalidAccessKeyId: attempted key 'unknown_key' not found. Available keys: 2, Auth enabled: true === RUN TestReproIssue7912/Positive_test_case:_properly_signed_credentials === RUN TestReproIssue7912/Nil_identity_tests_for_guards === RUN TestReproIssue7912/AuthSignatureOnly_path === RUN TestReproIssue7912/Wrong_secret_key === RUN TestReproIssue7912/Anonymous_request_to_protected_bucket === RUN TestReproIssue7912/Non-S3_request_should_be_denied === RUN TestReproIssue7912/Any_other_credentials W0824 22:18:41.681508 auth_signature_v4.go:255 InvalidAccessKeyId: attempted key 'some_other_key' not found. Available keys: 2, Auth enabled: true === RUN TestReproIssue7912/Streaming_unsigned_payload_bypass_attempt --- PASS: TestReproIssue7912 (0.07s) --- PASS: TestReproIssue7912/Unknown_access_key (0.03s) --- PASS: TestReproIssue7912/Positive_test_case:_properly_signed_credentials (0.02s) --- PASS: TestReproIssue7912/Nil_identity_tests_for_guards (0.00s) --- PASS: TestReproIssue7912/AuthSignatureOnly_path (0.02s) --- PASS: TestReproIssue7912/Wrong_secret_key (0.00s) --- PASS: TestReproIssue7912/Anonymous_request_to_protected_bucket (0.00s) --- PASS: TestReproIssue7912/Non-S3_request_should_be_denied (0.00s) --- PASS: TestReproIssue7912/Any_other_credentials (0.00s) --- PASS: TestReproIssue7912/Streaming_unsigned_payload_bypass_attempt (0.00s) === RUN TestAnonymousStreamingUnsignedUpload I0824 22:18:41.684971 config_loader.go:73 Using explicit credential store: memory --- PASS: TestAnonymousStreamingUnsignedUpload (0.00s) === RUN TestExternalUrlSignatureVerification === RUN TestExternalUrlSignatureVerification/non-standard_port_with_externalUrl I0824 22:18:41.702847 auth_credentials.go:312 S3 signature verification will use external host: "api.example.com:9000" (from "https://api.example.com:9000") I0824 22:18:41.703194 config_loader.go:73 Using explicit credential store: memory === RUN TestExternalUrlSignatureVerification/HTTPS_default_port_443_with_externalUrl_(port_stripped_by_SDK_and_parseExternalUrlToHost) I0824 22:18:41.708968 auth_credentials.go:312 S3 signature verification will use external host: "api.example.com" (from "https://api.example.com:443") I0824 22:18:41.709213 config_loader.go:73 Using explicit credential store: memory === RUN TestExternalUrlSignatureVerification/HTTPS_without_explicit_port_with_externalUrl I0824 22:18:41.713487 auth_credentials.go:312 S3 signature verification will use external host: "api.example.com" (from "https://api.example.com") I0824 22:18:41.713851 config_loader.go:73 Using explicit credential store: memory === RUN TestExternalUrlSignatureVerification/HTTP_default_port_80_with_externalUrl I0824 22:18:41.720107 auth_credentials.go:312 S3 signature verification will use external host: "api.example.com" (from "http://api.example.com:80") I0824 22:18:41.720319 config_loader.go:73 Using explicit credential store: memory === RUN TestExternalUrlSignatureVerification/without_externalUrl,_internal_host_causes_mismatch I0824 22:18:41.724932 config_loader.go:73 Using explicit credential store: memory === RUN TestExternalUrlSignatureVerification/without_externalUrl,_matching_host_works I0824 22:18:41.727325 config_loader.go:73 Using explicit credential store: memory --- PASS: TestExternalUrlSignatureVerification (0.04s) --- PASS: TestExternalUrlSignatureVerification/non-standard_port_with_externalUrl (0.02s) --- PASS: TestExternalUrlSignatureVerification/HTTPS_default_port_443_with_externalUrl_(port_stripped_by_SDK_and_parseExternalUrlToHost) (0.00s) --- PASS: TestExternalUrlSignatureVerification/HTTPS_without_explicit_port_with_externalUrl (0.01s) --- PASS: TestExternalUrlSignatureVerification/HTTP_default_port_80_with_externalUrl (0.00s) --- PASS: TestExternalUrlSignatureVerification/without_externalUrl,_internal_host_causes_mismatch (0.00s) --- PASS: TestExternalUrlSignatureVerification/without_externalUrl,_matching_host_works (0.00s) === RUN TestRealSDKSignerWithForwardedHeaders === RUN TestRealSDKSignerWithForwardedHeaders/HTTPS_non-standard_port I0824 22:18:41.729355 config_loader.go:73 Using explicit credential store: memory === RUN TestRealSDKSignerWithForwardedHeaders/HTTPS_standard_port_443_(SDK_strips,_we_must_too) I0824 22:18:41.731371 config_loader.go:73 Using explicit credential store: memory === RUN TestRealSDKSignerWithForwardedHeaders/HTTP_standard_port_80_(SDK_strips,_we_must_too) I0824 22:18:41.732703 config_loader.go:73 Using explicit credential store: memory === RUN TestRealSDKSignerWithForwardedHeaders/HTTP_non-standard_port_8080 I0824 22:18:41.734499 config_loader.go:73 Using explicit credential store: memory === RUN TestRealSDKSignerWithForwardedHeaders/X-Forwarded-Host_with_port_(Traefik_style),_HTTPS_443 I0824 22:18:41.736108 config_loader.go:73 Using explicit credential store: memory === RUN TestRealSDKSignerWithForwardedHeaders/X-Forwarded-Host_with_port_(Traefik_style),_HTTP_80 I0824 22:18:41.737498 config_loader.go:73 Using explicit credential store: memory === RUN TestRealSDKSignerWithForwardedHeaders/no_forwarded_headers,_direct_access I0824 22:18:41.738861 config_loader.go:73 Using explicit credential store: memory === RUN TestRealSDKSignerWithForwardedHeaders/empty_proto_with_port_80_(defaults_to_http,_strip) I0824 22:18:41.740164 config_loader.go:73 Using explicit credential store: memory --- PASS: TestRealSDKSignerWithForwardedHeaders (0.01s) --- PASS: TestRealSDKSignerWithForwardedHeaders/HTTPS_non-standard_port (0.00s) --- PASS: TestRealSDKSignerWithForwardedHeaders/HTTPS_standard_port_443_(SDK_strips,_we_must_too) (0.00s) --- PASS: TestRealSDKSignerWithForwardedHeaders/HTTP_standard_port_80_(SDK_strips,_we_must_too) (0.00s) --- PASS: TestRealSDKSignerWithForwardedHeaders/HTTP_non-standard_port_8080 (0.00s) --- PASS: TestRealSDKSignerWithForwardedHeaders/X-Forwarded-Host_with_port_(Traefik_style),_HTTPS_443 (0.00s) --- PASS: TestRealSDKSignerWithForwardedHeaders/X-Forwarded-Host_with_port_(Traefik_style),_HTTP_80 (0.00s) --- PASS: TestRealSDKSignerWithForwardedHeaders/no_forwarded_headers,_direct_access (0.00s) --- PASS: TestRealSDKSignerWithForwardedHeaders/empty_proto_with_port_80_(defaults_to_http,_strip) (0.00s) === RUN TestValidateV2AuthHeader === RUN TestValidateV2AuthHeader/valid_auth_header_with_space === RUN TestValidateV2AuthHeader/empty_auth_header === RUN TestValidateV2AuthHeader/wrong_algorithm_prefix === RUN TestValidateV2AuthHeader/algorithm_prefix_without_space === RUN TestValidateV2AuthHeader/missing_colon_separator === RUN TestValidateV2AuthHeader/empty_access_key === RUN TestValidateV2AuthHeader/empty_signature --- PASS: TestValidateV2AuthHeader (0.00s) --- PASS: TestValidateV2AuthHeader/valid_auth_header_with_space (0.00s) --- PASS: TestValidateV2AuthHeader/empty_auth_header (0.00s) --- PASS: TestValidateV2AuthHeader/wrong_algorithm_prefix (0.00s) --- PASS: TestValidateV2AuthHeader/algorithm_prefix_without_space (0.00s) --- PASS: TestValidateV2AuthHeader/missing_colon_separator (0.00s) --- PASS: TestValidateV2AuthHeader/empty_access_key (0.00s) --- PASS: TestValidateV2AuthHeader/empty_signature (0.00s) === RUN TestSignatureV2Format --- PASS: TestSignatureV2Format (0.00s) === RUN TestDoesSignV2Match === RUN TestDoesSignV2Match/valid_GET_request === RUN TestDoesSignV2Match/valid_PUT_request_with_content_headers === RUN TestDoesSignV2Match/request_with_query_parameters === RUN TestDoesSignV2Match/request_with_x-amz_headers === RUN TestDoesSignV2Match/invalid_signature === RUN TestDoesSignV2Match/non-existent_access_key W0824 22:18:41.744159 auth_signature_v2.go:137 InvalidAccessKeyId (V2 signed): attempted key 'NONEXISTENTKEY' not found. Available keys: 1, Auth enabled: false === RUN TestDoesSignV2Match/empty_authorization_header === RUN TestDoesSignV2Match/malformed_auth_-_missing_signature === RUN TestDoesSignV2Match/malformed_auth_-_wrong_prefix === RUN TestDoesSignV2Match/malformed_auth_-_no_space_after_AWS --- PASS: TestDoesSignV2Match (0.00s) --- PASS: TestDoesSignV2Match/valid_GET_request (0.00s) --- PASS: TestDoesSignV2Match/valid_PUT_request_with_content_headers (0.00s) --- PASS: TestDoesSignV2Match/request_with_query_parameters (0.00s) --- PASS: TestDoesSignV2Match/request_with_x-amz_headers (0.00s) --- PASS: TestDoesSignV2Match/invalid_signature (0.00s) --- PASS: TestDoesSignV2Match/non-existent_access_key (0.00s) --- PASS: TestDoesSignV2Match/empty_authorization_header (0.00s) --- PASS: TestDoesSignV2Match/malformed_auth_-_missing_signature (0.00s) --- PASS: TestDoesSignV2Match/malformed_auth_-_wrong_prefix (0.00s) --- PASS: TestDoesSignV2Match/malformed_auth_-_no_space_after_AWS (0.00s) === RUN TestDoesSignV2MatchUnicodePath === RUN TestDoesSignV2MatchUnicodePath/CJK_conversation_key_from_ai-dial-core === RUN TestDoesSignV2MatchUnicodePath/Japanese_filename_from_issue_8598 --- PASS: TestDoesSignV2MatchUnicodePath (0.00s) --- PASS: TestDoesSignV2MatchUnicodePath/CJK_conversation_key_from_ai-dial-core (0.00s) --- PASS: TestDoesSignV2MatchUnicodePath/Japanese_filename_from_issue_8598 (0.00s) === RUN TestDoesPresignV2MatchUnicodePath --- PASS: TestDoesPresignV2MatchUnicodePath (0.00s) === RUN TestValidateSTSSessionTokenAssignsDistinctAccount === RUN TestValidateSTSSessionTokenAssignsDistinctAccount/prefers_the_subject === RUN TestValidateSTSSessionTokenAssignsDistinctAccount/falls_back_to_the_assumed-role_user --- PASS: TestValidateSTSSessionTokenAssignsDistinctAccount (0.00s) --- PASS: TestValidateSTSSessionTokenAssignsDistinctAccount/prefers_the_subject (0.00s) --- PASS: TestValidateSTSSessionTokenAssignsDistinctAccount/falls_back_to_the_assumed-role_user (0.00s) === RUN TestVerifyV4SignatureWithSTSIdentity === RUN TestVerifyV4SignatureWithSTSIdentity/STS_identity_with_IAM_integration_-_should_authorize_via_IAM === RUN TestVerifyV4SignatureWithSTSIdentity/STS_identity_with_IAM_integration_-_IAM_denies === RUN TestVerifyV4SignatureWithSTSIdentity/STS_identity_without_IAM_integration_-_should_deny === RUN TestVerifyV4SignatureWithSTSIdentity/Traditional_identity_with_Actions_-_should_use_CanDo === RUN TestVerifyV4SignatureWithSTSIdentity/Traditional_identity_with_Actions_-_CanDo_denies === RUN TestVerifyV4SignatureWithSTSIdentity/shouldCheckPermissions_false_-_skip_authorization --- PASS: TestVerifyV4SignatureWithSTSIdentity (0.00s) --- PASS: TestVerifyV4SignatureWithSTSIdentity/STS_identity_with_IAM_integration_-_should_authorize_via_IAM (0.00s) --- PASS: TestVerifyV4SignatureWithSTSIdentity/STS_identity_with_IAM_integration_-_IAM_denies (0.00s) --- PASS: TestVerifyV4SignatureWithSTSIdentity/STS_identity_without_IAM_integration_-_should_deny (0.00s) --- PASS: TestVerifyV4SignatureWithSTSIdentity/Traditional_identity_with_Actions_-_should_use_CanDo (0.00s) --- PASS: TestVerifyV4SignatureWithSTSIdentity/Traditional_identity_with_Actions_-_CanDo_denies (0.00s) --- PASS: TestVerifyV4SignatureWithSTSIdentity/shouldCheckPermissions_false_-_skip_authorization (0.00s) === RUN TestVerifyV4SignatureSTSStreamingUpload --- PASS: TestVerifyV4SignatureSTSStreamingUpload (0.00s) === RUN TestVerifyV4SignatureActionDetermination === RUN TestVerifyV4SignatureActionDetermination/GET === RUN TestVerifyV4SignatureActionDetermination/HEAD === RUN TestVerifyV4SignatureActionDetermination/PUT === RUN TestVerifyV4SignatureActionDetermination/POST === RUN TestVerifyV4SignatureActionDetermination/DELETE --- PASS: TestVerifyV4SignatureActionDetermination (0.00s) --- PASS: TestVerifyV4SignatureActionDetermination/GET (0.00s) --- PASS: TestVerifyV4SignatureActionDetermination/HEAD (0.00s) --- PASS: TestVerifyV4SignatureActionDetermination/PUT (0.00s) --- PASS: TestVerifyV4SignatureActionDetermination/POST (0.00s) --- PASS: TestVerifyV4SignatureActionDetermination/DELETE (0.00s) === RUN TestExtractV4AuthInfoFromHeader_S3Tables === RUN TestExtractV4AuthInfoFromHeader_S3Tables/s3_service_should_not_auto-hash === RUN TestExtractV4AuthInfoFromHeader_S3Tables/s3tables_service_should_auto-hash --- PASS: TestExtractV4AuthInfoFromHeader_S3Tables (0.00s) --- PASS: TestExtractV4AuthInfoFromHeader_S3Tables/s3_service_should_not_auto-hash (0.00s) --- PASS: TestExtractV4AuthInfoFromHeader_S3Tables/s3tables_service_should_auto-hash (0.00s) === RUN TestParseSignedHeaderRejectsEmptyHeaderNames --- PASS: TestParseSignedHeaderRejectsEmptyHeaderNames (0.00s) === RUN TestExtractV4AuthInfoFromQueryRejectsEmptySignedHeaderNames === RUN TestExtractV4AuthInfoFromQueryRejectsEmptySignedHeaderNames/host; === RUN TestExtractV4AuthInfoFromQueryRejectsEmptySignedHeaderNames/;host === RUN TestExtractV4AuthInfoFromQueryRejectsEmptySignedHeaderNames/host;;x-amz-date --- PASS: TestExtractV4AuthInfoFromQueryRejectsEmptySignedHeaderNames (0.00s) --- PASS: TestExtractV4AuthInfoFromQueryRejectsEmptySignedHeaderNames/host; (0.00s) --- PASS: TestExtractV4AuthInfoFromQueryRejectsEmptySignedHeaderNames/;host (0.00s) --- PASS: TestExtractV4AuthInfoFromQueryRejectsEmptySignedHeaderNames/host;;x-amz-date (0.00s) === RUN TestGetCanonicalQueryString === RUN TestGetCanonicalQueryString/sorts_repeated_values === RUN TestGetCanonicalQueryString/removes_presigned_signature === RUN TestGetCanonicalQueryString/sorts_mixed_single_and_repeated_parameters === RUN TestGetCanonicalQueryString/encodes_query_parameter_values === RUN TestGetCanonicalQueryString/empty_query_string --- PASS: TestGetCanonicalQueryString (0.00s) --- PASS: TestGetCanonicalQueryString/sorts_repeated_values (0.00s) --- PASS: TestGetCanonicalQueryString/removes_presigned_signature (0.00s) --- PASS: TestGetCanonicalQueryString/sorts_mixed_single_and_repeated_parameters (0.00s) --- PASS: TestGetCanonicalQueryString/encodes_query_parameter_values (0.00s) --- PASS: TestGetCanonicalQueryString/empty_query_string (0.00s) === RUN TestBuildPathWithForwardedPrefix === RUN TestBuildPathWithForwardedPrefix/empty_prefix_returns_urlPath === RUN TestBuildPathWithForwardedPrefix/prefix_without_trailing_slash === RUN TestBuildPathWithForwardedPrefix/prefix_with_trailing_slash === RUN TestBuildPathWithForwardedPrefix/prefix_without_leading_slash === RUN TestBuildPathWithForwardedPrefix/prefix_without_leading_slash_and_with_trailing_slash === RUN TestBuildPathWithForwardedPrefix/preserve_double_slashes_in_key === RUN TestBuildPathWithForwardedPrefix/preserve_trailing_slash_in_urlPath === RUN TestBuildPathWithForwardedPrefix/preserve_trailing_slash_with_prefix_having_trailing_slash === RUN TestBuildPathWithForwardedPrefix/root_path === RUN TestBuildPathWithForwardedPrefix/complex_key_with_multiple_slashes === RUN TestBuildPathWithForwardedPrefix/urlPath_without_leading_slash === RUN TestBuildPathWithForwardedPrefix/empty_urlPath_with_prefix --- PASS: TestBuildPathWithForwardedPrefix (0.00s) --- PASS: TestBuildPathWithForwardedPrefix/empty_prefix_returns_urlPath (0.00s) --- PASS: TestBuildPathWithForwardedPrefix/prefix_without_trailing_slash (0.00s) --- PASS: TestBuildPathWithForwardedPrefix/prefix_with_trailing_slash (0.00s) --- PASS: TestBuildPathWithForwardedPrefix/prefix_without_leading_slash (0.00s) --- PASS: TestBuildPathWithForwardedPrefix/prefix_without_leading_slash_and_with_trailing_slash (0.00s) --- PASS: TestBuildPathWithForwardedPrefix/preserve_double_slashes_in_key (0.00s) --- PASS: TestBuildPathWithForwardedPrefix/preserve_trailing_slash_in_urlPath (0.00s) --- PASS: TestBuildPathWithForwardedPrefix/preserve_trailing_slash_with_prefix_having_trailing_slash (0.00s) --- PASS: TestBuildPathWithForwardedPrefix/root_path (0.00s) --- PASS: TestBuildPathWithForwardedPrefix/complex_key_with_multiple_slashes (0.00s) --- PASS: TestBuildPathWithForwardedPrefix/urlPath_without_leading_slash (0.00s) --- PASS: TestBuildPathWithForwardedPrefix/empty_urlPath_with_prefix (0.00s) === RUN TestExtractHostHeader === RUN TestExtractHostHeader/basic_host_without_forwarding === RUN TestExtractHostHeader/host_with_port_without_forwarding === RUN TestExtractHostHeader/X-Forwarded-Host_without_port === RUN TestExtractHostHeader/X-Forwarded-Host_with_X-Forwarded-Port_(HTTP_non-standard) === RUN TestExtractHostHeader/X-Forwarded-Host_with_X-Forwarded-Port_(HTTPS_non-standard) === RUN TestExtractHostHeader/X-Forwarded-Host_with_X-Forwarded-Port_(HTTP_standard_port_80) === RUN TestExtractHostHeader/X-Forwarded-Host_with_X-Forwarded-Port_(HTTPS_standard_port_443) === RUN TestExtractHostHeader/X-Forwarded-Host_with_port_already_included_(should_not_add_port_again) === RUN TestExtractHostHeader/X-Forwarded-Host_with_standard_port_already_included_(HTTPS_443) === RUN TestExtractHostHeader/X-Forwarded-Host_with_port,_no_X-Forwarded-Port_header === RUN TestExtractHostHeader/IPv6_address_with_brackets_and_port_in_X-Forwarded-Host === RUN TestExtractHostHeader/IPv6_address_without_brackets,_should_add_brackets_with_port === RUN TestExtractHostHeader/IPv6_address_without_brackets_and_standard_port,_should_strip_default_port === RUN TestExtractHostHeader/IPv6_address_without_brackets_and_standard_HTTPS_port,_should_strip_default_port === RUN TestExtractHostHeader/IPv6_address_with_brackets_but_no_port,_should_add_port === RUN TestExtractHostHeader/IPv6_full_address_with_brackets_and_default_port_(should_strip_default_port) === RUN TestExtractHostHeader/IPv4-mapped_IPv6_address_without_brackets,_should_add_brackets_with_port === RUN TestExtractHostHeader/Simple_port_442 === RUN TestExtractHostHeader/Port_442_with_X-Forwarded-Host === RUN TestExtractHostHeader/Port_442_with_X-Forwarded-Port === RUN TestExtractHostHeader/HTTPS_with_port_442_(should_NOT_strip) === RUN TestExtractHostHeader/X-Forwarded-Host_with_multiple_hosts_(including_port) === RUN TestExtractHostHeader/IPv6_with_port === RUN TestExtractHostHeader/X-Forwarded-Host_with_port_442,_but_X-Forwarded-Port_is_80_(should_PREFER_442) === RUN TestExtractHostHeader/externalHost_overrides_everything === RUN TestExtractHostHeader/externalHost_overrides_X-Forwarded-Host === RUN TestExtractHostHeader/externalHost_with_IPv6 === RUN TestExtractHostHeader/User_reported_case:_X-Forwarded-Port_misreports_443_but_Host_has_30007 === RUN TestExtractHostHeader/X-Forwarded-Host_already_contains_correct_port,_ignore_misaligned_X-Forwarded-Port === RUN TestExtractHostHeader/X-Forwarded-Host_has_no_port,_match_r.Host_hostname_and_take_its_port --- PASS: TestExtractHostHeader (0.01s) --- PASS: TestExtractHostHeader/basic_host_without_forwarding (0.00s) --- PASS: TestExtractHostHeader/host_with_port_without_forwarding (0.00s) --- PASS: TestExtractHostHeader/X-Forwarded-Host_without_port (0.00s) --- PASS: TestExtractHostHeader/X-Forwarded-Host_with_X-Forwarded-Port_(HTTP_non-standard) (0.00s) --- PASS: TestExtractHostHeader/X-Forwarded-Host_with_X-Forwarded-Port_(HTTPS_non-standard) (0.00s) --- PASS: TestExtractHostHeader/X-Forwarded-Host_with_X-Forwarded-Port_(HTTP_standard_port_80) (0.00s) --- PASS: TestExtractHostHeader/X-Forwarded-Host_with_X-Forwarded-Port_(HTTPS_standard_port_443) (0.00s) --- PASS: TestExtractHostHeader/X-Forwarded-Host_with_port_already_included_(should_not_add_port_again) (0.00s) --- PASS: TestExtractHostHeader/X-Forwarded-Host_with_standard_port_already_included_(HTTPS_443) (0.00s) --- PASS: TestExtractHostHeader/X-Forwarded-Host_with_port,_no_X-Forwarded-Port_header (0.00s) --- PASS: TestExtractHostHeader/IPv6_address_with_brackets_and_port_in_X-Forwarded-Host (0.00s) --- PASS: TestExtractHostHeader/IPv6_address_without_brackets,_should_add_brackets_with_port (0.00s) --- PASS: TestExtractHostHeader/IPv6_address_without_brackets_and_standard_port,_should_strip_default_port (0.00s) --- PASS: TestExtractHostHeader/IPv6_address_without_brackets_and_standard_HTTPS_port,_should_strip_default_port (0.00s) --- PASS: TestExtractHostHeader/IPv6_address_with_brackets_but_no_port,_should_add_port (0.00s) --- PASS: TestExtractHostHeader/IPv6_full_address_with_brackets_and_default_port_(should_strip_default_port) (0.00s) --- PASS: TestExtractHostHeader/IPv4-mapped_IPv6_address_without_brackets,_should_add_brackets_with_port (0.00s) --- PASS: TestExtractHostHeader/Simple_port_442 (0.00s) --- PASS: TestExtractHostHeader/Port_442_with_X-Forwarded-Host (0.00s) --- PASS: TestExtractHostHeader/Port_442_with_X-Forwarded-Port (0.00s) --- PASS: TestExtractHostHeader/HTTPS_with_port_442_(should_NOT_strip) (0.00s) --- PASS: TestExtractHostHeader/X-Forwarded-Host_with_multiple_hosts_(including_port) (0.00s) --- PASS: TestExtractHostHeader/IPv6_with_port (0.00s) --- PASS: TestExtractHostHeader/X-Forwarded-Host_with_port_442,_but_X-Forwarded-Port_is_80_(should_PREFER_442) (0.00s) --- PASS: TestExtractHostHeader/externalHost_overrides_everything (0.00s) --- PASS: TestExtractHostHeader/externalHost_overrides_X-Forwarded-Host (0.00s) --- PASS: TestExtractHostHeader/externalHost_with_IPv6 (0.00s) --- PASS: TestExtractHostHeader/User_reported_case:_X-Forwarded-Port_misreports_443_but_Host_has_30007 (0.00s) --- PASS: TestExtractHostHeader/X-Forwarded-Host_already_contains_correct_port,_ignore_misaligned_X-Forwarded-Port (0.00s) --- PASS: TestExtractHostHeader/X-Forwarded-Host_has_no_port,_match_r.Host_hostname_and_take_its_port (0.00s) === RUN TestExtractHostHeaderCandidates === RUN TestExtractHostHeaderCandidates/externalHost_is_the_only_candidate === RUN TestExtractHostHeaderCandidates/X-Forwarded-Host_with_port_is_trusted_as-is === RUN TestExtractHostHeaderCandidates/plain_Host_with_port_is_the_only_candidate === RUN TestExtractHostHeaderCandidates/portless_X-Forwarded-Host,_hostnames_match:_r.Host_port_first,_then_X-Forwarded-Port,_then_bare === RUN TestExtractHostHeaderCandidates/portless_X-Forwarded-Host,_hostnames_match,_no_X-Forwarded-Port:_bare_host_as_fallback === RUN TestExtractHostHeaderCandidates/portless_X-Forwarded-Host,_hostnames_differ:_X-Forwarded-Port,_then_bare === RUN TestExtractHostHeaderCandidates/default_X-Forwarded-Port_collapses_into_the_bare_candidate === RUN TestExtractHostHeaderCandidates/portless_Host_with_X-Forwarded-Port:_forwarded_port,_then_bare === RUN TestExtractHostHeaderCandidates/bracketed_portless_IPv6_X-Forwarded-Host_matches_the_request_host === RUN TestExtractHostHeaderCandidates/unbracketed_portless_IPv6_X-Forwarded-Host_with_X-Forwarded-Port --- PASS: TestExtractHostHeaderCandidates (0.00s) --- PASS: TestExtractHostHeaderCandidates/externalHost_is_the_only_candidate (0.00s) --- PASS: TestExtractHostHeaderCandidates/X-Forwarded-Host_with_port_is_trusted_as-is (0.00s) --- PASS: TestExtractHostHeaderCandidates/plain_Host_with_port_is_the_only_candidate (0.00s) --- PASS: TestExtractHostHeaderCandidates/portless_X-Forwarded-Host,_hostnames_match:_r.Host_port_first,_then_X-Forwarded-Port,_then_bare (0.00s) --- PASS: TestExtractHostHeaderCandidates/portless_X-Forwarded-Host,_hostnames_match,_no_X-Forwarded-Port:_bare_host_as_fallback (0.00s) --- PASS: TestExtractHostHeaderCandidates/portless_X-Forwarded-Host,_hostnames_differ:_X-Forwarded-Port,_then_bare (0.00s) --- PASS: TestExtractHostHeaderCandidates/default_X-Forwarded-Port_collapses_into_the_bare_candidate (0.00s) --- PASS: TestExtractHostHeaderCandidates/portless_Host_with_X-Forwarded-Port:_forwarded_port,_then_bare (0.00s) --- PASS: TestExtractHostHeaderCandidates/bracketed_portless_IPv6_X-Forwarded-Host_matches_the_request_host (0.00s) --- PASS: TestExtractHostHeaderCandidates/unbracketed_portless_IPv6_X-Forwarded-Host_with_X-Forwarded-Port (0.00s) === RUN TestExtractSignedHeadersCase === RUN TestExtractSignedHeadersCase/lowercase_host === RUN TestExtractSignedHeadersCase/uppercase_Host --- PASS: TestExtractSignedHeadersCase (0.00s) --- PASS: TestExtractSignedHeadersCase/lowercase_host (0.00s) --- PASS: TestExtractSignedHeadersCase/uppercase_Host (0.00s) === RUN TestVerifySignedHeadersCoverage_Unit === RUN TestVerifySignedHeadersCoverage_Unit/no_x-amz_headers_is_fine === RUN TestVerifySignedHeadersCoverage_Unit/signed_x-amz_header_is_accepted === RUN TestVerifySignedHeadersCoverage_Unit/unsigned_x-amz-tagging_is_rejected_(header-based) === RUN TestVerifySignedHeadersCoverage_Unit/unsigned_x-amz-meta-*_is_rejected === RUN TestVerifySignedHeadersCoverage_Unit/unsigned_x-amz-acl_is_rejected === RUN TestVerifySignedHeadersCoverage_Unit/unsigned_x-amz-storage-class_is_rejected === RUN TestVerifySignedHeadersCoverage_Unit/unsigned_x-amz-server-side-encryption_is_rejected === RUN TestVerifySignedHeadersCoverage_Unit/unsigned_x-amz-object-lock-retain-until-date_is_rejected === RUN TestVerifySignedHeadersCoverage_Unit/unsigned_x-amz-security-token_is_rejected_(even_presigned) === RUN TestVerifySignedHeadersCoverage_Unit/x-amz-content-sha256_is_always_exempt_(presigned) === RUN TestVerifySignedHeadersCoverage_Unit/x-amz-content-sha256_is_always_exempt_(header-based) === RUN TestVerifySignedHeadersCoverage_Unit/presigned_exempts_sigv4_query_params_echoed_as_headers === RUN TestVerifySignedHeadersCoverage_Unit/header-based_does_NOT_exempt_x-amz-date_when_unsigned === RUN TestVerifySignedHeadersCoverage_Unit/signed_headers_match_is_case-insensitive --- PASS: TestVerifySignedHeadersCoverage_Unit (0.00s) --- PASS: TestVerifySignedHeadersCoverage_Unit/no_x-amz_headers_is_fine (0.00s) --- PASS: TestVerifySignedHeadersCoverage_Unit/signed_x-amz_header_is_accepted (0.00s) --- PASS: TestVerifySignedHeadersCoverage_Unit/unsigned_x-amz-tagging_is_rejected_(header-based) (0.00s) --- PASS: TestVerifySignedHeadersCoverage_Unit/unsigned_x-amz-meta-*_is_rejected (0.00s) --- PASS: TestVerifySignedHeadersCoverage_Unit/unsigned_x-amz-acl_is_rejected (0.00s) --- PASS: TestVerifySignedHeadersCoverage_Unit/unsigned_x-amz-storage-class_is_rejected (0.00s) --- PASS: TestVerifySignedHeadersCoverage_Unit/unsigned_x-amz-server-side-encryption_is_rejected (0.00s) --- PASS: TestVerifySignedHeadersCoverage_Unit/unsigned_x-amz-object-lock-retain-until-date_is_rejected (0.00s) --- PASS: TestVerifySignedHeadersCoverage_Unit/unsigned_x-amz-security-token_is_rejected_(even_presigned) (0.00s) --- PASS: TestVerifySignedHeadersCoverage_Unit/x-amz-content-sha256_is_always_exempt_(presigned) (0.00s) --- PASS: TestVerifySignedHeadersCoverage_Unit/x-amz-content-sha256_is_always_exempt_(header-based) (0.00s) --- PASS: TestVerifySignedHeadersCoverage_Unit/presigned_exempts_sigv4_query_params_echoed_as_headers (0.00s) --- PASS: TestVerifySignedHeadersCoverage_Unit/header-based_does_NOT_exempt_x-amz-date_when_unsigned (0.00s) --- PASS: TestVerifySignedHeadersCoverage_Unit/signed_headers_match_is_case-insensitive (0.00s) === RUN TestPresignedPutRejectsUnsignedTagging --- PASS: TestPresignedPutRejectsUnsignedTagging (0.00s) === RUN TestPresignedPutAcceptsSignedTagging --- PASS: TestPresignedPutAcceptsSignedTagging (0.00s) === RUN TestPresignedPutRejectsUnsignedMetadataHeaders === RUN TestPresignedPutRejectsUnsignedMetadataHeaders/acl === RUN TestPresignedPutRejectsUnsignedMetadataHeaders/user-metadata === RUN TestPresignedPutRejectsUnsignedMetadataHeaders/storage-class === RUN TestPresignedPutRejectsUnsignedMetadataHeaders/server-side-encryption === RUN TestPresignedPutRejectsUnsignedMetadataHeaders/sse-kms-key-id === RUN TestPresignedPutRejectsUnsignedMetadataHeaders/object-lock-mode === RUN TestPresignedPutRejectsUnsignedMetadataHeaders/object-lock-retain-until === RUN TestPresignedPutRejectsUnsignedMetadataHeaders/website-redirect-location === RUN TestPresignedPutRejectsUnsignedMetadataHeaders/grant-full-control --- PASS: TestPresignedPutRejectsUnsignedMetadataHeaders (0.00s) --- PASS: TestPresignedPutRejectsUnsignedMetadataHeaders/acl (0.00s) --- PASS: TestPresignedPutRejectsUnsignedMetadataHeaders/user-metadata (0.00s) --- PASS: TestPresignedPutRejectsUnsignedMetadataHeaders/storage-class (0.00s) --- PASS: TestPresignedPutRejectsUnsignedMetadataHeaders/server-side-encryption (0.00s) --- PASS: TestPresignedPutRejectsUnsignedMetadataHeaders/sse-kms-key-id (0.00s) --- PASS: TestPresignedPutRejectsUnsignedMetadataHeaders/object-lock-mode (0.00s) --- PASS: TestPresignedPutRejectsUnsignedMetadataHeaders/object-lock-retain-until (0.00s) --- PASS: TestPresignedPutRejectsUnsignedMetadataHeaders/website-redirect-location (0.00s) --- PASS: TestPresignedPutRejectsUnsignedMetadataHeaders/grant-full-control (0.00s) === RUN TestSTSIdentityPolicyNamesPopulation I0824 22:18:41.770131 config_loader.go:73 Using explicit credential store: memory auth_sts_identity_test.go:94: ✓ STS identity will use IAM authorization path (correct behavior) --- PASS: TestSTSIdentityPolicyNamesPopulation (0.05s) === RUN TestSTSIdentityAuthorizationFlow auth_sts_identity_test.go:154: ✓ Verified: STS identity correctly bypasses legacy CanDo() to use IAM authorization path --- PASS: TestSTSIdentityAuthorizationFlow (0.00s) === RUN TestSTSIdentityWithoutPolicyNames auth_sts_identity_test.go:174: Bug scenario: Actions=0, PolicyNames=0 → Would be denied auth_sts_identity_test.go:176: Fixed scenario: Actions=0, PolicyNames=1 → Can use IAM authorization --- PASS: TestSTSIdentityWithoutPolicyNames (0.00s) === RUN TestCanDoPathConstruction === RUN TestCanDoPathConstruction/Wildcard_match_for_write auth_sts_identity_test.go:249: Testing path: test-bucket/path/to/object.txt === RUN TestCanDoPathConstruction/Wildcard_match_for_read_in_docs_path auth_sts_identity_test.go:249: Testing path: test-bucket/docs/file.txt === RUN TestCanDoPathConstruction/No_match_-_wrong_action auth_sts_identity_test.go:249: Testing path: test-bucket/file.txt === RUN TestCanDoPathConstruction/No_match_-_read_outside_docs_path auth_sts_identity_test.go:249: Testing path: test-bucket/other/file.txt === RUN TestCanDoPathConstruction/ObjectKey_with_leading_slash auth_sts_identity_test.go:249: Testing path: test-bucket/docs/file.txt --- PASS: TestCanDoPathConstruction (0.00s) --- PASS: TestCanDoPathConstruction/Wildcard_match_for_write (0.00s) --- PASS: TestCanDoPathConstruction/Wildcard_match_for_read_in_docs_path (0.00s) --- PASS: TestCanDoPathConstruction/No_match_-_wrong_action (0.00s) --- PASS: TestCanDoPathConstruction/No_match_-_read_outside_docs_path (0.00s) --- PASS: TestCanDoPathConstruction/ObjectKey_with_leading_slash (0.00s) === RUN TestValidateSTSSessionTokenIntegration I0824 22:18:41.817859 config_loader.go:73 Using explicit credential store: memory auth_sts_identity_test.go:304: ✓ Integration test passed: STS identity properly configured for IAM authorization --- PASS: TestValidateSTSSessionTokenIntegration (0.00s) === RUN TestSTSIdentityClaimsPopulation I0824 22:18:41.820563 config_loader.go:73 Using explicit credential store: memory auth_sts_identity_test.go:369: ✓ Claims properly populated from RequestContext for policy variable substitution --- PASS: TestSTSIdentityClaimsPopulation (0.00s) === RUN TestAuthorizeWithIAMSessionTokenExtraction === RUN TestAuthorizeWithIAMSessionTokenExtraction/Extracts_X-SeaweedFS-Session-Token_from_JWT_auth === RUN TestAuthorizeWithIAMSessionTokenExtraction/Extracts_X-Amz-Security-Token_from_V4_STS_auth_header === RUN TestAuthorizeWithIAMSessionTokenExtraction/Extracts_X-Amz-Security-Token_from_query_parameter_(presigned_URL) === RUN TestAuthorizeWithIAMSessionTokenExtraction/JWT_token_takes_precedence_over_X-Amz-Security-Token --- PASS: TestAuthorizeWithIAMSessionTokenExtraction (0.03s) --- PASS: TestAuthorizeWithIAMSessionTokenExtraction/Extracts_X-SeaweedFS-Session-Token_from_JWT_auth (0.02s) --- PASS: TestAuthorizeWithIAMSessionTokenExtraction/Extracts_X-Amz-Security-Token_from_V4_STS_auth_header (0.00s) --- PASS: TestAuthorizeWithIAMSessionTokenExtraction/Extracts_X-Amz-Security-Token_from_query_parameter_(presigned_URL) (0.00s) --- PASS: TestAuthorizeWithIAMSessionTokenExtraction/JWT_token_takes_precedence_over_X-Amz-Security-Token (0.00s) === RUN TestSTSSessionTokenIntoCredentials --- PASS: TestSTSSessionTokenIntoCredentials (0.00s) === RUN TestActionConstantsForV4Auth --- PASS: TestActionConstantsForV4Auth (0.00s) === RUN TestIsRequestPresignedSignatureV4 --- PASS: TestIsRequestPresignedSignatureV4 (0.00s) === RUN TestIsReqAuthenticated --- PASS: TestIsReqAuthenticated (0.00s) === RUN TestCheckaAnonymousRequestAuthType --- PASS: TestCheckaAnonymousRequestAuthType (0.00s) === RUN TestAnonymousListBucketsWithPrefixAction --- PASS: TestAnonymousListBucketsWithPrefixAction (0.00s) === RUN TestCheckAdminRequestAuthType --- PASS: TestCheckAdminRequestAuthType (0.00s) === RUN TestSignatureV4WithForwardedPrefix === RUN TestSignatureV4WithForwardedPrefix/prefix_without_trailing_slash === RUN TestSignatureV4WithForwardedPrefix/prefix_with_trailing_slash --- PASS: TestSignatureV4WithForwardedPrefix (0.00s) --- PASS: TestSignatureV4WithForwardedPrefix/prefix_without_trailing_slash (0.00s) --- PASS: TestSignatureV4WithForwardedPrefix/prefix_with_trailing_slash (0.00s) === RUN TestSignatureV4WithForwardedPrefixTrailingSlash === RUN TestSignatureV4WithForwardedPrefixTrailingSlash/bucket_listObjects_with_trailing_slash === RUN TestSignatureV4WithForwardedPrefixTrailingSlash/prefix_path_with_trailing_slash === RUN TestSignatureV4WithForwardedPrefixTrailingSlash/root_bucket_with_trailing_slash === RUN TestSignatureV4WithForwardedPrefixTrailingSlash/nested_folder_with_trailing_slash --- PASS: TestSignatureV4WithForwardedPrefixTrailingSlash (0.00s) --- PASS: TestSignatureV4WithForwardedPrefixTrailingSlash/bucket_listObjects_with_trailing_slash (0.00s) --- PASS: TestSignatureV4WithForwardedPrefixTrailingSlash/prefix_path_with_trailing_slash (0.00s) --- PASS: TestSignatureV4WithForwardedPrefixTrailingSlash/root_bucket_with_trailing_slash (0.00s) --- PASS: TestSignatureV4WithForwardedPrefixTrailingSlash/nested_folder_with_trailing_slash (0.00s) === RUN TestSignatureV4WithoutProxy === RUN TestSignatureV4WithoutProxy/HTTP_with_non-standard_port === RUN TestSignatureV4WithoutProxy/HTTPS_with_non-standard_port === RUN TestSignatureV4WithoutProxy/HTTP_with_standard_port === RUN TestSignatureV4WithoutProxy/HTTPS_with_standard_port === RUN TestSignatureV4WithoutProxy/HTTP_without_port === RUN TestSignatureV4WithoutProxy/HTTPS_without_port === RUN TestSignatureV4WithoutProxy/IPv6_HTTP_with_non-standard_port === RUN TestSignatureV4WithoutProxy/IPv6_HTTPS_with_non-standard_port === RUN TestSignatureV4WithoutProxy/IPv6_HTTP_with_standard_port === RUN TestSignatureV4WithoutProxy/IPv6_HTTPS_with_standard_port === RUN TestSignatureV4WithoutProxy/IPv6_HTTP_without_port === RUN TestSignatureV4WithoutProxy/IPv6_HTTPS_without_port --- PASS: TestSignatureV4WithoutProxy (0.01s) --- PASS: TestSignatureV4WithoutProxy/HTTP_with_non-standard_port (0.00s) --- PASS: TestSignatureV4WithoutProxy/HTTPS_with_non-standard_port (0.00s) --- PASS: TestSignatureV4WithoutProxy/HTTP_with_standard_port (0.00s) --- PASS: TestSignatureV4WithoutProxy/HTTPS_with_standard_port (0.00s) --- PASS: TestSignatureV4WithoutProxy/HTTP_without_port (0.00s) --- PASS: TestSignatureV4WithoutProxy/HTTPS_without_port (0.00s) --- PASS: TestSignatureV4WithoutProxy/IPv6_HTTP_with_non-standard_port (0.00s) --- PASS: TestSignatureV4WithoutProxy/IPv6_HTTPS_with_non-standard_port (0.00s) --- PASS: TestSignatureV4WithoutProxy/IPv6_HTTP_with_standard_port (0.00s) --- PASS: TestSignatureV4WithoutProxy/IPv6_HTTPS_with_standard_port (0.00s) --- PASS: TestSignatureV4WithoutProxy/IPv6_HTTP_without_port (0.00s) --- PASS: TestSignatureV4WithoutProxy/IPv6_HTTPS_without_port (0.00s) === RUN TestSignatureV4WithForwardedPort === RUN TestSignatureV4WithForwardedPort/HTTP_with_non-standard_port === RUN TestSignatureV4WithForwardedPort/HTTPS_with_non-standard_port === RUN TestSignatureV4WithForwardedPort/HTTP_with_standard_port_(80) === RUN TestSignatureV4WithForwardedPort/HTTPS_with_standard_port_(443) === RUN TestSignatureV4WithForwardedPort/empty_proto_with_non-standard_port === RUN TestSignatureV4WithForwardedPort/empty_proto_with_port_80_(scheme_defaults_to_https_from_URL,_so_80_is_NOT_default) === RUN TestSignatureV4WithForwardedPort/X-Forwarded-Host_with_port_already_included_(Traefik/HAProxy_style) === RUN TestSignatureV4WithForwardedPort/X-Forwarded-Host_with_port,_no_X-Forwarded-Port_header === RUN TestSignatureV4WithForwardedPort/X-Forwarded-Host_with_standard_https_port_already_included_(Traefik/HAProxy_style) === RUN TestSignatureV4WithForwardedPort/X-Forwarded-Host_with_standard_http_port_already_included_(Traefik/HAProxy_style) === RUN TestSignatureV4WithForwardedPort/IPv6_X-Forwarded-Host_with_standard_https_port_already_included_(Traefik/HAProxy_style) === RUN TestSignatureV4WithForwardedPort/IPv6_X-Forwarded-Host_with_standard_http_port_already_included_(Traefik/HAProxy_style) === RUN TestSignatureV4WithForwardedPort/IPv6_with_port_in_brackets === RUN TestSignatureV4WithForwardedPort/IPv6_without_port_-_should_add_port_with_brackets === RUN TestSignatureV4WithForwardedPort/IPv6_in_brackets_without_port_-_should_add_port === RUN TestSignatureV4WithForwardedPort/IPv4-mapped_IPv6_without_port_-_should_add_port_with_brackets --- PASS: TestSignatureV4WithForwardedPort (0.01s) --- PASS: TestSignatureV4WithForwardedPort/HTTP_with_non-standard_port (0.00s) --- PASS: TestSignatureV4WithForwardedPort/HTTPS_with_non-standard_port (0.00s) --- PASS: TestSignatureV4WithForwardedPort/HTTP_with_standard_port_(80) (0.00s) --- PASS: TestSignatureV4WithForwardedPort/HTTPS_with_standard_port_(443) (0.00s) --- PASS: TestSignatureV4WithForwardedPort/empty_proto_with_non-standard_port (0.00s) --- PASS: TestSignatureV4WithForwardedPort/empty_proto_with_port_80_(scheme_defaults_to_https_from_URL,_so_80_is_NOT_default) (0.00s) --- PASS: TestSignatureV4WithForwardedPort/X-Forwarded-Host_with_port_already_included_(Traefik/HAProxy_style) (0.00s) --- PASS: TestSignatureV4WithForwardedPort/X-Forwarded-Host_with_port,_no_X-Forwarded-Port_header (0.00s) --- PASS: TestSignatureV4WithForwardedPort/X-Forwarded-Host_with_standard_https_port_already_included_(Traefik/HAProxy_style) (0.00s) --- PASS: TestSignatureV4WithForwardedPort/X-Forwarded-Host_with_standard_http_port_already_included_(Traefik/HAProxy_style) (0.00s) --- PASS: TestSignatureV4WithForwardedPort/IPv6_X-Forwarded-Host_with_standard_https_port_already_included_(Traefik/HAProxy_style) (0.00s) --- PASS: TestSignatureV4WithForwardedPort/IPv6_X-Forwarded-Host_with_standard_http_port_already_included_(Traefik/HAProxy_style) (0.00s) --- PASS: TestSignatureV4WithForwardedPort/IPv6_with_port_in_brackets (0.00s) --- PASS: TestSignatureV4WithForwardedPort/IPv6_without_port_-_should_add_port_with_brackets (0.00s) --- PASS: TestSignatureV4WithForwardedPort/IPv6_in_brackets_without_port_-_should_add_port (0.00s) --- PASS: TestSignatureV4WithForwardedPort/IPv4-mapped_IPv6_without_port_-_should_add_port_with_brackets (0.00s) === RUN TestPresignedSignatureV4Basic --- PASS: TestPresignedSignatureV4Basic (0.00s) === RUN TestPresignedSignatureV4MissingExpires --- PASS: TestPresignedSignatureV4MissingExpires (0.00s) === RUN TestPresignedSignatureV4WithForwardedPrefix === RUN TestPresignedSignatureV4WithForwardedPrefix/prefix_without_trailing_slash === RUN TestPresignedSignatureV4WithForwardedPrefix/prefix_with_trailing_slash --- PASS: TestPresignedSignatureV4WithForwardedPrefix (0.00s) --- PASS: TestPresignedSignatureV4WithForwardedPrefix/prefix_without_trailing_slash (0.00s) --- PASS: TestPresignedSignatureV4WithForwardedPrefix/prefix_with_trailing_slash (0.00s) === RUN TestPresignedSignatureV4WithForwardedPrefixTrailingSlash === RUN TestPresignedSignatureV4WithForwardedPrefixTrailingSlash/bucket_listObjects_with_trailing_slash === RUN TestPresignedSignatureV4WithForwardedPrefixTrailingSlash/prefix_path_with_trailing_slash === RUN TestPresignedSignatureV4WithForwardedPrefixTrailingSlash/api_path_with_trailing_slash --- PASS: TestPresignedSignatureV4WithForwardedPrefixTrailingSlash (0.01s) --- PASS: TestPresignedSignatureV4WithForwardedPrefixTrailingSlash/bucket_listObjects_with_trailing_slash (0.01s) --- PASS: TestPresignedSignatureV4WithForwardedPrefixTrailingSlash/prefix_path_with_trailing_slash (0.00s) --- PASS: TestPresignedSignatureV4WithForwardedPrefixTrailingSlash/api_path_with_trailing_slash (0.00s) === RUN TestGetStringToSignPUT --- PASS: TestGetStringToSignPUT (0.01s) === RUN TestGetStringToSignGETEmptyStringHash --- PASS: TestGetStringToSignGETEmptyStringHash (0.00s) === RUN TestIAMPayloadHashComputation --- PASS: TestIAMPayloadHashComputation (0.00s) === RUN TestS3PayloadHashNoRegression --- PASS: TestS3PayloadHashNoRegression (0.00s) === RUN TestIAMEmptyBodyPayloadHash --- PASS: TestIAMEmptyBodyPayloadHash (0.00s) === RUN TestSTSPayloadHashComputation --- PASS: TestSTSPayloadHashComputation (0.00s) === RUN TestGitHubIssue7080Scenario --- PASS: TestGitHubIssue7080Scenario (0.00s) === RUN TestIAMSignatureServiceMatching --- PASS: TestIAMSignatureServiceMatching (0.01s) === RUN TestStreamingSignatureServiceField --- PASS: TestStreamingSignatureServiceField (0.00s) === RUN TestIAMLargeBodySecurityLimit --- PASS: TestIAMLargeBodySecurityLimit (0.79s) === RUN TestStreamHashRequestBody === RUN TestStreamHashRequestBody/empty_body === RUN TestStreamHashRequestBody/small_payload === RUN TestStreamHashRequestBody/medium_payload === RUN TestStreamHashRequestBody/large_payload_within_limit --- PASS: TestStreamHashRequestBody (0.25s) --- PASS: TestStreamHashRequestBody/empty_body (0.00s) --- PASS: TestStreamHashRequestBody/small_payload (0.16s) --- PASS: TestStreamHashRequestBody/medium_payload (0.00s) --- PASS: TestStreamHashRequestBody/large_payload_within_limit (0.06s) === RUN TestStreamingVsNonStreamingConsistency === RUN TestStreamingVsNonStreamingConsistency/payload_0 === RUN TestStreamingVsNonStreamingConsistency/payload_1 === RUN TestStreamingVsNonStreamingConsistency/payload_2 === RUN TestStreamingVsNonStreamingConsistency/payload_3 === RUN TestStreamingVsNonStreamingConsistency/payload_4 === RUN TestStreamingVsNonStreamingConsistency/payload_5 --- PASS: TestStreamingVsNonStreamingConsistency (0.01s) --- PASS: TestStreamingVsNonStreamingConsistency/payload_0 (0.00s) --- PASS: TestStreamingVsNonStreamingConsistency/payload_1 (0.00s) --- PASS: TestStreamingVsNonStreamingConsistency/payload_2 (0.00s) --- PASS: TestStreamingVsNonStreamingConsistency/payload_3 (0.00s) --- PASS: TestStreamingVsNonStreamingConsistency/payload_4 (0.00s) --- PASS: TestStreamingVsNonStreamingConsistency/payload_5 (0.00s) === RUN TestStreamingWithSizeLimit --- PASS: TestStreamingWithSizeLimit (1.35s) === RUN TestBuildBucketMetadata W0824 22:18:44.329729 bucket_metadata.go:116 Invalid ownership: , bucket: ownershipEmptyStr W0824 22:18:44.330163 bucket_metadata.go:127 owner[id=xxxxx] is invalid, bucket: acpEmptyObject --- PASS: TestBuildBucketMetadata (0.00s) === RUN TestGetBucketMetadata --- PASS: TestGetBucketMetadata (1.01s) === RUN TestBucketOwnerFromIdentityId === RUN TestBucketOwnerFromIdentityId/account-less_identity === RUN TestBucketOwnerFromIdentityId/identity_scoped_to_an_account === RUN TestBucketOwnerFromIdentityId/unknown_identity --- PASS: TestBucketOwnerFromIdentityId (0.00s) --- PASS: TestBucketOwnerFromIdentityId/account-less_identity (0.00s) --- PASS: TestBucketOwnerFromIdentityId/identity_scoped_to_an_account (0.00s) --- PASS: TestBucketOwnerFromIdentityId/unknown_identity (0.00s) === RUN TestBucketOwnerPrefersAccountId --- PASS: TestBucketOwnerPrefersAccountId (0.00s) === RUN TestCollectionInfoLogicalSize === RUN TestCollectionInfoLogicalSize/no_garbage === RUN TestCollectionInfoLogicalSize/some_garbage === RUN TestCollectionInfoLogicalSize/all_garbage === RUN TestCollectionInfoLogicalSize/deleted_exceeds_size_clamps_to_zero --- PASS: TestCollectionInfoLogicalSize (0.01s) --- PASS: TestCollectionInfoLogicalSize/no_garbage (0.00s) --- PASS: TestCollectionInfoLogicalSize/some_garbage (0.00s) --- PASS: TestCollectionInfoLogicalSize/all_garbage (0.00s) --- PASS: TestCollectionInfoLogicalSize/deleted_exceeds_size_clamps_to_zero (0.00s) === RUN TestChunkedEncodingMixedFormat --- PASS: TestChunkedEncodingMixedFormat (0.00s) === RUN TestExtractChecksumAlgorithmIsCaseInsensitive --- PASS: TestExtractChecksumAlgorithmIsCaseInsensitive (0.00s) === RUN TestNewSignV4ChunkedReaderStreamingUnsignedPayloadTrailer --- PASS: TestNewSignV4ChunkedReaderStreamingUnsignedPayloadTrailer (0.00s) === RUN TestSignedStreamingUpload --- PASS: TestSignedStreamingUpload (0.00s) === RUN TestSignedStreamingUploadWithTrailer --- PASS: TestSignedStreamingUploadWithTrailer (0.00s) === RUN TestSignedStreamingUploadWithTrailerInvalidSignature --- PASS: TestSignedStreamingUploadWithTrailerInvalidSignature (0.00s) === RUN TestSignedStreamingUploadInvalidSignature --- PASS: TestSignedStreamingUploadInvalidSignature (0.00s) === RUN TestCopySourceWithExclamationMark === RUN TestCopySourceWithExclamationMark/encoded_exclamation_mark_-_different_dest === RUN TestCopySourceWithExclamationMark/unencoded_exclamation_mark_-_different_dest === RUN TestCopySourceWithExclamationMark/encoded_exclamation_mark_-_same_dest === RUN TestCopySourceWithExclamationMark/encoded_path_with_versionId === RUN TestCopySourceWithExclamationMark/unencoded_path_with_versionId === RUN TestCopySourceWithExclamationMark/plus_sign_in_key_with_versionId --- PASS: TestCopySourceWithExclamationMark (0.04s) --- PASS: TestCopySourceWithExclamationMark/encoded_exclamation_mark_-_different_dest (0.00s) --- PASS: TestCopySourceWithExclamationMark/unencoded_exclamation_mark_-_different_dest (0.00s) --- PASS: TestCopySourceWithExclamationMark/encoded_exclamation_mark_-_same_dest (0.00s) --- PASS: TestCopySourceWithExclamationMark/encoded_path_with_versionId (0.04s) --- PASS: TestCopySourceWithExclamationMark/unencoded_path_with_versionId (0.00s) --- PASS: TestCopySourceWithExclamationMark/plus_sign_in_key_with_versionId (0.00s) === RUN TestCopySourceDecodingPlusSign --- PASS: TestCopySourceDecodingPlusSign (0.00s) === RUN TestCopySourceRejectsTraversal === RUN TestCopySourceRejectsTraversal/clean_source_passes === RUN TestCopySourceRejectsTraversal/leading_slash_source_passes === RUN TestCopySourceRejectsTraversal/source_with_versionId_passes === RUN TestCopySourceRejectsTraversal/dotdot_escapes_bucket === RUN TestCopySourceRejectsTraversal/leading_slash_dotdot_escapes_bucket === RUN TestCopySourceRejectsTraversal/percent-encoded_dotdot_escapes_bucket === RUN TestCopySourceRejectsTraversal/encoded_slash_dotdot_escapes_bucket === RUN TestCopySourceRejectsTraversal/backslash_dotdot_escapes_bucket === RUN TestCopySourceRejectsTraversal/nested_dotdot_escapes_bucket === RUN TestCopySourceRejectsTraversal/bare_dot_segment_rejected === RUN TestCopySourceRejectsTraversal/dotdot_bucket_rejected === RUN TestCopySourceRejectsTraversal/dotdot_with_versionId_escapes_bucket === RUN TestCopySourceRejectsTraversal/versionId_encoded_slash_rejected === RUN TestCopySourceRejectsTraversal/versionId_backslash_rejected --- PASS: TestCopySourceRejectsTraversal (0.01s) --- PASS: TestCopySourceRejectsTraversal/clean_source_passes (0.00s) --- PASS: TestCopySourceRejectsTraversal/leading_slash_source_passes (0.00s) --- PASS: TestCopySourceRejectsTraversal/source_with_versionId_passes (0.00s) --- PASS: TestCopySourceRejectsTraversal/dotdot_escapes_bucket (0.00s) --- PASS: TestCopySourceRejectsTraversal/leading_slash_dotdot_escapes_bucket (0.00s) --- PASS: TestCopySourceRejectsTraversal/percent-encoded_dotdot_escapes_bucket (0.00s) --- PASS: TestCopySourceRejectsTraversal/encoded_slash_dotdot_escapes_bucket (0.00s) --- PASS: TestCopySourceRejectsTraversal/backslash_dotdot_escapes_bucket (0.00s) --- PASS: TestCopySourceRejectsTraversal/nested_dotdot_escapes_bucket (0.00s) --- PASS: TestCopySourceRejectsTraversal/bare_dot_segment_rejected (0.00s) --- PASS: TestCopySourceRejectsTraversal/dotdot_bucket_rejected (0.00s) --- PASS: TestCopySourceRejectsTraversal/dotdot_with_versionId_escapes_bucket (0.00s) --- PASS: TestCopySourceRejectsTraversal/versionId_encoded_slash_rejected (0.00s) --- PASS: TestCopySourceRejectsTraversal/versionId_backslash_rejected (0.00s) === RUN TestCopySourceRoutingWithSpecialChars === RUN TestCopySourceRoutingWithSpecialChars/different_keys,_source_has_encoded_! === RUN TestCopySourceRoutingWithSpecialChars/same_key_with_!,_source_encoded,_dest_unencoded === RUN TestCopySourceRoutingWithSpecialChars/same_key_with_!,_both_percent-encoded_differently === RUN TestCopySourceRoutingWithSpecialChars/key_with_+_sign,_source_has_literal_+ === RUN TestCopySourceRoutingWithSpecialChars/key_with_+_sign,_source_has_%2B === RUN TestCopySourceRoutingWithSpecialChars/lowercase_%2f_in_copy_source --- PASS: TestCopySourceRoutingWithSpecialChars (0.08s) --- PASS: TestCopySourceRoutingWithSpecialChars/different_keys,_source_has_encoded_! (0.07s) --- PASS: TestCopySourceRoutingWithSpecialChars/same_key_with_!,_source_encoded,_dest_unencoded (0.00s) --- PASS: TestCopySourceRoutingWithSpecialChars/same_key_with_!,_both_percent-encoded_differently (0.00s) --- PASS: TestCopySourceRoutingWithSpecialChars/key_with_+_sign,_source_has_literal_+ (0.00s) --- PASS: TestCopySourceRoutingWithSpecialChars/key_with_+_sign,_source_has_%2B (0.00s) --- PASS: TestCopySourceRoutingWithSpecialChars/lowercase_%2f_in_copy_source (0.00s) === RUN TestCombineCRC64NVME --- PASS: TestCombineCRC64NVME (0.11s) === RUN TestCombineCRC32 --- PASS: TestCombineCRC32 (0.11s) === RUN TestCombineCRC32C --- PASS: TestCombineCRC32C (0.15s) === RUN TestCombineCRC64NVMEEmptySecond --- PASS: TestCombineCRC64NVMEEmptySecond (0.00s) === RUN TestCombineCRC32EmptySecond --- PASS: TestCombineCRC32EmptySecond (0.00s) === RUN TestComputeFullObjectChecksumCRC64NVME --- PASS: TestComputeFullObjectChecksumCRC64NVME (0.20s) === RUN TestComputeFullObjectChecksumCRC32 --- PASS: TestComputeFullObjectChecksumCRC32 (0.17s) === RUN TestResolveMultipartChecksumType === RUN TestResolveMultipartChecksumType/crc64_default_full === RUN TestResolveMultipartChecksumType/crc64_explicit_full === RUN TestResolveMultipartChecksumType/crc64_composite_rejected === RUN TestResolveMultipartChecksumType/crc32_default_composite === RUN TestResolveMultipartChecksumType/crc32_explicit_composite === RUN TestResolveMultipartChecksumType/crc32_explicit_full === RUN TestResolveMultipartChecksumType/crc32c_explicit_full === RUN TestResolveMultipartChecksumType/sha256_default_composite === RUN TestResolveMultipartChecksumType/sha256_full_rejected === RUN TestResolveMultipartChecksumType/invalid_checksum_type --- PASS: TestResolveMultipartChecksumType (0.00s) --- PASS: TestResolveMultipartChecksumType/crc64_default_full (0.00s) --- PASS: TestResolveMultipartChecksumType/crc64_explicit_full (0.00s) --- PASS: TestResolveMultipartChecksumType/crc64_composite_rejected (0.00s) --- PASS: TestResolveMultipartChecksumType/crc32_default_composite (0.00s) --- PASS: TestResolveMultipartChecksumType/crc32_explicit_composite (0.00s) --- PASS: TestResolveMultipartChecksumType/crc32_explicit_full (0.00s) --- PASS: TestResolveMultipartChecksumType/crc32c_explicit_full (0.00s) --- PASS: TestResolveMultipartChecksumType/sha256_default_composite (0.00s) --- PASS: TestResolveMultipartChecksumType/sha256_full_rejected (0.00s) --- PASS: TestResolveMultipartChecksumType/invalid_checksum_type (0.00s) === RUN TestGetEtagFromEntryPrefersStoredExtendedETag --- PASS: TestGetEtagFromEntryPrefersStoredExtendedETag (0.00s) === RUN TestGetEtagFromEntryFallbacksToFilerETag --- PASS: TestGetEtagFromEntryFallbacksToFilerETag (0.01s) === RUN TestCalculateMultipartETagUsesStoredPartETags --- PASS: TestCalculateMultipartETagUsesStoredPartETags (0.00s) === RUN TestCompletedMultipartChunkBackfillsSSES3MetadataFromUploadEntry --- PASS: TestCompletedMultipartChunkBackfillsSSES3MetadataFromUploadEntry (0.00s) === RUN TestCompletedMultipartChunkPreservesExistingSSES3Metadata --- PASS: TestCompletedMultipartChunkPreservesExistingSSES3Metadata (0.00s) === RUN TestApplyMultipartSSES3HeadersFromUploadEntry === RUN TestApplyMultipartSSES3HeadersFromUploadEntry/backfills_missing_canonical_attributes === RUN TestApplyMultipartSSES3HeadersFromUploadEntry/does_not_clobber_existing_canonical_attributes === RUN TestApplyMultipartSSES3HeadersFromUploadEntry/nil_info_is_a_no-op --- PASS: TestApplyMultipartSSES3HeadersFromUploadEntry (0.00s) --- PASS: TestApplyMultipartSSES3HeadersFromUploadEntry/backfills_missing_canonical_attributes (0.00s) --- PASS: TestApplyMultipartSSES3HeadersFromUploadEntry/does_not_clobber_existing_canonical_attributes (0.00s) --- PASS: TestApplyMultipartSSES3HeadersFromUploadEntry/nil_info_is_a_no-op (0.00s) === RUN TestCompletedMultipartChunkBackfilledIVDecryptsActualCiphertext === RUN TestCompletedMultipartChunkBackfilledIVDecryptsActualCiphertext/part1 === RUN TestCompletedMultipartChunkBackfilledIVDecryptsActualCiphertext/part2 --- PASS: TestCompletedMultipartChunkBackfilledIVDecryptsActualCiphertext (4.50s) --- PASS: TestCompletedMultipartChunkBackfilledIVDecryptsActualCiphertext/part1 (2.51s) --- PASS: TestCompletedMultipartChunkBackfilledIVDecryptsActualCiphertext/part2 (1.39s) === RUN TestCompletedMultipartChunkRejectsPartNumberMultiplierFormula --- PASS: TestCompletedMultipartChunkRejectsPartNumberMultiplierFormula (0.00s) === RUN TestDeleteObjectEntryDemotesNonEmptyDirectoryMarker --- PASS: TestDeleteObjectEntryDemotesNonEmptyDirectoryMarker (0.00s) === RUN TestDeleteObjectEntryTreatsImplicitDirectoryAsSuccessfulNoop --- PASS: TestDeleteObjectEntryTreatsImplicitDirectoryAsSuccessfulNoop (0.00s) === RUN TestDeleteObjectEntryIgnoresConcurrentUpdateNotFound --- PASS: TestDeleteObjectEntryIgnoresConcurrentUpdateNotFound (0.02s) === RUN TestDeleteObjectEntryPropagatesNonDirectoryDeleteErrors --- PASS: TestDeleteObjectEntryPropagatesNonDirectoryDeleteErrors (0.00s) === RUN TestAuthorizeObjectDelete_AwsCanonicalPolicy --- PASS: TestAuthorizeObjectDelete_AwsCanonicalPolicy (0.00s) === RUN TestAuthorizeObjectDelete_PrefixScopedPolicy --- PASS: TestAuthorizeObjectDelete_PrefixScopedPolicy (0.00s) === RUN TestLoadIAMManagerFromConfig_Defaults --- PASS: TestLoadIAMManagerFromConfig_Defaults (0.02s) === RUN TestLoadIAMManagerFromConfig_Overrides --- PASS: TestLoadIAMManagerFromConfig_Overrides (0.00s) === RUN TestLoadIAMManagerFromConfig_PartialDefaults --- PASS: TestLoadIAMManagerFromConfig_PartialDefaults (0.00s) === RUN TestLoadIAMManagerFromConfig_ExplicitEmptyKey --- PASS: TestLoadIAMManagerFromConfig_ExplicitEmptyKey (0.00s) === RUN TestLoadIAMManagerFromConfig_MissingKeyError --- PASS: TestLoadIAMManagerFromConfig_MissingKeyError (0.00s) === RUN TestLoadIAMManagerFromConfig_ExplicitFileDefaultsToDeny --- PASS: TestLoadIAMManagerFromConfig_ExplicitFileDefaultsToDeny (0.00s) === RUN TestLoadIAMManagerFromConfig_NoFileDefaultsToAllow --- PASS: TestLoadIAMManagerFromConfig_NoFileDefaultsToAllow (0.00s) === RUN TestLoadIAMManagerFromConfig_ExplicitFileEnforcesUserScopedPolicy --- PASS: TestLoadIAMManagerFromConfig_ExplicitFileEnforcesUserScopedPolicy (0.01s) === RUN TestEvaluateIAMPolicies_ListBucketWithPrefix --- PASS: TestEvaluateIAMPolicies_ListBucketWithPrefix (0.00s) === RUN TestEvaluateIAMPolicies_ListBucketPrefixCondition --- PASS: TestEvaluateIAMPolicies_ListBucketPrefixCondition (0.00s) === RUN TestEvaluateIAMPolicies_ListBucketVersionsWithPrefix --- PASS: TestEvaluateIAMPolicies_ListBucketVersionsWithPrefix (0.00s) === RUN TestLoadIAMManagerWithNoConfig I0824 22:18:50.879406 config_loader.go:73 Using explicit credential store: memory --- PASS: TestLoadIAMManagerWithNoConfig (0.00s) === RUN TestLoadIAMManagerFromConfig_EmptyConfigWithFallbackKey I0824 22:18:50.880878 config_loader.go:73 Using explicit credential store: memory --- PASS: TestLoadIAMManagerFromConfig_EmptyConfigWithFallbackKey (0.00s) === RUN TestSetIAMIntegrationKeepsAuthDisabledWithoutConfig I0824 22:18:50.881701 config_loader.go:73 Using explicit credential store: memory --- PASS: TestSetIAMIntegrationKeepsAuthDisabledWithoutConfig (0.00s) === RUN TestIdentityFieldsForS3TablesReflection --- PASS: TestIdentityFieldsForS3TablesReflection (0.00s) === RUN TestMapBaseActionToS3Format_ServicePrefixPassthrough === RUN TestMapBaseActionToS3Format_ServicePrefixPassthrough/s3_prefix === RUN TestMapBaseActionToS3Format_ServicePrefixPassthrough/iam_prefix === RUN TestMapBaseActionToS3Format_ServicePrefixPassthrough/sts:AssumeRole === RUN TestMapBaseActionToS3Format_ServicePrefixPassthrough/sts:GetFederationToken === RUN TestMapBaseActionToS3Format_ServicePrefixPassthrough/sts:GetCallerIdentity === RUN TestMapBaseActionToS3Format_ServicePrefixPassthrough/coarse_Read_maps_to_s3:GetObject === RUN TestMapBaseActionToS3Format_ServicePrefixPassthrough/coarse_Write_maps_to_s3:PutObject --- PASS: TestMapBaseActionToS3Format_ServicePrefixPassthrough (0.00s) --- PASS: TestMapBaseActionToS3Format_ServicePrefixPassthrough/s3_prefix (0.00s) --- PASS: TestMapBaseActionToS3Format_ServicePrefixPassthrough/iam_prefix (0.00s) --- PASS: TestMapBaseActionToS3Format_ServicePrefixPassthrough/sts:AssumeRole (0.00s) --- PASS: TestMapBaseActionToS3Format_ServicePrefixPassthrough/sts:GetFederationToken (0.00s) --- PASS: TestMapBaseActionToS3Format_ServicePrefixPassthrough/sts:GetCallerIdentity (0.00s) --- PASS: TestMapBaseActionToS3Format_ServicePrefixPassthrough/coarse_Read_maps_to_s3:GetObject (0.00s) --- PASS: TestMapBaseActionToS3Format_ServicePrefixPassthrough/coarse_Write_maps_to_s3:PutObject (0.00s) === RUN TestResolveS3Action_STSActionsPassthrough === RUN TestResolveS3Action_STSActionsPassthrough/nil_request_sts:AssumeRole === RUN TestResolveS3Action_STSActionsPassthrough/with_request_sts:AssumeRole === RUN TestResolveS3Action_STSActionsPassthrough/nil_request_sts:GetFederationToken === RUN TestResolveS3Action_STSActionsPassthrough/with_request_sts:GetFederationToken === RUN TestResolveS3Action_STSActionsPassthrough/nil_request_sts:GetCallerIdentity === RUN TestResolveS3Action_STSActionsPassthrough/with_request_sts:GetCallerIdentity --- PASS: TestResolveS3Action_STSActionsPassthrough (0.00s) --- PASS: TestResolveS3Action_STSActionsPassthrough/nil_request_sts:AssumeRole (0.00s) --- PASS: TestResolveS3Action_STSActionsPassthrough/with_request_sts:AssumeRole (0.00s) --- PASS: TestResolveS3Action_STSActionsPassthrough/nil_request_sts:GetFederationToken (0.00s) --- PASS: TestResolveS3Action_STSActionsPassthrough/with_request_sts:GetFederationToken (0.00s) --- PASS: TestResolveS3Action_STSActionsPassthrough/nil_request_sts:GetCallerIdentity (0.00s) --- PASS: TestResolveS3Action_STSActionsPassthrough/with_request_sts:GetCallerIdentity (0.00s) === RUN TestResolveS3Action_AttributesBeforeVersionId === RUN TestResolveS3Action_AttributesBeforeVersionId/attributes_only === RUN TestResolveS3Action_AttributesBeforeVersionId/attributes_with_versionId === RUN TestResolveS3Action_AttributesBeforeVersionId/versionId_only_GET === RUN TestResolveS3Action_AttributesBeforeVersionId/versionId_only_DELETE --- PASS: TestResolveS3Action_AttributesBeforeVersionId (0.00s) --- PASS: TestResolveS3Action_AttributesBeforeVersionId/attributes_only (0.00s) --- PASS: TestResolveS3Action_AttributesBeforeVersionId/attributes_with_versionId (0.00s) --- PASS: TestResolveS3Action_AttributesBeforeVersionId/versionId_only_GET (0.00s) --- PASS: TestResolveS3Action_AttributesBeforeVersionId/versionId_only_DELETE (0.00s) === RUN TestResolveS3ActionKeepsNonS3Service === RUN TestResolveS3ActionKeepsNonS3Service/iam_action_with_batch_delete_query === RUN TestResolveS3ActionKeepsNonS3Service/iam_action_with_acl_query === RUN TestResolveS3ActionKeepsNonS3Service/iam_action_with_tagging_query === RUN TestResolveS3ActionKeepsNonS3Service/sts_action_with_batch_delete_query --- PASS: TestResolveS3ActionKeepsNonS3Service (0.00s) --- PASS: TestResolveS3ActionKeepsNonS3Service/iam_action_with_batch_delete_query (0.00s) --- PASS: TestResolveS3ActionKeepsNonS3Service/iam_action_with_acl_query (0.00s) --- PASS: TestResolveS3ActionKeepsNonS3Service/iam_action_with_tagging_query (0.00s) --- PASS: TestResolveS3ActionKeepsNonS3Service/sts_action_with_batch_delete_query (0.00s) === RUN TestContentEncodingPreservation === RUN TestContentEncodingPreservation/gzip_encoding === RUN TestContentEncodingPreservation/zstd_encoding === RUN TestContentEncodingPreservation/deflate_encoding === RUN TestContentEncodingPreservation/br_(Brotli)_encoding === RUN TestContentEncodingPreservation/multiple_encodings === RUN TestContentEncodingPreservation/encoding_with_language === RUN TestContentEncodingPreservation/language_only --- PASS: TestContentEncodingPreservation (0.01s) --- PASS: TestContentEncodingPreservation/gzip_encoding (0.00s) --- PASS: TestContentEncodingPreservation/zstd_encoding (0.00s) --- PASS: TestContentEncodingPreservation/deflate_encoding (0.00s) --- PASS: TestContentEncodingPreservation/br_(Brotli)_encoding (0.00s) --- PASS: TestContentEncodingPreservation/multiple_encodings (0.00s) --- PASS: TestContentEncodingPreservation/encoding_with_language (0.00s) --- PASS: TestContentEncodingPreservation/language_only (0.00s) === RUN TestContentEncodingWithOtherHeaders --- PASS: TestContentEncodingWithOtherHeaders (0.00s) === RUN TestEncodePath --- PASS: TestEncodePath (0.00s) === RUN TestEncodePathEqual --- PASS: TestEncodePathEqual (0.00s) === RUN TestS3EndToEndWithJWT === RUN TestS3EndToEndWithJWT/S3_Read-Only_Role_Complete_Workflow === RUN TestS3EndToEndWithJWT/S3_Read-Only_Role_Complete_Workflow/S3_Read-Only_Role_Complete_Workflow_CreateBucket s3_end_to_end_test.go:793: S3 Operation: PUT /test-bucket -> 403 (DENIED) === RUN TestS3EndToEndWithJWT/S3_Read-Only_Role_Complete_Workflow/S3_Read-Only_Role_Complete_Workflow_ListBucket s3_end_to_end_test.go:793: S3 Operation: GET /test-bucket -> 200 (ALLOWED) === RUN TestS3EndToEndWithJWT/S3_Read-Only_Role_Complete_Workflow/S3_Read-Only_Role_Complete_Workflow_PutObject s3_end_to_end_test.go:793: S3 Operation: PUT /test-bucket/test-file.txt -> 403 (DENIED) === RUN TestS3EndToEndWithJWT/S3_Read-Only_Role_Complete_Workflow/S3_Read-Only_Role_Complete_Workflow_GetObject s3_end_to_end_test.go:793: S3 Operation: GET /test-bucket/test-file.txt -> 200 (ALLOWED) === RUN TestS3EndToEndWithJWT/S3_Read-Only_Role_Complete_Workflow/S3_Read-Only_Role_Complete_Workflow_HeadObject s3_end_to_end_test.go:793: S3 Operation: HEAD /test-bucket/test-file.txt -> 200 (ALLOWED) === RUN TestS3EndToEndWithJWT/S3_Read-Only_Role_Complete_Workflow/S3_Read-Only_Role_Complete_Workflow_DeleteObject s3_end_to_end_test.go:793: S3 Operation: DELETE /test-bucket/test-file.txt -> 403 (DENIED) === RUN TestS3EndToEndWithJWT/S3_Admin_Role_Complete_Workflow === RUN TestS3EndToEndWithJWT/S3_Admin_Role_Complete_Workflow/S3_Admin_Role_Complete_Workflow_CreateBucket s3_end_to_end_test.go:793: S3 Operation: PUT /admin-bucket -> 200 (ALLOWED) === RUN TestS3EndToEndWithJWT/S3_Admin_Role_Complete_Workflow/S3_Admin_Role_Complete_Workflow_PutObject s3_end_to_end_test.go:793: S3 Operation: PUT /admin-bucket/admin-file.txt -> 200 (ALLOWED) === RUN TestS3EndToEndWithJWT/S3_Admin_Role_Complete_Workflow/S3_Admin_Role_Complete_Workflow_GetObject s3_end_to_end_test.go:793: S3 Operation: GET /admin-bucket/admin-file.txt -> 200 (ALLOWED) === RUN TestS3EndToEndWithJWT/S3_Admin_Role_Complete_Workflow/S3_Admin_Role_Complete_Workflow_DeleteObject s3_end_to_end_test.go:793: S3 Operation: DELETE /admin-bucket/admin-file.txt -> 200 (ALLOWED) === RUN TestS3EndToEndWithJWT/S3_Admin_Role_Complete_Workflow/S3_Admin_Role_Complete_Workflow_DeleteBucket s3_end_to_end_test.go:793: S3 Operation: DELETE /admin-bucket -> 200 (ALLOWED) === RUN TestS3EndToEndWithJWT/S3_IP-Restricted_Role === RUN TestS3EndToEndWithJWT/S3_IP-Restricted_Role/S3_IP-Restricted_Role_GetObject s3_end_to_end_test.go:793: S3 Operation: GET /restricted-bucket/file.txt -> 200 (ALLOWED) === RUN TestS3EndToEndWithJWT/S3_IP-Restricted_Role/S3_IP-Restricted_Role_GetObject#01 s3_end_to_end_test.go:793: S3 Operation: GET /restricted-bucket/file.txt -> 403 (DENIED) --- PASS: TestS3EndToEndWithJWT (0.10s) --- PASS: TestS3EndToEndWithJWT/S3_Read-Only_Role_Complete_Workflow (0.05s) --- PASS: TestS3EndToEndWithJWT/S3_Read-Only_Role_Complete_Workflow/S3_Read-Only_Role_Complete_Workflow_CreateBucket (0.01s) --- PASS: TestS3EndToEndWithJWT/S3_Read-Only_Role_Complete_Workflow/S3_Read-Only_Role_Complete_Workflow_ListBucket (0.00s) --- PASS: TestS3EndToEndWithJWT/S3_Read-Only_Role_Complete_Workflow/S3_Read-Only_Role_Complete_Workflow_PutObject (0.01s) --- PASS: TestS3EndToEndWithJWT/S3_Read-Only_Role_Complete_Workflow/S3_Read-Only_Role_Complete_Workflow_GetObject (0.00s) --- PASS: TestS3EndToEndWithJWT/S3_Read-Only_Role_Complete_Workflow/S3_Read-Only_Role_Complete_Workflow_HeadObject (0.01s) --- PASS: TestS3EndToEndWithJWT/S3_Read-Only_Role_Complete_Workflow/S3_Read-Only_Role_Complete_Workflow_DeleteObject (0.00s) --- PASS: TestS3EndToEndWithJWT/S3_Admin_Role_Complete_Workflow (0.03s) --- PASS: TestS3EndToEndWithJWT/S3_Admin_Role_Complete_Workflow/S3_Admin_Role_Complete_Workflow_CreateBucket (0.01s) --- PASS: TestS3EndToEndWithJWT/S3_Admin_Role_Complete_Workflow/S3_Admin_Role_Complete_Workflow_PutObject (0.00s) --- PASS: TestS3EndToEndWithJWT/S3_Admin_Role_Complete_Workflow/S3_Admin_Role_Complete_Workflow_GetObject (0.00s) --- PASS: TestS3EndToEndWithJWT/S3_Admin_Role_Complete_Workflow/S3_Admin_Role_Complete_Workflow_DeleteObject (0.00s) --- PASS: TestS3EndToEndWithJWT/S3_Admin_Role_Complete_Workflow/S3_Admin_Role_Complete_Workflow_DeleteBucket (0.00s) --- PASS: TestS3EndToEndWithJWT/S3_IP-Restricted_Role (0.02s) --- PASS: TestS3EndToEndWithJWT/S3_IP-Restricted_Role/S3_IP-Restricted_Role_GetObject (0.00s) --- PASS: TestS3EndToEndWithJWT/S3_IP-Restricted_Role/S3_IP-Restricted_Role_GetObject#01 (0.01s) === RUN TestS3MultipartUploadWithJWT === RUN TestS3MultipartUploadWithJWT/Initialize_Multipart_Upload s3_end_to_end_test.go:793: S3 Operation: POST /multipart-bucket/large-file.txt?uploads -> 404 (DENIED) s3_end_to_end_test.go:799: Non-auth error: Not found === RUN TestS3MultipartUploadWithJWT/Upload_Part s3_end_to_end_test.go:793: S3 Operation: PUT /multipart-bucket/large-file.txt?partNumber=1&uploadId=test-upload-id -> 200 (ALLOWED) === RUN TestS3MultipartUploadWithJWT/List_Parts s3_end_to_end_test.go:793: S3 Operation: GET /multipart-bucket/large-file.txt?uploadId=test-upload-id -> 200 (ALLOWED) === RUN TestS3MultipartUploadWithJWT/Complete_Multipart_Upload s3_end_to_end_test.go:793: S3 Operation: POST /multipart-bucket/large-file.txt?uploadId=test-upload-id -> 404 (DENIED) s3_end_to_end_test.go:799: Non-auth error: Not found --- PASS: TestS3MultipartUploadWithJWT (0.02s) --- PASS: TestS3MultipartUploadWithJWT/Initialize_Multipart_Upload (0.00s) --- PASS: TestS3MultipartUploadWithJWT/Upload_Part (0.01s) --- PASS: TestS3MultipartUploadWithJWT/List_Parts (0.00s) --- PASS: TestS3MultipartUploadWithJWT/Complete_Multipart_Upload (0.00s) === RUN TestS3ListObjectsV2PrefixCondition === RUN TestS3ListObjectsV2PrefixCondition/ListObjectsV2_with_matching_prefix_query_param_and_empty_objectKey === RUN TestS3ListObjectsV2PrefixCondition/ListObjectsV2_with_matching_prefix_propagated_as_objectKey === RUN TestS3ListObjectsV2PrefixCondition/ListObjectsV1_with_matching_prefix_query_param === RUN TestS3ListObjectsV2PrefixCondition/ListObjectsV2_with_non-matching_prefix_should_be_denied === RUN TestS3ListObjectsV2PrefixCondition/ListObjectsV2_with_no_prefix_should_be_denied --- PASS: TestS3ListObjectsV2PrefixCondition (0.02s) --- PASS: TestS3ListObjectsV2PrefixCondition/ListObjectsV2_with_matching_prefix_query_param_and_empty_objectKey (0.00s) --- PASS: TestS3ListObjectsV2PrefixCondition/ListObjectsV2_with_matching_prefix_propagated_as_objectKey (0.00s) --- PASS: TestS3ListObjectsV2PrefixCondition/ListObjectsV1_with_matching_prefix_query_param (0.00s) --- PASS: TestS3ListObjectsV2PrefixCondition/ListObjectsV2_with_non-matching_prefix_should_be_denied (0.00s) --- PASS: TestS3ListObjectsV2PrefixCondition/ListObjectsV2_with_no_prefix_should_be_denied (0.00s) === RUN TestS3CORSWithJWT --- PASS: TestS3CORSWithJWT (0.02s) === RUN TestS3PerformanceWithIAM s3_end_to_end_test.go:793: S3 Operation: GET /perf-bucket/file-0.txt -> 200 (ALLOWED) s3_end_to_end_test.go:793: S3 Operation: GET /perf-bucket/file-1.txt -> 200 (ALLOWED) s3_end_to_end_test.go:793: S3 Operation: GET /perf-bucket/file-2.txt -> 200 (ALLOWED) s3_end_to_end_test.go:793: S3 Operation: GET /perf-bucket/file-3.txt -> 200 (ALLOWED) s3_end_to_end_test.go:793: S3 Operation: GET /perf-bucket/file-4.txt -> 200 (ALLOWED) s3_end_to_end_test.go:793: S3 Operation: GET /perf-bucket/file-5.txt -> 200 (ALLOWED) s3_end_to_end_test.go:793: S3 Operation: GET /perf-bucket/file-6.txt -> 200 (ALLOWED) s3_end_to_end_test.go:793: S3 Operation: GET /perf-bucket/file-7.txt -> 200 (ALLOWED) s3_end_to_end_test.go:793: S3 Operation: GET /perf-bucket/file-8.txt -> 200 (ALLOWED) s3_end_to_end_test.go:793: S3 Operation: GET /perf-bucket/file-9.txt -> 200 (ALLOWED) s3_end_to_end_test.go:793: S3 Operation: GET /perf-bucket/file-10.txt -> 200 (ALLOWED) s3_end_to_end_test.go:793: S3 Operation: GET /perf-bucket/file-11.txt -> 200 (ALLOWED) s3_end_to_end_test.go:793: S3 Operation: GET /perf-bucket/file-12.txt -> 200 (ALLOWED) s3_end_to_end_test.go:793: S3 Operation: GET /perf-bucket/file-13.txt -> 200 (ALLOWED) s3_end_to_end_test.go:793: S3 Operation: GET /perf-bucket/file-14.txt -> 200 (ALLOWED) s3_end_to_end_test.go:793: S3 Operation: GET /perf-bucket/file-15.txt -> 200 (ALLOWED) s3_end_to_end_test.go:793: S3 Operation: GET /perf-bucket/file-16.txt -> 200 (ALLOWED) s3_end_to_end_test.go:793: S3 Operation: GET /perf-bucket/file-17.txt -> 200 (ALLOWED) s3_end_to_end_test.go:793: S3 Operation: GET /perf-bucket/file-18.txt -> 200 (ALLOWED) s3_end_to_end_test.go:793: S3 Operation: GET /perf-bucket/file-19.txt -> 200 (ALLOWED) s3_end_to_end_test.go:793: S3 Operation: GET /perf-bucket/file-20.txt -> 200 (ALLOWED) s3_end_to_end_test.go:793: S3 Operation: GET /perf-bucket/file-21.txt -> 200 (ALLOWED) s3_end_to_end_test.go:793: S3 Operation: GET /perf-bucket/file-22.txt -> 200 (ALLOWED) s3_end_to_end_test.go:793: S3 Operation: GET /perf-bucket/file-23.txt -> 200 (ALLOWED) s3_end_to_end_test.go:793: S3 Operation: GET /perf-bucket/file-24.txt -> 200 (ALLOWED) s3_end_to_end_test.go:793: S3 Operation: GET /perf-bucket/file-25.txt -> 200 (ALLOWED) s3_end_to_end_test.go:793: S3 Operation: GET /perf-bucket/file-26.txt -> 200 (ALLOWED) s3_end_to_end_test.go:793: S3 Operation: GET /perf-bucket/file-27.txt -> 200 (ALLOWED) s3_end_to_end_test.go:793: S3 Operation: GET /perf-bucket/file-28.txt -> 200 (ALLOWED) s3_end_to_end_test.go:793: S3 Operation: GET /perf-bucket/file-29.txt -> 200 (ALLOWED) s3_end_to_end_test.go:793: S3 Operation: GET /perf-bucket/file-30.txt -> 200 (ALLOWED) s3_end_to_end_test.go:793: S3 Operation: GET /perf-bucket/file-31.txt -> 200 (ALLOWED) s3_end_to_end_test.go:793: S3 Operation: GET /perf-bucket/file-32.txt -> 200 (ALLOWED) s3_end_to_end_test.go:793: S3 Operation: GET /perf-bucket/file-33.txt -> 200 (ALLOWED) s3_end_to_end_test.go:793: S3 Operation: GET /perf-bucket/file-34.txt -> 200 (ALLOWED) s3_end_to_end_test.go:793: S3 Operation: GET /perf-bucket/file-35.txt -> 200 (ALLOWED) s3_end_to_end_test.go:793: S3 Operation: GET /perf-bucket/file-36.txt -> 200 (ALLOWED) s3_end_to_end_test.go:793: S3 Operation: GET /perf-bucket/file-37.txt -> 200 (ALLOWED) s3_end_to_end_test.go:793: S3 Operation: GET /perf-bucket/file-38.txt -> 200 (ALLOWED) s3_end_to_end_test.go:793: S3 Operation: GET /perf-bucket/file-39.txt -> 200 (ALLOWED) s3_end_to_end_test.go:793: S3 Operation: GET /perf-bucket/file-40.txt -> 200 (ALLOWED) s3_end_to_end_test.go:793: S3 Operation: GET /perf-bucket/file-41.txt -> 200 (ALLOWED) s3_end_to_end_test.go:793: S3 Operation: GET /perf-bucket/file-42.txt -> 200 (ALLOWED) s3_end_to_end_test.go:793: S3 Operation: GET /perf-bucket/file-43.txt -> 200 (ALLOWED) s3_end_to_end_test.go:793: S3 Operation: GET /perf-bucket/file-44.txt -> 200 (ALLOWED) s3_end_to_end_test.go:793: S3 Operation: GET /perf-bucket/file-45.txt -> 200 (ALLOWED) s3_end_to_end_test.go:793: S3 Operation: GET /perf-bucket/file-46.txt -> 200 (ALLOWED) s3_end_to_end_test.go:793: S3 Operation: GET /perf-bucket/file-47.txt -> 200 (ALLOWED) s3_end_to_end_test.go:793: S3 Operation: GET /perf-bucket/file-48.txt -> 200 (ALLOWED) s3_end_to_end_test.go:793: S3 Operation: GET /perf-bucket/file-49.txt -> 200 (ALLOWED) s3_end_to_end_test.go:793: S3 Operation: GET /perf-bucket/file-50.txt -> 200 (ALLOWED) s3_end_to_end_test.go:793: S3 Operation: GET /perf-bucket/file-51.txt -> 200 (ALLOWED) s3_end_to_end_test.go:793: S3 Operation: GET /perf-bucket/file-52.txt -> 200 (ALLOWED) s3_end_to_end_test.go:793: S3 Operation: GET /perf-bucket/file-53.txt -> 200 (ALLOWED) s3_end_to_end_test.go:793: S3 Operation: GET /perf-bucket/file-54.txt -> 200 (ALLOWED) s3_end_to_end_test.go:793: S3 Operation: GET /perf-bucket/file-55.txt -> 200 (ALLOWED) s3_end_to_end_test.go:793: S3 Operation: GET /perf-bucket/file-56.txt -> 200 (ALLOWED) s3_end_to_end_test.go:793: S3 Operation: GET /perf-bucket/file-57.txt -> 200 (ALLOWED) s3_end_to_end_test.go:793: S3 Operation: GET /perf-bucket/file-58.txt -> 200 (ALLOWED) s3_end_to_end_test.go:793: S3 Operation: GET /perf-bucket/file-59.txt -> 200 (ALLOWED) s3_end_to_end_test.go:793: S3 Operation: GET /perf-bucket/file-60.txt -> 200 (ALLOWED) s3_end_to_end_test.go:793: S3 Operation: GET /perf-bucket/file-61.txt -> 200 (ALLOWED) s3_end_to_end_test.go:793: S3 Operation: GET /perf-bucket/file-62.txt -> 200 (ALLOWED) s3_end_to_end_test.go:793: S3 Operation: GET /perf-bucket/file-63.txt -> 200 (ALLOWED) s3_end_to_end_test.go:793: S3 Operation: GET /perf-bucket/file-64.txt -> 200 (ALLOWED) s3_end_to_end_test.go:793: S3 Operation: GET /perf-bucket/file-65.txt -> 200 (ALLOWED) s3_end_to_end_test.go:793: S3 Operation: GET /perf-bucket/file-66.txt -> 200 (ALLOWED) s3_end_to_end_test.go:793: S3 Operation: GET /perf-bucket/file-67.txt -> 200 (ALLOWED) s3_end_to_end_test.go:793: S3 Operation: GET /perf-bucket/file-68.txt -> 200 (ALLOWED) s3_end_to_end_test.go:793: S3 Operation: GET /perf-bucket/file-69.txt -> 200 (ALLOWED) s3_end_to_end_test.go:793: S3 Operation: GET /perf-bucket/file-70.txt -> 200 (ALLOWED) s3_end_to_end_test.go:793: S3 Operation: GET /perf-bucket/file-71.txt -> 200 (ALLOWED) s3_end_to_end_test.go:793: S3 Operation: GET /perf-bucket/file-72.txt -> 200 (ALLOWED) s3_end_to_end_test.go:793: S3 Operation: GET /perf-bucket/file-73.txt -> 200 (ALLOWED) s3_end_to_end_test.go:793: S3 Operation: GET /perf-bucket/file-74.txt -> 200 (ALLOWED) s3_end_to_end_test.go:793: S3 Operation: GET /perf-bucket/file-75.txt -> 200 (ALLOWED) s3_end_to_end_test.go:793: S3 Operation: GET /perf-bucket/file-76.txt -> 200 (ALLOWED) s3_end_to_end_test.go:793: S3 Operation: GET /perf-bucket/file-77.txt -> 200 (ALLOWED) s3_end_to_end_test.go:793: S3 Operation: GET /perf-bucket/file-78.txt -> 200 (ALLOWED) s3_end_to_end_test.go:793: S3 Operation: GET /perf-bucket/file-79.txt -> 200 (ALLOWED) s3_end_to_end_test.go:793: S3 Operation: GET /perf-bucket/file-80.txt -> 200 (ALLOWED) s3_end_to_end_test.go:793: S3 Operation: GET /perf-bucket/file-81.txt -> 200 (ALLOWED) s3_end_to_end_test.go:793: S3 Operation: GET /perf-bucket/file-82.txt -> 200 (ALLOWED) s3_end_to_end_test.go:793: S3 Operation: GET /perf-bucket/file-83.txt -> 200 (ALLOWED) s3_end_to_end_test.go:793: S3 Operation: GET /perf-bucket/file-84.txt -> 200 (ALLOWED) s3_end_to_end_test.go:793: S3 Operation: GET /perf-bucket/file-85.txt -> 200 (ALLOWED) s3_end_to_end_test.go:793: S3 Operation: GET /perf-bucket/file-86.txt -> 200 (ALLOWED) s3_end_to_end_test.go:793: S3 Operation: GET /perf-bucket/file-87.txt -> 200 (ALLOWED) s3_end_to_end_test.go:793: S3 Operation: GET /perf-bucket/file-88.txt -> 200 (ALLOWED) s3_end_to_end_test.go:793: S3 Operation: GET /perf-bucket/file-89.txt -> 200 (ALLOWED) s3_end_to_end_test.go:793: S3 Operation: GET /perf-bucket/file-90.txt -> 200 (ALLOWED) s3_end_to_end_test.go:793: S3 Operation: GET /perf-bucket/file-91.txt -> 200 (ALLOWED) s3_end_to_end_test.go:793: S3 Operation: GET /perf-bucket/file-92.txt -> 200 (ALLOWED) s3_end_to_end_test.go:793: S3 Operation: GET /perf-bucket/file-93.txt -> 200 (ALLOWED) s3_end_to_end_test.go:793: S3 Operation: GET /perf-bucket/file-94.txt -> 200 (ALLOWED) s3_end_to_end_test.go:793: S3 Operation: GET /perf-bucket/file-95.txt -> 200 (ALLOWED) s3_end_to_end_test.go:793: S3 Operation: GET /perf-bucket/file-96.txt -> 200 (ALLOWED) s3_end_to_end_test.go:793: S3 Operation: GET /perf-bucket/file-97.txt -> 200 (ALLOWED) s3_end_to_end_test.go:793: S3 Operation: GET /perf-bucket/file-98.txt -> 200 (ALLOWED) s3_end_to_end_test.go:793: S3 Operation: GET /perf-bucket/file-99.txt -> 200 (ALLOWED) s3_end_to_end_test.go:432: Performance Results: s3_end_to_end_test.go:433: - Total requests: 100 s3_end_to_end_test.go:434: - Total time: 209.641006ms s3_end_to_end_test.go:435: - Average latency: 2.09641ms s3_end_to_end_test.go:436: - Requests per second: 477.01 --- PASS: TestS3PerformanceWithIAM (0.21s) === RUN TestS3AuthenticationDenied === RUN TestS3AuthenticationDenied/no_authorization_header === RUN TestS3AuthenticationDenied/empty_bearer_token === RUN TestS3AuthenticationDenied/invalid_jwt_token === RUN TestS3AuthenticationDenied/malformed_authorization_header === RUN TestS3AuthenticationDenied/expired_jwt_token --- PASS: TestS3AuthenticationDenied (0.00s) --- PASS: TestS3AuthenticationDenied/no_authorization_header (0.00s) --- PASS: TestS3AuthenticationDenied/empty_bearer_token (0.00s) --- PASS: TestS3AuthenticationDenied/invalid_jwt_token (0.00s) --- PASS: TestS3AuthenticationDenied/malformed_authorization_header (0.00s) --- PASS: TestS3AuthenticationDenied/expired_jwt_token (0.00s) === RUN TestS3IAMOnlyModeRejectsAnonymous --- PASS: TestS3IAMOnlyModeRejectsAnonymous (0.00s) === RUN TestCheckPolicyWithEntry === RUN TestCheckPolicyWithEntry/object_with_public_tag_-_allow === RUN TestCheckPolicyWithEntry/object_with_private_tag_-_indeterminate_(no_matching_statement) === RUN TestCheckPolicyWithEntry/object_with_no_tags_-_indeterminate_(no_matching_statement) === RUN TestCheckPolicyWithEntry/nil_entry_-_indeterminate_(condition_cannot_be_evaluated) === RUN TestCheckPolicyWithEntry/object_with_multiple_tags_including_public_-_allow --- PASS: TestCheckPolicyWithEntry (0.01s) --- PASS: TestCheckPolicyWithEntry/object_with_public_tag_-_allow (0.01s) --- PASS: TestCheckPolicyWithEntry/object_with_private_tag_-_indeterminate_(no_matching_statement) (0.00s) --- PASS: TestCheckPolicyWithEntry/object_with_no_tags_-_indeterminate_(no_matching_statement) (0.00s) --- PASS: TestCheckPolicyWithEntry/nil_entry_-_indeterminate_(condition_cannot_be_evaluated) (0.00s) --- PASS: TestCheckPolicyWithEntry/object_with_multiple_tags_including_public_-_allow (0.00s) === RUN TestCheckPolicyWithEntryNoPolicyForBucket --- PASS: TestCheckPolicyWithEntryNoPolicyForBucket (0.00s) === RUN TestCheckPolicyWithEntryNilPolicyEngine --- PASS: TestCheckPolicyWithEntryNilPolicyEngine (0.00s) === RUN TestCheckPolicyWithEntryDenyPolicy === RUN TestCheckPolicyWithEntryDenyPolicy/public_object_-_allow === RUN TestCheckPolicyWithEntryDenyPolicy/confidential_object_-_deny === RUN TestCheckPolicyWithEntryDenyPolicy/no_classification_tag_-_allow --- PASS: TestCheckPolicyWithEntryDenyPolicy (0.00s) --- PASS: TestCheckPolicyWithEntryDenyPolicy/public_object_-_allow (0.00s) --- PASS: TestCheckPolicyWithEntryDenyPolicy/confidential_object_-_deny (0.00s) --- PASS: TestCheckPolicyWithEntryDenyPolicy/no_classification_tag_-_allow (0.00s) === RUN TestHasPolicyForBucket --- PASS: TestHasPolicyForBucket (0.00s) === RUN TestGranularActionMappingSecurity === RUN TestGranularActionMappingSecurity/delete_object_security_fix s3_granular_action_security_test.go:167: SECURITY IMPROVEMENT: DELETE object operations should map to s3:DeleteObject, not s3:PutObject s3_granular_action_security_test.go:168: Problem Fixed: Old mapping incorrectly mapped DELETE object to s3:PutObject, allowing users with only PUT permissions to delete objects - a critical security flaw s3_granular_action_security_test.go:169: Granular Action: s3:DeleteObject === RUN TestGranularActionMappingSecurity/get_object_acl_precision s3_granular_action_security_test.go:167: SECURITY IMPROVEMENT: GET object ACL should map to s3:GetObjectAcl, not generic s3:GetObject s3_granular_action_security_test.go:168: Problem Fixed: Old mapping would allow users with s3:GetObject permission to read ACLs, potentially exposing sensitive permission information s3_granular_action_security_test.go:169: Granular Action: s3:GetObjectAcl === RUN TestGranularActionMappingSecurity/put_object_tagging_precision s3_granular_action_security_test.go:167: SECURITY IMPROVEMENT: PUT object tagging should map to s3:PutObjectTagging, not generic s3:PutObject s3_granular_action_security_test.go:168: Problem Fixed: Old mapping couldn't distinguish between actual object uploads and metadata operations like tagging, making fine-grained permissions impossible s3_granular_action_security_test.go:169: Granular Action: s3:PutObjectTagging === RUN TestGranularActionMappingSecurity/multipart_upload_precision s3_granular_action_security_test.go:167: SECURITY IMPROVEMENT: Multipart upload initiation should map to s3:CreateMultipartUpload s3_granular_action_security_test.go:168: Problem Fixed: Old mapping would treat multipart operations as generic s3:PutObject, preventing policies that allow regular uploads but restrict large multipart operations s3_granular_action_security_test.go:169: Granular Action: s3:CreateMultipartUpload === RUN TestGranularActionMappingSecurity/bucket_policy_vs_bucket_creation s3_granular_action_security_test.go:167: SECURITY IMPROVEMENT: Bucket policy modifications should map to s3:PutBucketPolicy, not s3:CreateBucket s3_granular_action_security_test.go:168: Problem Fixed: Old mapping couldn't distinguish between creating buckets and modifying bucket policies, potentially allowing unauthorized policy changes s3_granular_action_security_test.go:169: Granular Action: s3:PutBucketPolicy === RUN TestGranularActionMappingSecurity/list_vs_read_distinction s3_granular_action_security_test.go:167: SECURITY IMPROVEMENT: Listing multipart uploads should map to s3:ListBucketMultipartUploads s3_granular_action_security_test.go:168: Problem Fixed: Old mapping would use generic s3:ListBucket for all bucket operations, preventing fine-grained control over who can see ongoing multipart operations s3_granular_action_security_test.go:169: Granular Action: s3:ListBucketMultipartUploads === RUN TestGranularActionMappingSecurity/delete_object_tagging_precision s3_granular_action_security_test.go:167: SECURITY IMPROVEMENT: Delete object tagging should map to s3:DeleteObjectTagging, not s3:DeleteObject s3_granular_action_security_test.go:168: Problem Fixed: Old mapping couldn't distinguish between deleting objects and deleting tags, preventing policies that allow tag management but not object deletion s3_granular_action_security_test.go:169: Granular Action: s3:DeleteObjectTagging --- PASS: TestGranularActionMappingSecurity (0.00s) --- PASS: TestGranularActionMappingSecurity/delete_object_security_fix (0.00s) --- PASS: TestGranularActionMappingSecurity/get_object_acl_precision (0.00s) --- PASS: TestGranularActionMappingSecurity/put_object_tagging_precision (0.00s) --- PASS: TestGranularActionMappingSecurity/multipart_upload_precision (0.00s) --- PASS: TestGranularActionMappingSecurity/bucket_policy_vs_bucket_creation (0.00s) --- PASS: TestGranularActionMappingSecurity/list_vs_read_distinction (0.00s) --- PASS: TestGranularActionMappingSecurity/delete_object_tagging_precision (0.00s) === RUN TestBackwardCompatibilityFallback === RUN TestBackwardCompatibilityFallback/generic_read_fallback s3_granular_action_security_test.go:237: COMPATIBILITY: Generic read operations should fall back to s3:GetObject for compatibility - s3:GetObject === RUN TestBackwardCompatibilityFallback/generic_write_fallback s3_granular_action_security_test.go:237: COMPATIBILITY: Generic write operations should fall back to s3:PutObject for compatibility - s3:PutObject === RUN TestBackwardCompatibilityFallback/already_granular_passthrough s3_granular_action_security_test.go:237: COMPATIBILITY: Already granular actions should pass through unchanged - s3:GetBucketLocation === RUN TestBackwardCompatibilityFallback/unknown_action_conversion s3_granular_action_security_test.go:237: COMPATIBILITY: Unknown actions should be converted to S3 format for consistency - s3:CustomAction --- PASS: TestBackwardCompatibilityFallback (0.00s) --- PASS: TestBackwardCompatibilityFallback/generic_read_fallback (0.00s) --- PASS: TestBackwardCompatibilityFallback/generic_write_fallback (0.00s) --- PASS: TestBackwardCompatibilityFallback/already_granular_passthrough (0.00s) --- PASS: TestBackwardCompatibilityFallback/unknown_action_conversion (0.00s) === RUN TestPolicyEnforcementScenarios === RUN TestPolicyEnforcementScenarios/allow_read_deny_acl_access s3_granular_action_security_test.go:338: SECURITY SCENARIO: allow_read_deny_acl_access s3_granular_action_security_test.go:339: Expected Action: s3:GetObjectAcl s3_granular_action_security_test.go:340: Security Benefit: Policy allows reading objects but denies ACL access - granular actions enable this distinction s3_granular_action_security_test.go:341: Policy Example: { "Version": "2012-10-17", "Statement": [ { "Effect": "Allow", "Action": "s3:GetObject", "Resource": "arn:aws:s3:::sensitive-bucket/*" } ] } === RUN TestPolicyEnforcementScenarios/allow_tagging_deny_object_modification s3_granular_action_security_test.go:338: SECURITY SCENARIO: allow_tagging_deny_object_modification s3_granular_action_security_test.go:339: Expected Action: s3:PutObjectTagging s3_granular_action_security_test.go:340: Security Benefit: Policy allows tag management but prevents actual object uploads - critical for metadata-only roles s3_granular_action_security_test.go:341: Policy Example: { "Version": "2012-10-17", "Statement": [ { "Effect": "Allow", "Action": ["s3:PutObjectTagging", "s3:DeleteObjectTagging"], "Resource": "arn:aws:s3:::data-bucket/*" } ] } === RUN TestPolicyEnforcementScenarios/restrict_multipart_uploads s3_granular_action_security_test.go:338: SECURITY SCENARIO: restrict_multipart_uploads s3_granular_action_security_test.go:339: Expected Action: s3:CreateMultipartUpload s3_granular_action_security_test.go:340: Security Benefit: Policy allows regular uploads but blocks large multipart uploads - prevents resource abuse s3_granular_action_security_test.go:341: Policy Example: { "Version": "2012-10-17", "Statement": [ { "Effect": "Allow", "Action": "s3:PutObject", "Resource": "arn:aws:s3:::uploads/*" }, { "Effect": "Deny", "Action": ["s3:CreateMultipartUpload", "s3:UploadPart"], "Resource": "arn:aws:s3:::uploads/*" } ] } --- PASS: TestPolicyEnforcementScenarios (0.00s) --- PASS: TestPolicyEnforcementScenarios/allow_read_deny_acl_access (0.00s) --- PASS: TestPolicyEnforcementScenarios/allow_tagging_deny_object_modification (0.00s) --- PASS: TestPolicyEnforcementScenarios/restrict_multipart_uploads (0.00s) === RUN TestDeleteObjectPolicyEnforcement === RUN TestDeleteObjectPolicyEnforcement/delete_object_maps_to_correct_action s3_granular_action_security_test.go:414: ARCHITECTURAL FIX VERIFIED: delete_object_maps_to_correct_action s3_granular_action_security_test.go:415: Method: DELETE -> S3 Action: s3:DeleteObject s3_granular_action_security_test.go:416: Policy Scenario: Policy that denies s3:DeleteObject but allows s3:PutObject should now work correctly === RUN TestDeleteObjectPolicyEnforcement/put_object_maps_to_correct_action s3_granular_action_security_test.go:414: ARCHITECTURAL FIX VERIFIED: put_object_maps_to_correct_action s3_granular_action_security_test.go:415: Method: PUT -> S3 Action: s3:PutObject s3_granular_action_security_test.go:416: Policy Scenario: Policy that allows s3:PutObject but denies s3:DeleteObject should allow uploads === RUN TestDeleteObjectPolicyEnforcement/batch_delete_maps_to_delete_action s3_granular_action_security_test.go:414: ARCHITECTURAL FIX VERIFIED: batch_delete_maps_to_delete_action s3_granular_action_security_test.go:415: Method: POST -> S3 Action: s3:DeleteObject s3_granular_action_security_test.go:416: Policy Scenario: Batch delete operations should also map to s3:DeleteObject --- PASS: TestDeleteObjectPolicyEnforcement (0.00s) --- PASS: TestDeleteObjectPolicyEnforcement/delete_object_maps_to_correct_action (0.00s) --- PASS: TestDeleteObjectPolicyEnforcement/put_object_maps_to_correct_action (0.00s) --- PASS: TestDeleteObjectPolicyEnforcement/batch_delete_maps_to_delete_action (0.00s) === RUN TestFineGrainedPolicyExample s3_granular_action_security_test.go:486: Testing Fine-Grained Policy: s3_granular_action_security_test.go:487: { "Version": "2012-10-17", "Statement": [ { "Sid": "AllowObjectUploads", "Effect": "Allow", "Action": "s3:PutObject", "Resource": "arn:aws:s3:::test-bucket/*" }, { "Sid": "DenyObjectDeletion", "Effect": "Deny", "Action": "s3:DeleteObject", "Resource": "arn:aws:s3:::test-bucket/*" } ] } === RUN TestFineGrainedPolicyExample/PUT_object s3_granular_action_security_test.go:519: [ALLOWED] PUT object -> s3:PutObject s3_granular_action_security_test.go:520: Rationale: Policy explicitly allows s3:PutObject - upload should succeed === RUN TestFineGrainedPolicyExample/DELETE_object s3_granular_action_security_test.go:519: [DENIED] DELETE object -> s3:DeleteObject s3_granular_action_security_test.go:520: Rationale: Policy explicitly denies s3:DeleteObject - deletion should be blocked === RUN TestFineGrainedPolicyExample/Batch_DELETE s3_granular_action_security_test.go:519: [DENIED] Batch DELETE -> s3:DeleteObject s3_granular_action_security_test.go:520: Rationale: Policy explicitly denies s3:DeleteObject - batch deletion should be blocked === NAME TestFineGrainedPolicyExample s3_granular_action_security_test.go:524: ARCHITECTURAL LIMITATION RESOLVED! s3_granular_action_security_test.go:525: Fine-grained policies like 'allow PUT but deny DELETE' now work correctly s3_granular_action_security_test.go:526: The policy engine can distinguish between s3:PutObject and s3:DeleteObject --- PASS: TestFineGrainedPolicyExample (0.00s) --- PASS: TestFineGrainedPolicyExample/PUT_object (0.00s) --- PASS: TestFineGrainedPolicyExample/DELETE_object (0.00s) --- PASS: TestFineGrainedPolicyExample/Batch_DELETE (0.00s) === RUN TestCoarseActionResolution === RUN TestCoarseActionResolution/PUT_with_ACTION_WRITE_resolves_to_PutObject s3_granular_action_security_test.go:595: SUCCESS: PUT_with_ACTION_WRITE_resolves_to_PutObject s3_granular_action_security_test.go:596: Input: PUT /test-bucket/test-file.txt + ACTION_WRITE s3_granular_action_security_test.go:597: Output: s3:PutObject s3_granular_action_security_test.go:598: Policy impact: Policy allowing s3:PutObject should match PUT requests === RUN TestCoarseActionResolution/DELETE_with_ACTION_WRITE_resolves_to_DeleteObject s3_granular_action_security_test.go:595: SUCCESS: DELETE_with_ACTION_WRITE_resolves_to_DeleteObject s3_granular_action_security_test.go:596: Input: DELETE /test-bucket/test-file.txt + ACTION_WRITE s3_granular_action_security_test.go:597: Output: s3:DeleteObject s3_granular_action_security_test.go:598: Policy impact: Policy denying s3:DeleteObject should block DELETE requests === RUN TestCoarseActionResolution/batch_DELETE_with_ACTION_WRITE_resolves_to_DeleteObject s3_granular_action_security_test.go:595: SUCCESS: batch_DELETE_with_ACTION_WRITE_resolves_to_DeleteObject s3_granular_action_security_test.go:596: Input: POST /test-bucket + ACTION_WRITE s3_granular_action_security_test.go:597: Output: s3:DeleteObject s3_granular_action_security_test.go:598: Policy impact: Policy denying s3:DeleteObject should block batch DELETE === RUN TestCoarseActionResolution/POST_multipart_with_ACTION_WRITE_resolves_to_CreateMultipartUpload s3_granular_action_security_test.go:595: SUCCESS: POST_multipart_with_ACTION_WRITE_resolves_to_CreateMultipartUpload s3_granular_action_security_test.go:596: Input: POST /test-bucket/large-file.mp4 + ACTION_WRITE s3_granular_action_security_test.go:597: Output: s3:CreateMultipartUpload s3_granular_action_security_test.go:598: Policy impact: Policy allowing s3:PutObject but denying s3:CreateMultipartUpload can now work === NAME TestCoarseActionResolution s3_granular_action_security_test.go:602: === ARCHITECTURAL LIMITATION RESOLVED === s3_granular_action_security_test.go:603: Handlers can use coarse ACTION_WRITE constant, and the context-aware s3_granular_action_security_test.go:604: resolver will map it to the correct specific S3 action (PutObject, s3_granular_action_security_test.go:605: DeleteObject, CreateMultipartUpload, etc.) based on HTTP method and s3_granular_action_security_test.go:606: query parameters. This enables fine-grained bucket policies like: s3_granular_action_security_test.go:607: - Allow s3:PutObject but Deny s3:DeleteObject (append-only) s3_granular_action_security_test.go:608: - Allow regular uploads but Deny multipart (size limits) --- PASS: TestCoarseActionResolution (0.00s) --- PASS: TestCoarseActionResolution/PUT_with_ACTION_WRITE_resolves_to_PutObject (0.00s) --- PASS: TestCoarseActionResolution/DELETE_with_ACTION_WRITE_resolves_to_DeleteObject (0.00s) --- PASS: TestCoarseActionResolution/batch_DELETE_with_ACTION_WRITE_resolves_to_DeleteObject (0.00s) --- PASS: TestCoarseActionResolution/POST_multipart_with_ACTION_WRITE_resolves_to_CreateMultipartUpload (0.00s) === RUN TestOIDCBearerEnforcesRoleTrustPolicy --- PASS: TestOIDCBearerEnforcesRoleTrustPolicy (0.62s) === RUN TestGroupsRolesPolicyEnforcement === RUN TestGroupsRolesPolicyEnforcement/group_member_gets_team-a_bucket_only === RUN TestGroupsRolesPolicyEnforcement/role_holder_gets_team-b_bucket_only === RUN TestGroupsRolesPolicyEnforcement/member_of_both_gets_both_buckets_in_one_session === RUN TestGroupsRolesPolicyEnforcement/user_without_matching_claims_is_denied_everywhere --- PASS: TestGroupsRolesPolicyEnforcement (0.03s) --- PASS: TestGroupsRolesPolicyEnforcement/group_member_gets_team-a_bucket_only (0.01s) --- PASS: TestGroupsRolesPolicyEnforcement/role_holder_gets_team-b_bucket_only (0.01s) --- PASS: TestGroupsRolesPolicyEnforcement/member_of_both_gets_both_buckets_in_one_session (0.01s) --- PASS: TestGroupsRolesPolicyEnforcement/user_without_matching_claims_is_denied_everywhere (0.01s) === RUN TestGroupsRolesSessionRequestContext --- PASS: TestGroupsRolesSessionRequestContext (0.00s) === RUN TestSelectPrimaryRole === RUN TestSelectPrimaryRole/empty_roles_returns_empty === RUN TestSelectPrimaryRole/single_role_returns_that_role === RUN TestSelectPrimaryRole/multiple_roles_returns_first === RUN TestSelectPrimaryRole/order_matters === RUN TestSelectPrimaryRole/complex_enterprise_roles --- PASS: TestSelectPrimaryRole (0.00s) --- PASS: TestSelectPrimaryRole/empty_roles_returns_empty (0.00s) --- PASS: TestSelectPrimaryRole/single_role_returns_that_role (0.00s) --- PASS: TestSelectPrimaryRole/multiple_roles_returns_first (0.00s) --- PASS: TestSelectPrimaryRole/order_matters (0.00s) --- PASS: TestSelectPrimaryRole/complex_enterprise_roles (0.00s) === RUN TestJWTAuthenticationFlow === RUN TestJWTAuthenticationFlow/Read-Only_JWT_Authentication === RUN TestJWTAuthenticationFlow/Read-Only_JWT_Authentication/Read === RUN TestJWTAuthenticationFlow/Read-Only_JWT_Authentication/Write === RUN TestJWTAuthenticationFlow/Read-Only_JWT_Authentication/List === RUN TestJWTAuthenticationFlow/Admin_JWT_Authentication === RUN TestJWTAuthenticationFlow/Admin_JWT_Authentication/Read === RUN TestJWTAuthenticationFlow/Admin_JWT_Authentication/Write === RUN TestJWTAuthenticationFlow/Admin_JWT_Authentication/DeleteBucket --- PASS: TestJWTAuthenticationFlow (0.03s) --- PASS: TestJWTAuthenticationFlow/Read-Only_JWT_Authentication (0.01s) --- PASS: TestJWTAuthenticationFlow/Read-Only_JWT_Authentication/Read (0.00s) --- PASS: TestJWTAuthenticationFlow/Read-Only_JWT_Authentication/Write (0.00s) --- PASS: TestJWTAuthenticationFlow/Read-Only_JWT_Authentication/List (0.00s) --- PASS: TestJWTAuthenticationFlow/Admin_JWT_Authentication (0.02s) --- PASS: TestJWTAuthenticationFlow/Admin_JWT_Authentication/Read (0.01s) --- PASS: TestJWTAuthenticationFlow/Admin_JWT_Authentication/Write (0.00s) --- PASS: TestJWTAuthenticationFlow/Admin_JWT_Authentication/DeleteBucket (0.00s) === RUN TestJWTTokenValidation === RUN TestJWTTokenValidation/Empty_token === RUN TestJWTTokenValidation/Invalid_token_format === RUN TestJWTTokenValidation/Expired_token --- PASS: TestJWTTokenValidation (0.00s) --- PASS: TestJWTTokenValidation/Empty_token (0.00s) --- PASS: TestJWTTokenValidation/Invalid_token_format (0.00s) --- PASS: TestJWTTokenValidation/Expired_token (0.00s) === RUN TestRequestContextExtraction === RUN TestRequestContextExtraction/Standard_request_with_IP === RUN TestRequestContextExtraction/Request_with_X-Real-IP --- PASS: TestRequestContextExtraction (0.00s) --- PASS: TestRequestContextExtraction/Standard_request_with_IP (0.00s) --- PASS: TestRequestContextExtraction/Request_with_X-Real-IP (0.00s) === RUN TestIPBasedPolicyEnforcement === RUN TestIPBasedPolicyEnforcement/Allow_from_office_IP === RUN TestIPBasedPolicyEnforcement/Block_from_external_IP === RUN TestIPBasedPolicyEnforcement/Allow_from_internal_range --- PASS: TestIPBasedPolicyEnforcement (0.01s) --- PASS: TestIPBasedPolicyEnforcement/Allow_from_office_IP (0.00s) --- PASS: TestIPBasedPolicyEnforcement/Block_from_external_IP (0.00s) --- PASS: TestIPBasedPolicyEnforcement/Allow_from_internal_range (0.00s) === RUN TestListPartsActionMapping === RUN TestListPartsActionMapping/get_object_without_uploadId === RUN TestListPartsActionMapping/get_object_with_uploadId === RUN TestListPartsActionMapping/get_object_with_uploadId_and_other_params === RUN TestListPartsActionMapping/get_object_with_versionId === RUN TestListPartsActionMapping/get_object_acl_without_uploadId === RUN TestListPartsActionMapping/post_multipart_upload_without_uploadId --- PASS: TestListPartsActionMapping (0.01s) --- PASS: TestListPartsActionMapping/get_object_without_uploadId (0.00s) --- PASS: TestListPartsActionMapping/get_object_with_uploadId (0.00s) --- PASS: TestListPartsActionMapping/get_object_with_uploadId_and_other_params (0.00s) --- PASS: TestListPartsActionMapping/get_object_with_versionId (0.00s) --- PASS: TestListPartsActionMapping/get_object_acl_without_uploadId (0.00s) --- PASS: TestListPartsActionMapping/post_multipart_upload_without_uploadId (0.00s) === RUN TestListPartsActionMappingSecurityScenarios === RUN TestListPartsActionMappingSecurityScenarios/privilege_separation_listparts_vs_getobject === RUN TestListPartsActionMappingSecurityScenarios/policy_enforcement_precision --- PASS: TestListPartsActionMappingSecurityScenarios (0.02s) --- PASS: TestListPartsActionMappingSecurityScenarios/privilege_separation_listparts_vs_getobject (0.00s) --- PASS: TestListPartsActionMappingSecurityScenarios/policy_enforcement_precision (0.00s) === RUN TestListPartsActionRealWorldScenarios === RUN TestListPartsActionRealWorldScenarios/large_file_upload_workflow === RUN TestListPartsActionRealWorldScenarios/edge_case_upload_ids --- PASS: TestListPartsActionRealWorldScenarios (0.00s) --- PASS: TestListPartsActionRealWorldScenarios/large_file_upload_workflow (0.00s) --- PASS: TestListPartsActionRealWorldScenarios/edge_case_upload_ids (0.00s) === RUN TestCalculateIVWithOffset === RUN TestCalculateIVWithOffset/BlockAligned_0 === RUN TestCalculateIVWithOffset/BlockAligned_16 === RUN TestCalculateIVWithOffset/BlockAligned_32 === RUN TestCalculateIVWithOffset/BlockAligned_48 === RUN TestCalculateIVWithOffset/NonAligned_1 === RUN TestCalculateIVWithOffset/NonAligned_5 === RUN TestCalculateIVWithOffset/NonAligned_10 === RUN TestCalculateIVWithOffset/NonAligned_15 === RUN TestCalculateIVWithOffset/NonAligned_17 === RUN TestCalculateIVWithOffset/NonAligned_21 === RUN TestCalculateIVWithOffset/NonAligned_33 === RUN TestCalculateIVWithOffset/NonAligned_47 === RUN TestCalculateIVWithOffset/LargeOffset --- PASS: TestCalculateIVWithOffset (0.01s) --- PASS: TestCalculateIVWithOffset/BlockAligned_0 (0.00s) --- PASS: TestCalculateIVWithOffset/BlockAligned_16 (0.00s) --- PASS: TestCalculateIVWithOffset/BlockAligned_32 (0.00s) --- PASS: TestCalculateIVWithOffset/BlockAligned_48 (0.00s) --- PASS: TestCalculateIVWithOffset/NonAligned_1 (0.00s) --- PASS: TestCalculateIVWithOffset/NonAligned_5 (0.00s) --- PASS: TestCalculateIVWithOffset/NonAligned_10 (0.00s) --- PASS: TestCalculateIVWithOffset/NonAligned_15 (0.00s) --- PASS: TestCalculateIVWithOffset/NonAligned_17 (0.00s) --- PASS: TestCalculateIVWithOffset/NonAligned_21 (0.00s) --- PASS: TestCalculateIVWithOffset/NonAligned_33 (0.00s) --- PASS: TestCalculateIVWithOffset/NonAligned_47 (0.00s) --- PASS: TestCalculateIVWithOffset/LargeOffset (0.00s) === RUN TestCTRDecryptionWithNonBlockAlignedOffset === RUN TestCTRDecryptionWithNonBlockAlignedOffset/A === RUN TestCTRDecryptionWithNonBlockAlignedOffset/B === RUN TestCTRDecryptionWithNonBlockAlignedOffset/F === RUN TestCTRDecryptionWithNonBlockAlignedOffset/K === RUN TestCTRDecryptionWithNonBlockAlignedOffset/P === RUN TestCTRDecryptionWithNonBlockAlignedOffset/Q === RUN TestCTRDecryptionWithNonBlockAlignedOffset/R === RUN TestCTRDecryptionWithNonBlockAlignedOffset/V === RUN TestCTRDecryptionWithNonBlockAlignedOffset/a === RUN TestCTRDecryptionWithNonBlockAlignedOffset/b === RUN TestCTRDecryptionWithNonBlockAlignedOffset/p === RUN TestCTRDecryptionWithNonBlockAlignedOffset/q === RUN TestCTRDecryptionWithNonBlockAlignedOffset/¥ === RUN TestCTRDecryptionWithNonBlockAlignedOffset/ȵ --- PASS: TestCTRDecryptionWithNonBlockAlignedOffset (0.01s) --- PASS: TestCTRDecryptionWithNonBlockAlignedOffset/A (0.00s) --- PASS: TestCTRDecryptionWithNonBlockAlignedOffset/B (0.00s) --- PASS: TestCTRDecryptionWithNonBlockAlignedOffset/F (0.00s) --- PASS: TestCTRDecryptionWithNonBlockAlignedOffset/K (0.00s) --- PASS: TestCTRDecryptionWithNonBlockAlignedOffset/P (0.00s) --- PASS: TestCTRDecryptionWithNonBlockAlignedOffset/Q (0.00s) --- PASS: TestCTRDecryptionWithNonBlockAlignedOffset/R (0.00s) --- PASS: TestCTRDecryptionWithNonBlockAlignedOffset/V (0.00s) --- PASS: TestCTRDecryptionWithNonBlockAlignedOffset/a (0.00s) --- PASS: TestCTRDecryptionWithNonBlockAlignedOffset/b (0.00s) --- PASS: TestCTRDecryptionWithNonBlockAlignedOffset/p (0.00s) --- PASS: TestCTRDecryptionWithNonBlockAlignedOffset/q (0.00s) --- PASS: TestCTRDecryptionWithNonBlockAlignedOffset/¥ (0.00s) --- PASS: TestCTRDecryptionWithNonBlockAlignedOffset/ȵ (0.00s) === RUN TestCTRRangeRequestSimulation === RUN TestCTRRangeRequestSimulation/First_byte === RUN TestCTRRangeRequestSimulation/First_100_bytes === RUN TestCTRRangeRequestSimulation/Mid-block_range === RUN TestCTRRangeRequestSimulation/Single_mid-block_byte === RUN TestCTRRangeRequestSimulation/Cross-block_range === RUN TestCTRRangeRequestSimulation/Large_range === RUN TestCTRRangeRequestSimulation/Tail_range --- PASS: TestCTRRangeRequestSimulation (0.21s) --- PASS: TestCTRRangeRequestSimulation/First_byte (0.00s) --- PASS: TestCTRRangeRequestSimulation/First_100_bytes (0.00s) --- PASS: TestCTRRangeRequestSimulation/Mid-block_range (0.00s) --- PASS: TestCTRRangeRequestSimulation/Single_mid-block_byte (0.00s) --- PASS: TestCTRRangeRequestSimulation/Cross-block_range (0.00s) --- PASS: TestCTRRangeRequestSimulation/Large_range (0.00s) --- PASS: TestCTRRangeRequestSimulation/Tail_range (0.00s) === RUN TestCTRDecryptionWithIOReader === RUN TestCTRDecryptionWithIOReader/A === RUN TestCTRDecryptionWithIOReader/F === RUN TestCTRDecryptionWithIOReader/K === RUN TestCTRDecryptionWithIOReader/Q === RUN TestCTRDecryptionWithIOReader/R === RUN TestCTRDecryptionWithIOReader/_ --- PASS: TestCTRDecryptionWithIOReader (0.00s) --- PASS: TestCTRDecryptionWithIOReader/A (0.00s) --- PASS: TestCTRDecryptionWithIOReader/F (0.00s) --- PASS: TestCTRDecryptionWithIOReader/K (0.00s) --- PASS: TestCTRDecryptionWithIOReader/Q (0.00s) --- PASS: TestCTRDecryptionWithIOReader/R (0.00s) --- PASS: TestCTRDecryptionWithIOReader/_ (0.00s) === RUN TestSSES3EndToEndSmallFile === RUN TestSSES3EndToEndSmallFile/tiny_file_(10_bytes) === RUN TestSSES3EndToEndSmallFile/small_file_(50_bytes) === RUN TestSSES3EndToEndSmallFile/medium_file_(256_bytes) --- PASS: TestSSES3EndToEndSmallFile (0.01s) --- PASS: TestSSES3EndToEndSmallFile/tiny_file_(10_bytes) (0.00s) --- PASS: TestSSES3EndToEndSmallFile/small_file_(50_bytes) (0.00s) --- PASS: TestSSES3EndToEndSmallFile/medium_file_(256_bytes) (0.00s) === RUN TestSSES3EndToEndChunkedFile --- PASS: TestSSES3EndToEndChunkedFile (0.00s) === RUN TestSSES3EndToEndWithDetectPrimaryType === RUN TestSSES3EndToEndWithDetectPrimaryType/Inline_SSE-S3_file_(no_chunks) === RUN TestSSES3EndToEndWithDetectPrimaryType/Single_chunk_SSE-S3_file === RUN TestSSES3EndToEndWithDetectPrimaryType/SSE-KMS_file_(has_KMS_key_ID) --- PASS: TestSSES3EndToEndWithDetectPrimaryType (0.00s) --- PASS: TestSSES3EndToEndWithDetectPrimaryType/Inline_SSE-S3_file_(no_chunks) (0.00s) --- PASS: TestSSES3EndToEndWithDetectPrimaryType/Single_chunk_SSE-S3_file (0.00s) --- PASS: TestSSES3EndToEndWithDetectPrimaryType/SSE-KMS_file_(has_KMS_key_ID) (0.00s) === RUN TestBuildMultipartSSES3Reader_PerChunkKeys --- PASS: TestBuildMultipartSSES3Reader_PerChunkKeys (0.00s) === RUN TestBuildMultipartSSES3Reader_InvalidIVLength === RUN TestBuildMultipartSSES3Reader_InvalidIVLength/missing_IV === RUN TestBuildMultipartSSES3Reader_InvalidIVLength/short_IV --- PASS: TestBuildMultipartSSES3Reader_InvalidIVLength (0.00s) --- PASS: TestBuildMultipartSSES3Reader_InvalidIVLength/missing_IV (0.00s) --- PASS: TestBuildMultipartSSES3Reader_InvalidIVLength/short_IV (0.00s) === RUN TestBuildMultipartSSES3Reader_RejectsBadChunkBeforeAnyFetch --- PASS: TestBuildMultipartSSES3Reader_RejectsBadChunkBeforeAnyFetch (0.00s) === RUN TestSetKEKPassphraseEnablesEncryptedRoundTrip --- PASS: TestSetKEKPassphraseEnablesEncryptedRoundTrip (0.01s) === RUN TestSetKEKPassphraseDifferentInstancesNoCollision --- PASS: TestSetKEKPassphraseDifferentInstancesNoCollision (0.00s) === RUN TestNoPassphraseKeepsLegacyHexDecodePath --- PASS: TestNoPassphraseKeepsLegacyHexDecodePath (0.00s) === RUN TestSSES3MultipartChunkViewDecryption === RUN TestSSES3MultipartChunkViewDecryption/Part_1 === RUN TestSSES3MultipartChunkViewDecryption/Part_2 --- PASS: TestSSES3MultipartChunkViewDecryption (0.00s) --- PASS: TestSSES3MultipartChunkViewDecryption/Part_1 (0.00s) --- PASS: TestSSES3MultipartChunkViewDecryption/Part_2 (0.00s) === RUN TestSSES3SinglePartChunkViewDecryption --- PASS: TestSSES3SinglePartChunkViewDecryption (0.00s) === RUN TestSSES3IVOffsetCalculation === RUN TestSSES3IVOffsetCalculation/Part_1 === RUN TestSSES3IVOffsetCalculation/Part_2 === RUN TestSSES3IVOffsetCalculation/Part_3 === RUN TestSSES3IVOffsetCalculation/Part_10 --- PASS: TestSSES3IVOffsetCalculation (0.00s) --- PASS: TestSSES3IVOffsetCalculation/Part_1 (0.00s) --- PASS: TestSSES3IVOffsetCalculation/Part_2 (0.00s) --- PASS: TestSSES3IVOffsetCalculation/Part_3 (0.00s) --- PASS: TestSSES3IVOffsetCalculation/Part_10 (0.00s) === RUN TestSSES3ChunkMetadataDetection === RUN TestSSES3ChunkMetadataDetection/Multipart_chunk_with_metadata === RUN TestSSES3ChunkMetadataDetection/Single-part_chunk_without_metadata === RUN TestSSES3ChunkMetadataDetection/Non-SSE-S3_chunk --- PASS: TestSSES3ChunkMetadataDetection (0.01s) --- PASS: TestSSES3ChunkMetadataDetection/Multipart_chunk_with_metadata (0.00s) --- PASS: TestSSES3ChunkMetadataDetection/Single-part_chunk_without_metadata (0.00s) --- PASS: TestSSES3ChunkMetadataDetection/Non-SSE-S3_chunk (0.00s) === RUN TestSSES3EncryptionConsistency --- PASS: TestSSES3EncryptionConsistency (0.00s) === RUN TestS3IAMIntegration_isSTSIssuer === RUN TestS3IAMIntegration_isSTSIssuer/exact_match_with_configured_issuer === RUN TestS3IAMIntegration_isSTSIssuer/similar_but_not_exact_issuer === RUN TestS3IAMIntegration_isSTSIssuer/substring_of_configured_issuer === RUN TestS3IAMIntegration_isSTSIssuer/contains_configured_issuer_as_substring === RUN TestS3IAMIntegration_isSTSIssuer/case_sensitive_-_different_case === RUN TestS3IAMIntegration_isSTSIssuer/Google_OIDC === RUN TestS3IAMIntegration_isSTSIssuer/Azure_AD === RUN TestS3IAMIntegration_isSTSIssuer/Auth0 === RUN TestS3IAMIntegration_isSTSIssuer/Keycloak === RUN TestS3IAMIntegration_isSTSIssuer/Empty_string --- PASS: TestS3IAMIntegration_isSTSIssuer (0.01s) --- PASS: TestS3IAMIntegration_isSTSIssuer/exact_match_with_configured_issuer (0.00s) --- PASS: TestS3IAMIntegration_isSTSIssuer/similar_but_not_exact_issuer (0.00s) --- PASS: TestS3IAMIntegration_isSTSIssuer/substring_of_configured_issuer (0.00s) --- PASS: TestS3IAMIntegration_isSTSIssuer/contains_configured_issuer_as_substring (0.00s) --- PASS: TestS3IAMIntegration_isSTSIssuer/case_sensitive_-_different_case (0.00s) --- PASS: TestS3IAMIntegration_isSTSIssuer/Google_OIDC (0.00s) --- PASS: TestS3IAMIntegration_isSTSIssuer/Azure_AD (0.00s) --- PASS: TestS3IAMIntegration_isSTSIssuer/Auth0 (0.00s) --- PASS: TestS3IAMIntegration_isSTSIssuer/Keycloak (0.00s) --- PASS: TestS3IAMIntegration_isSTSIssuer/Empty_string (0.00s) === RUN TestS3IAMIntegration_isSTSIssuer_NoSTSService --- PASS: TestS3IAMIntegration_isSTSIssuer_NoSTSService (0.00s) === RUN TestVerifyKeyCommitment_DefaultAcceptsMissing --- PASS: TestVerifyKeyCommitment_DefaultAcceptsMissing (0.00s) === RUN TestVerifyKeyCommitment_StrictRejectsMissing --- PASS: TestVerifyKeyCommitment_StrictRejectsMissing (0.00s) === RUN TestVerifyKeyCommitment_StrictAcceptsValidCommitment --- PASS: TestVerifyKeyCommitment_StrictAcceptsValidCommitment (0.00s) === RUN TestVerifyKeyCommitment_RejectsTamperedKey === RUN TestVerifyKeyCommitment_RejectsTamperedKey/tampered_key === RUN TestVerifyKeyCommitment_RejectsTamperedKey/tampered_IV === RUN TestVerifyKeyCommitment_RejectsTamperedKey/tampered_algorithm === RUN TestVerifyKeyCommitment_RejectsTamperedKey/tampered_commitment --- PASS: TestVerifyKeyCommitment_RejectsTamperedKey (0.00s) --- PASS: TestVerifyKeyCommitment_RejectsTamperedKey/tampered_key (0.00s) --- PASS: TestVerifyKeyCommitment_RejectsTamperedKey/tampered_IV (0.00s) --- PASS: TestVerifyKeyCommitment_RejectsTamperedKey/tampered_algorithm (0.00s) --- PASS: TestVerifyKeyCommitment_RejectsTamperedKey/tampered_commitment (0.00s) === RUN TestSetRequireKeyCommitment --- PASS: TestSetRequireKeyCommitment (0.00s) === RUN TestBucketConfigStubs === RUN TestBucketConfigStubs/AnalyticsList === RUN TestBucketConfigStubs/InventoryList === RUN TestBucketConfigStubs/IntelligentTieringList === RUN TestBucketConfigStubs/MetricsList === RUN TestBucketConfigStubs/AnalyticsGet === RUN TestBucketConfigStubs/InventoryGet === RUN TestBucketConfigStubs/IntelligentTieringGet === RUN TestBucketConfigStubs/MetricsGet --- PASS: TestBucketConfigStubs (0.01s) --- PASS: TestBucketConfigStubs/AnalyticsList (0.00s) --- PASS: TestBucketConfigStubs/InventoryList (0.00s) --- PASS: TestBucketConfigStubs/IntelligentTieringList (0.00s) --- PASS: TestBucketConfigStubs/MetricsList (0.00s) --- PASS: TestBucketConfigStubs/AnalyticsGet (0.00s) --- PASS: TestBucketConfigStubs/InventoryGet (0.00s) --- PASS: TestBucketConfigStubs/IntelligentTieringGet (0.00s) --- PASS: TestBucketConfigStubs/MetricsGet (0.00s) === RUN TestUpdateBucketConfigDoesNotMutateCacheOnPersistFailure W0824 22:18:52.354605 s3api_server.go:517 getFilerAddress: no filer addresses available E0824 22:18:52.367623 s3api_bucket_config.go:489 updateBucketConfig: failed to get bucket entry for cleanup-test-net: getOrCreateConnection : fail to dial : grpc: no transport security set (use grpc.WithTransportCredentials(insecure.NewCredentials()) explicitly or set credentials) --- PASS: TestUpdateBucketConfigDoesNotMutateCacheOnPersistFailure (0.02s) === RUN TestGetListBucketsArgs === RUN TestGetListBucketsArgs/defaults === RUN TestGetListBucketsArgs/max-buckets === RUN TestGetListBucketsArgs/max-buckets_upper_bound === RUN TestGetListBucketsArgs/max-buckets_zero === RUN TestGetListBucketsArgs/max-buckets_negative === RUN TestGetListBucketsArgs/max-buckets_too_large === RUN TestGetListBucketsArgs/max-buckets_not_a_number === RUN TestGetListBucketsArgs/prefix === RUN TestGetListBucketsArgs/continuation_token === RUN TestGetListBucketsArgs/bad_continuation_token --- PASS: TestGetListBucketsArgs (0.00s) --- PASS: TestGetListBucketsArgs/defaults (0.00s) --- PASS: TestGetListBucketsArgs/max-buckets (0.00s) --- PASS: TestGetListBucketsArgs/max-buckets_upper_bound (0.00s) --- PASS: TestGetListBucketsArgs/max-buckets_zero (0.00s) --- PASS: TestGetListBucketsArgs/max-buckets_negative (0.00s) --- PASS: TestGetListBucketsArgs/max-buckets_too_large (0.00s) --- PASS: TestGetListBucketsArgs/max-buckets_not_a_number (0.00s) --- PASS: TestGetListBucketsArgs/prefix (0.00s) --- PASS: TestGetListBucketsArgs/continuation_token (0.00s) --- PASS: TestGetListBucketsArgs/bad_continuation_token (0.00s) === RUN TestCanListBucketsFromOwnerIndex === RUN TestCanListBucketsFromOwnerIndex/nil_identity_scans === RUN TestCanListBucketsFromOwnerIndex/admin_scans === RUN TestCanListBucketsFromOwnerIndex/bare_List_scans === RUN TestCanListBucketsFromOwnerIndex/wildcard_scans === RUN TestCanListBucketsFromOwnerIndex/no_actions_is_owned-only === RUN TestCanListBucketsFromOwnerIndex/attached_policy_with_no_loaded_document_scans === RUN TestCanListBucketsFromOwnerIndex/named_grants_enumerate === RUN TestCanListBucketsFromOwnerIndex/unsorted_grants_keep_action_order === RUN TestCanListBucketsFromOwnerIndex/session_token_scans --- PASS: TestCanListBucketsFromOwnerIndex (0.00s) --- PASS: TestCanListBucketsFromOwnerIndex/nil_identity_scans (0.00s) --- PASS: TestCanListBucketsFromOwnerIndex/admin_scans (0.00s) --- PASS: TestCanListBucketsFromOwnerIndex/bare_List_scans (0.00s) --- PASS: TestCanListBucketsFromOwnerIndex/wildcard_scans (0.00s) --- PASS: TestCanListBucketsFromOwnerIndex/no_actions_is_owned-only (0.00s) --- PASS: TestCanListBucketsFromOwnerIndex/attached_policy_with_no_loaded_document_scans (0.00s) --- PASS: TestCanListBucketsFromOwnerIndex/named_grants_enumerate (0.00s) --- PASS: TestCanListBucketsFromOwnerIndex/unsorted_grants_keep_action_order (0.00s) --- PASS: TestCanListBucketsFromOwnerIndex/session_token_scans (0.00s) === RUN TestCanListBucketsFromOwnerIndexAttachedPolicy === RUN TestCanListBucketsFromOwnerIndexAttachedPolicy/named_bucket_enumerates === RUN TestCanListBucketsFromOwnerIndexAttachedPolicy/wildcard_resource_scans === RUN TestCanListBucketsFromOwnerIndexAttachedPolicy/not-resource_scans === RUN TestCanListBucketsFromOwnerIndexAttachedPolicy/unknown_policy_scans === RUN TestCanListBucketsFromOwnerIndexAttachedPolicy/one_wildcard_policy_scans === RUN TestCanListBucketsFromOwnerIndexAttachedPolicy/action_wildcard_on_a_named_bucket_enumerates === RUN TestCanListBucketsFromOwnerIndexAttachedPolicy/denied_bucket_stays_a_candidate === RUN TestCanListBucketsFromOwnerIndexAttachedPolicy/group_policy_contributes === RUN TestCanListBucketsFromOwnerIndexAttachedPolicy/disabled_group_is_ignored === RUN TestCanListBucketsFromOwnerIndexAttachedPolicy/resource_variable_scans === RUN TestCanListBucketsFromOwnerIndexAttachedPolicy/action_variable_keeps_its_bucket === RUN TestCanListBucketsFromOwnerIndexAttachedPolicy/mixed_case_action_enumerates === RUN TestCanListBucketsFromOwnerIndexAttachedPolicy/mixed_case_wildcard_action_enumerates --- PASS: TestCanListBucketsFromOwnerIndexAttachedPolicy (0.01s) --- PASS: TestCanListBucketsFromOwnerIndexAttachedPolicy/named_bucket_enumerates (0.00s) --- PASS: TestCanListBucketsFromOwnerIndexAttachedPolicy/wildcard_resource_scans (0.00s) --- PASS: TestCanListBucketsFromOwnerIndexAttachedPolicy/not-resource_scans (0.00s) --- PASS: TestCanListBucketsFromOwnerIndexAttachedPolicy/unknown_policy_scans (0.00s) --- PASS: TestCanListBucketsFromOwnerIndexAttachedPolicy/one_wildcard_policy_scans (0.00s) --- PASS: TestCanListBucketsFromOwnerIndexAttachedPolicy/action_wildcard_on_a_named_bucket_enumerates (0.00s) --- PASS: TestCanListBucketsFromOwnerIndexAttachedPolicy/denied_bucket_stays_a_candidate (0.00s) --- PASS: TestCanListBucketsFromOwnerIndexAttachedPolicy/group_policy_contributes (0.00s) --- PASS: TestCanListBucketsFromOwnerIndexAttachedPolicy/disabled_group_is_ignored (0.00s) --- PASS: TestCanListBucketsFromOwnerIndexAttachedPolicy/resource_variable_scans (0.00s) --- PASS: TestCanListBucketsFromOwnerIndexAttachedPolicy/action_variable_keeps_its_bucket (0.00s) --- PASS: TestCanListBucketsFromOwnerIndexAttachedPolicy/mixed_case_action_enumerates (0.00s) --- PASS: TestCanListBucketsFromOwnerIndexAttachedPolicy/mixed_case_wildcard_action_enumerates (0.00s) === RUN TestContinuationTokenRoundTrip --- PASS: TestContinuationTokenRoundTrip (0.00s) === RUN TestHasExplicitBucketACL === RUN TestHasExplicitBucketACL/none === RUN TestHasExplicitBucketACL/private_is_default === RUN TestHasExplicitBucketACL/canned_public-read === RUN TestHasExplicitBucketACL/canned_case-insensitive_private === RUN TestHasExplicitBucketACL/grant_read === RUN TestHasExplicitBucketACL/grant_full_control --- PASS: TestHasExplicitBucketACL (0.00s) --- PASS: TestHasExplicitBucketACL/none (0.00s) --- PASS: TestHasExplicitBucketACL/private_is_default (0.00s) --- PASS: TestHasExplicitBucketACL/canned_public-read (0.00s) --- PASS: TestHasExplicitBucketACL/canned_case-insensitive_private (0.00s) --- PASS: TestHasExplicitBucketACL/grant_read (0.00s) --- PASS: TestHasExplicitBucketACL/grant_full_control (0.00s) === RUN TestGetBucketPolicyStatusIsPublic === RUN TestGetBucketPolicyStatusIsPublic/public_allow_star === RUN TestGetBucketPolicyStatusIsPublic/deny_is_not_public === RUN TestGetBucketPolicyStatusIsPublic/condition_makes_it_non-public === RUN TestGetBucketPolicyStatusIsPublic/specific_principal_is_not_public --- PASS: TestGetBucketPolicyStatusIsPublic (0.00s) --- PASS: TestGetBucketPolicyStatusIsPublic/public_allow_star (0.00s) --- PASS: TestGetBucketPolicyStatusIsPublic/deny_is_not_public (0.00s) --- PASS: TestGetBucketPolicyStatusIsPublic/condition_makes_it_non-public (0.00s) --- PASS: TestGetBucketPolicyStatusIsPublic/specific_principal_is_not_public (0.00s) === RUN TestPutBucketRequestPaymentBucketOwner --- PASS: TestPutBucketRequestPaymentBucketOwner (0.00s) === RUN TestPutBucketRequestPaymentRequesterRejected --- PASS: TestPutBucketRequestPaymentRequesterRejected (0.00s) === RUN TestPutBucketOwnershipControlsRejectsRuleWithoutObjectOwnership --- PASS: TestPutBucketOwnershipControlsRejectsRuleWithoutObjectOwnership (0.00s) === RUN TestGetBucketOwnershipControlsDefaultsToBucketOwnerEnforced --- PASS: TestGetBucketOwnershipControlsDefaultsToBucketOwnerEnforced (0.00s) === RUN TestGetBucketAccelerateConfiguration --- PASS: TestGetBucketAccelerateConfiguration (0.00s) === RUN TestGetBucketLogging --- PASS: TestGetBucketLogging (0.00s) === RUN TestHandleAutoCreateBucketDisabled W0824 22:18:52.408631 s3api_bucket_handlers.go:766 PutObjectHandler: auto-create bucket test-bucket: bucket auto-creation is disabled --- PASS: TestHandleAutoCreateBucketDisabled (0.00s) === RUN TestHandleAutoCreateBucketNonAdmin W0824 22:18:52.409782 s3api_bucket_handlers.go:766 PutObjectHandler: auto-create bucket test-bucket: permission denied - requires Admin permission --- PASS: TestHandleAutoCreateBucketNonAdmin (0.00s) === RUN TestUploadMissingBucketAutoCreateDisabled === RUN TestUploadMissingBucketAutoCreateDisabled/put_object W0824 22:18:52.411512 s3api_server.go:517 getFilerAddress: no filer addresses available W0824 22:18:52.412218 s3api_bucket_handlers.go:766 PutObjectHandler: auto-create bucket missing: bucket auto-creation is disabled === RUN TestUploadMissingBucketAutoCreateDisabled/put_directory_marker W0824 22:18:52.413499 s3api_server.go:517 getFilerAddress: no filer addresses available W0824 22:18:52.413984 s3api_bucket_handlers.go:766 PutObjectHandler: auto-create bucket missing: bucket auto-creation is disabled === RUN TestUploadMissingBucketAutoCreateDisabled/new_multipart_upload W0824 22:18:52.417983 s3api_bucket_handlers.go:766 NewMultipartUploadHandler: auto-create bucket missing: bucket auto-creation is disabled --- PASS: TestUploadMissingBucketAutoCreateDisabled (0.01s) --- PASS: TestUploadMissingBucketAutoCreateDisabled/put_object (0.00s) --- PASS: TestUploadMissingBucketAutoCreateDisabled/put_directory_marker (0.00s) --- PASS: TestUploadMissingBucketAutoCreateDisabled/new_multipart_upload (0.00s) === RUN TestGetBucketLifecycleConfigurationHandlerUsesStoredLifecycleConfig --- PASS: TestGetBucketLifecycleConfigurationHandlerUsesStoredLifecycleConfig (0.00s) === RUN TestGetBucketLifecycleConfigurationHandlerDefaultsTransitionMinimumObjectSize --- PASS: TestGetBucketLifecycleConfigurationHandlerDefaultsTransitionMinimumObjectSize (0.00s) === RUN TestPutBucketLifecycleConfigurationHandlerRejectsOversizedBody W0824 22:18:52.430037 s3api_bucket_handlers.go:1098 PutBucketLifecycleConfigurationHandler read body: http: request body too large --- PASS: TestPutBucketLifecycleConfigurationHandlerRejectsOversizedBody (0.01s) === RUN TestPutBucketLifecycleConfigurationHandlerMapsReadErrorsToInvalidRequest W0824 22:18:52.432072 s3api_bucket_handlers.go:1098 PutBucketLifecycleConfigurationHandler read body: read failed --- PASS: TestPutBucketLifecycleConfigurationHandlerMapsReadErrorsToInvalidRequest (0.00s) === RUN TestBucketMetadataStruct --- PASS: TestBucketMetadataStruct (0.00s) === RUN TestBucketMetadataUpdatePattern --- PASS: TestBucketMetadataUpdatePattern (0.00s) === RUN TestBucketMetadataHelperFunctions --- PASS: TestBucketMetadataHelperFunctions (0.00s) === RUN TestEscapeOwnerName --- PASS: TestEscapeOwnerName (0.00s) === RUN TestListMergedBuckets === RUN TestListMergedBuckets/interleave_and_dedup === RUN TestListMergedBuckets/truncation_and_resume === RUN TestListMergedBuckets/no_owned_stream === RUN TestListMergedBuckets/owned_only --- PASS: TestListMergedBuckets (0.00s) --- PASS: TestListMergedBuckets/interleave_and_dedup (0.00s) --- PASS: TestListMergedBuckets/truncation_and_resume (0.00s) --- PASS: TestListMergedBuckets/no_owned_stream (0.00s) --- PASS: TestListMergedBuckets/owned_only (0.00s) === RUN TestBucketEntryOwner --- PASS: TestBucketEntryOwner (0.00s) === RUN TestBuildResourceARN === RUN TestBuildResourceARN/bucket_only === RUN TestBuildResourceARN/bucket_with_slash === RUN TestBuildResourceARN/bucket_and_object === RUN TestBuildResourceARN/bucket_and_object_with_leading_slash --- PASS: TestBuildResourceARN (0.00s) --- PASS: TestBuildResourceARN/bucket_only (0.00s) --- PASS: TestBuildResourceARN/bucket_with_slash (0.00s) --- PASS: TestBuildResourceARN/bucket_and_object (0.00s) --- PASS: TestBuildResourceARN/bucket_and_object_with_leading_slash (0.00s) === RUN TestBuildPrincipalARN === RUN TestBuildPrincipalARN/nil_identity_(anonymous) === RUN TestBuildPrincipalARN/explicit_principal_ARN === RUN TestBuildPrincipalARN/anonymous_user_by_name === RUN TestBuildPrincipalARN/anonymous_user_by_account_ID === RUN TestBuildPrincipalARN/identity_with_account_and_name === RUN TestBuildPrincipalARN/identity_without_account_ID === RUN TestBuildPrincipalARN/identity_without_name --- PASS: TestBuildPrincipalARN (0.00s) --- PASS: TestBuildPrincipalARN/nil_identity_(anonymous) (0.00s) --- PASS: TestBuildPrincipalARN/explicit_principal_ARN (0.00s) --- PASS: TestBuildPrincipalARN/anonymous_user_by_name (0.00s) --- PASS: TestBuildPrincipalARN/anonymous_user_by_account_ID (0.00s) --- PASS: TestBuildPrincipalARN/identity_with_account_and_name (0.00s) --- PASS: TestBuildPrincipalARN/identity_without_account_ID (0.00s) --- PASS: TestBuildPrincipalARN/identity_without_name (0.00s) === RUN TestBucketPolicyRoutesUseTheirOwnActions --- PASS: TestBucketPolicyRoutesUseTheirOwnActions (0.02s) === RUN TestBucketPolicyWriteIsNotImpliedByObjectWrite --- PASS: TestBucketPolicyWriteIsNotImpliedByObjectWrite (0.00s) === RUN TestBucketPolicyWriteAllowedForAdmin === RUN TestBucketPolicyWriteAllowedForAdmin/global_admin === RUN TestBucketPolicyWriteAllowedForAdmin/bucket_admin === RUN TestBucketPolicyWriteAllowedForAdmin/explicit_delegation --- PASS: TestBucketPolicyWriteAllowedForAdmin (0.00s) --- PASS: TestBucketPolicyWriteAllowedForAdmin/global_admin (0.00s) --- PASS: TestBucketPolicyWriteAllowedForAdmin/bucket_admin (0.00s) --- PASS: TestBucketPolicyWriteAllowedForAdmin/explicit_delegation (0.00s) === RUN TestBucketPolicyActionsAreGrantableViaIamPolicy --- PASS: TestBucketPolicyActionsAreGrantableViaIamPolicy (0.00s) === RUN TestBucketPolicyActionsMapToAwsActionNames --- PASS: TestBucketPolicyActionsMapToAwsActionNames (0.00s) === RUN TestUnroutedBucketSubresource --- PASS: TestUnroutedBucketSubresource (0.00s) === RUN TestManifestizeOrKeepFlatFolds --- PASS: TestManifestizeOrKeepFlatFolds (0.05s) === RUN TestManifestizeOrKeepFlatBelowThreshold --- PASS: TestManifestizeOrKeepFlatBelowThreshold (0.02s) === RUN TestManifestizeOrKeepFlatSkipsSse --- PASS: TestManifestizeOrKeepFlatSkipsSse (0.05s) === RUN TestManifestizeOrKeepFlatRollsBackPartialFold I0824 22:18:52.702273 s3api_chunk_manifest.go:52 MaybeManifestize /buckets/b/o: save 2 failed --- PASS: TestManifestizeOrKeepFlatRollsBackPartialFold (0.12s) === RUN TestPartRange --- PASS: TestPartRange (0.00s) === RUN TestLimit --- PASS: TestLimit (0.00s) === RUN TestLimitInterceptor === RUN TestLimitInterceptor/nil_interceptor_is_a_no-op === RUN TestLimitInterceptor/interceptor_can_reject === RUN TestLimitInterceptor/interceptor_can_pass_through === RUN TestLimitInterceptor/interceptor_installed_after_registration_takes_effect --- PASS: TestLimitInterceptor (0.00s) --- PASS: TestLimitInterceptor/nil_interceptor_is_a_no-op (0.00s) --- PASS: TestLimitInterceptor/interceptor_can_reject (0.00s) --- PASS: TestLimitInterceptor/interceptor_can_pass_through (0.00s) --- PASS: TestLimitInterceptor/interceptor_installed_after_registration_takes_effect (0.00s) === RUN TestClassifyCopySourceError === RUN TestClassifyCopySourceError/regular_entry === RUN TestClassifyCopySourceError/nil_entry === RUN TestClassifyCopySourceError/directory_entry === RUN TestClassifyCopySourceError/delete_marker_entry === RUN TestClassifyCopySourceError/not_found_sentinel === RUN TestClassifyCopySourceError/wrapped_not_found === RUN TestClassifyCopySourceError/grpc_not_found === RUN TestClassifyCopySourceError/invalid_version_id === RUN TestClassifyCopySourceError/delete_marker_error === RUN TestClassifyCopySourceError/transient_unavailable === RUN TestClassifyCopySourceError/store_error === RUN TestClassifyCopySourceError/sentinel_text_in_message --- PASS: TestClassifyCopySourceError (0.00s) --- PASS: TestClassifyCopySourceError/regular_entry (0.00s) --- PASS: TestClassifyCopySourceError/nil_entry (0.00s) --- PASS: TestClassifyCopySourceError/directory_entry (0.00s) --- PASS: TestClassifyCopySourceError/delete_marker_entry (0.00s) --- PASS: TestClassifyCopySourceError/not_found_sentinel (0.00s) --- PASS: TestClassifyCopySourceError/wrapped_not_found (0.00s) --- PASS: TestClassifyCopySourceError/grpc_not_found (0.00s) --- PASS: TestClassifyCopySourceError/invalid_version_id (0.00s) --- PASS: TestClassifyCopySourceError/delete_marker_error (0.00s) --- PASS: TestClassifyCopySourceError/transient_unavailable (0.00s) --- PASS: TestClassifyCopySourceError/store_error (0.00s) --- PASS: TestClassifyCopySourceError/sentinel_text_in_message (0.00s) === RUN TestClassifyDomainNames === RUN TestClassifyDomainNames/Mixed_path-style_and_virtual-host_with_single_parent === RUN TestClassifyDomainNames/Multiple_subdomains_with_same_parent === RUN TestClassifyDomainNames/Subdomain_without_parent_in_list === RUN TestClassifyDomainNames/Only_top-level_domain === RUN TestClassifyDomainNames/Multiple_independent_domains === RUN TestClassifyDomainNames/Mixed_with_nested_levels === RUN TestClassifyDomainNames/Domain_without_dot === RUN TestClassifyDomainNames/Empty_list === RUN TestClassifyDomainNames/Mixed_localhost_and_domain === RUN TestClassifyDomainNames/Three-level_subdomain_hierarchy === RUN TestClassifyDomainNames/Real-world_example_from_issue_#7356 --- PASS: TestClassifyDomainNames (0.00s) --- PASS: TestClassifyDomainNames/Mixed_path-style_and_virtual-host_with_single_parent (0.00s) --- PASS: TestClassifyDomainNames/Multiple_subdomains_with_same_parent (0.00s) --- PASS: TestClassifyDomainNames/Subdomain_without_parent_in_list (0.00s) --- PASS: TestClassifyDomainNames/Only_top-level_domain (0.00s) --- PASS: TestClassifyDomainNames/Multiple_independent_domains (0.00s) --- PASS: TestClassifyDomainNames/Mixed_with_nested_levels (0.00s) --- PASS: TestClassifyDomainNames/Domain_without_dot (0.00s) --- PASS: TestClassifyDomainNames/Empty_list (0.00s) --- PASS: TestClassifyDomainNames/Mixed_localhost_and_domain (0.00s) --- PASS: TestClassifyDomainNames/Three-level_subdomain_hierarchy (0.00s) --- PASS: TestClassifyDomainNames/Real-world_example_from_issue_#7356 (0.00s) === RUN TestClassifyDomainNamesOrder === RUN TestClassifyDomainNamesOrder/Parent_before_child === RUN TestClassifyDomainNamesOrder/Child_before_parent === RUN TestClassifyDomainNamesOrder/Mixed_order_with_multiple_children --- PASS: TestClassifyDomainNamesOrder (0.00s) --- PASS: TestClassifyDomainNamesOrder/Parent_before_child (0.00s) --- PASS: TestClassifyDomainNamesOrder/Child_before_parent (0.00s) --- PASS: TestClassifyDomainNamesOrder/Mixed_order_with_multiple_children (0.00s) === RUN TestClassifyDomainNamesEdgeCases === RUN TestClassifyDomainNamesEdgeCases/Duplicate_domains === RUN TestClassifyDomainNamesEdgeCases/Very_long_domain_name === RUN TestClassifyDomainNamesEdgeCases/Similar_but_different_domains === RUN TestClassifyDomainNamesEdgeCases/IP_address_as_domain --- PASS: TestClassifyDomainNamesEdgeCases (0.00s) --- PASS: TestClassifyDomainNamesEdgeCases/Duplicate_domains (0.00s) --- PASS: TestClassifyDomainNamesEdgeCases/Very_long_domain_name (0.00s) --- PASS: TestClassifyDomainNamesEdgeCases/Similar_but_different_domains (0.00s) --- PASS: TestClassifyDomainNamesEdgeCases/IP_address_as_domain (0.00s) === RUN TestClassifyDomainNamesUseCases === RUN TestClassifyDomainNamesUseCases/Issue_#7356_-_Prometheus_blackbox_exporter_scenario === RUN TestClassifyDomainNamesUseCases/Multi-environment_setup === RUN TestClassifyDomainNamesUseCases/Mixed_production_setup --- PASS: TestClassifyDomainNamesUseCases (0.00s) --- PASS: TestClassifyDomainNamesUseCases/Issue_#7356_-_Prometheus_blackbox_exporter_scenario (0.00s) --- PASS: TestClassifyDomainNamesUseCases/Multi-environment_setup (0.00s) --- PASS: TestClassifyDomainNamesUseCases/Mixed_production_setup (0.00s) === RUN TestAuthorizeIamActionDeniesDataPlanePolicy === RUN TestAuthorizeIamActionDeniesDataPlanePolicy/query= === RUN TestAuthorizeIamActionDeniesDataPlanePolicy/query=delete === RUN TestAuthorizeIamActionDeniesDataPlanePolicy/query=acl === RUN TestAuthorizeIamActionDeniesDataPlanePolicy/query=tagging === RUN TestAuthorizeIamActionDeniesDataPlanePolicy/query=policy === RUN TestAuthorizeIamActionDeniesDataPlanePolicy/query=versions --- PASS: TestAuthorizeIamActionDeniesDataPlanePolicy (0.03s) --- PASS: TestAuthorizeIamActionDeniesDataPlanePolicy/query= (0.01s) --- PASS: TestAuthorizeIamActionDeniesDataPlanePolicy/query=delete (0.00s) --- PASS: TestAuthorizeIamActionDeniesDataPlanePolicy/query=acl (0.00s) --- PASS: TestAuthorizeIamActionDeniesDataPlanePolicy/query=tagging (0.00s) --- PASS: TestAuthorizeIamActionDeniesDataPlanePolicy/query=policy (0.00s) --- PASS: TestAuthorizeIamActionDeniesDataPlanePolicy/query=versions (0.00s) === RUN TestAuthorizeIamActionAllowsIamPolicy === RUN TestAuthorizeIamActionAllowsIamPolicy/query= === RUN TestAuthorizeIamActionAllowsIamPolicy/query=delete --- PASS: TestAuthorizeIamActionAllowsIamPolicy (0.00s) --- PASS: TestAuthorizeIamActionAllowsIamPolicy/query= (0.00s) --- PASS: TestAuthorizeIamActionAllowsIamPolicy/query=delete (0.00s) === RUN TestAuthorizeIamActionSelfServiceAndAdmin --- PASS: TestAuthorizeIamActionSelfServiceAndAdmin (0.00s) === RUN TestAuthorizeIamActionDeniesUngrantedIdentity --- PASS: TestAuthorizeIamActionDeniesUngrantedIdentity (0.00s) === RUN TestAuthorizeIamActionDeniesAnonymous --- PASS: TestAuthorizeIamActionDeniesAnonymous (0.00s) === RUN TestEmbeddedIamCreateUser --- PASS: TestEmbeddedIamCreateUser (0.04s) === RUN TestEmbeddedIamCreateUserDoesNotSaveAllUsers --- PASS: TestEmbeddedIamCreateUserDoesNotSaveAllUsers (0.01s) === RUN TestEmbeddedIamCreateUserSkipPersist --- PASS: TestEmbeddedIamCreateUserSkipPersist (0.00s) === RUN TestEmbeddedIamListUsers --- PASS: TestEmbeddedIamListUsers (0.01s) === RUN TestEmbeddedIamListAccessKeys --- PASS: TestEmbeddedIamListAccessKeys (0.01s) === RUN TestEmbeddedIamGetUser --- PASS: TestEmbeddedIamGetUser (0.01s) === RUN TestEmbeddedIamGetUserImplicitUsername --- PASS: TestEmbeddedIamGetUserImplicitUsername (0.01s) === RUN TestEmbeddedIamCreatePolicy --- PASS: TestEmbeddedIamCreatePolicy (0.01s) === RUN TestEmbeddedIamCreatePolicyVersion --- PASS: TestEmbeddedIamCreatePolicyVersion (0.01s) === RUN TestEmbeddedIamCreatePolicyVersionMissingPolicy E0824 22:18:52.892505 s3api_embedded_iam.go:2768 CreatePolicyVersion: policy does-not-exist not found E0824 22:18:52.892849 s3api_embedded_iam.go:208 IAM Response policy does-not-exist not found --- PASS: TestEmbeddedIamCreatePolicyVersionMissingPolicy (0.01s) === RUN TestEmbeddedIamCreatePolicyVersionRequiresSetAsDefault E0824 22:18:52.902214 s3api_embedded_iam.go:2768 CreatePolicyVersion: SetAsDefault must be true: SeaweedFS stores a single managed policy version E0824 22:18:52.902600 s3api_embedded_iam.go:208 IAM Response SetAsDefault must be true: SeaweedFS stores a single managed policy version --- PASS: TestEmbeddedIamCreatePolicyVersionRequiresSetAsDefault (0.01s) === RUN TestEmbeddedIamPutUserPolicy --- PASS: TestEmbeddedIamPutUserPolicy (0.01s) === RUN TestEmbeddedIamPutUserPolicyError E0824 22:18:52.917392 s3api_embedded_iam.go:2673 PutUserPolicy: the user with name InvalidUser cannot be found E0824 22:18:52.917911 s3api_embedded_iam.go:208 IAM Response the user with name InvalidUser cannot be found --- PASS: TestEmbeddedIamPutUserPolicyError (0.00s) === RUN TestEmbeddedIamGetUserPolicy --- PASS: TestEmbeddedIamGetUserPolicy (0.00s) === RUN TestEmbeddedIamDeleteUserPolicy --- PASS: TestEmbeddedIamDeleteUserPolicy (0.00s) === RUN TestEmbeddedIamDeleteUserPolicyUserNotFound E0824 22:18:52.930478 s3api_embedded_iam.go:208 IAM Response the user with name NonExistentUser cannot be found. --- PASS: TestEmbeddedIamDeleteUserPolicyUserNotFound (0.00s) === RUN TestEmbeddedIamListUserPolicies E0824 22:18:52.937625 s3api_embedded_iam.go:208 IAM Response the user with name NonExistentUser cannot be found. --- PASS: TestEmbeddedIamListUserPolicies (0.01s) === RUN TestEmbeddedIamGroupInlinePolicies_CRUD --- PASS: TestEmbeddedIamGroupInlinePolicies_CRUD (0.00s) === RUN TestEmbeddedIamAttachUserPolicy W0824 22:18:52.942966 auth_credentials.go:3053 IAM policy cache rebuild: skipping invalid policy "TestManagedPolicy": invalid policy: invalid policy: policy must contain at least one statement W0824 22:18:52.943978 auth_credentials.go:3053 IAM policy cache rebuild: skipping invalid policy "TestManagedPolicy": invalid policy: invalid policy: policy must contain at least one statement --- PASS: TestEmbeddedIamAttachUserPolicy (0.00s) === RUN TestEmbeddedIamAttachUserPolicyRefreshesIAM --- PASS: TestEmbeddedIamAttachUserPolicyRefreshesIAM (0.00s) === RUN TestEmbeddedIamAttachUserPolicyNoSuchPolicy E0824 22:18:52.950363 s3api_embedded_iam.go:208 IAM Response policy DoesNotExist not found --- PASS: TestEmbeddedIamAttachUserPolicyNoSuchPolicy (0.00s) === RUN TestEmbeddedIamDetachUserPolicy W0824 22:18:52.954343 auth_credentials.go:3053 IAM policy cache rebuild: skipping invalid policy "TestManagedPolicy": invalid policy: invalid policy: policy must contain at least one statement W0824 22:18:52.955193 auth_credentials.go:3053 IAM policy cache rebuild: skipping invalid policy "KeepPolicy": invalid policy: invalid policy: policy must contain at least one statement W0824 22:18:52.956052 auth_credentials.go:3053 IAM policy cache rebuild: skipping invalid policy "TestManagedPolicy": invalid policy: invalid policy: policy must contain at least one statement W0824 22:18:52.956534 auth_credentials.go:3053 IAM policy cache rebuild: skipping invalid policy "KeepPolicy": invalid policy: invalid policy: policy must contain at least one statement --- PASS: TestEmbeddedIamDetachUserPolicy (0.01s) === RUN TestEmbeddedIamDeletePolicyInUse E0824 22:18:52.960112 s3api_embedded_iam.go:2665 DeletePolicy: policy TestPolicy is attached to user TestUser E0824 22:18:52.960372 s3api_embedded_iam.go:208 IAM Response policy TestPolicy is attached to user TestUser --- PASS: TestEmbeddedIamDeletePolicyInUse (0.00s) === RUN TestEmbeddedIamAttachAlreadyAttachedPolicy W0824 22:18:52.964612 auth_credentials.go:3053 IAM policy cache rebuild: skipping invalid policy "TestManagedPolicy": invalid policy: invalid policy: policy must contain at least one statement --- PASS: TestEmbeddedIamAttachAlreadyAttachedPolicy (0.00s) === RUN TestEmbeddedIamDetachNotAttachedPolicy E0824 22:18:52.968035 s3api_embedded_iam.go:208 IAM Response policy MissingPolicy not attached to user TestUser --- PASS: TestEmbeddedIamDetachNotAttachedPolicy (0.00s) === RUN TestEmbeddedIamAttachPolicyLimitExceeded E0824 22:18:52.971824 s3api_embedded_iam.go:208 IAM Response cannot attach more than 10 managed policies to user TestUser --- PASS: TestEmbeddedIamAttachPolicyLimitExceeded (0.00s) === RUN TestEmbeddedIamListAttachedUserPolicies --- PASS: TestEmbeddedIamListAttachedUserPolicies (0.00s) === RUN TestEmbeddedIamUpdateUser --- PASS: TestEmbeddedIamUpdateUser (0.00s) === RUN TestEmbeddedIamUpdateUserPreservesInlinePolicies --- PASS: TestEmbeddedIamUpdateUserPreservesInlinePolicies (0.00s) === RUN TestEmbeddedIamDeleteUser --- PASS: TestEmbeddedIamDeleteUser (0.00s) === RUN TestEmbeddedIamCreateAccessKey --- PASS: TestEmbeddedIamCreateAccessKey (0.00s) === RUN TestEmbeddedIamCreateAccessKeyRejectsMissingUser E0824 22:18:52.993767 s3api_embedded_iam.go:2648 CreateAccessKey: the user with name GhostUser cannot be found. E0824 22:18:52.994037 s3api_embedded_iam.go:208 IAM Response the user with name GhostUser cannot be found. --- PASS: TestEmbeddedIamCreateAccessKeyRejectsMissingUser (0.00s) === RUN TestEmbeddedIamCreateAccessKeyWithCallerSuppliedKeys --- PASS: TestEmbeddedIamCreateAccessKeyWithCallerSuppliedKeys (0.00s) === RUN TestEmbeddedIamCreateAccessKeyRejectsWeakKeys E0824 22:18:52.999470 s3api_embedded_iam.go:2648 CreateAccessKey: AccessKeyId must be 4 to 128 alphanumeric characters E0824 22:18:52.999623 s3api_embedded_iam.go:208 IAM Response AccessKeyId must be 4 to 128 alphanumeric characters E0824 22:18:53.000176 s3api_embedded_iam.go:2648 CreateAccessKey: SecretAccessKey must be between 8 and 128 characters E0824 22:18:53.000276 s3api_embedded_iam.go:208 IAM Response SecretAccessKey must be between 8 and 128 characters E0824 22:18:53.001600 s3api_embedded_iam.go:2648 CreateAccessKey: AccessKeyId must be 4 to 128 alphanumeric characters E0824 22:18:53.001752 s3api_embedded_iam.go:208 IAM Response AccessKeyId must be 4 to 128 alphanumeric characters --- PASS: TestEmbeddedIamCreateAccessKeyRejectsWeakKeys (0.00s) === RUN TestEmbeddedIamCreateAccessKeyRejectsCollision E0824 22:18:53.003128 s3api_embedded_iam.go:2648 CreateAccessKey: AccessKeyId is already in use E0824 22:18:53.003276 s3api_embedded_iam.go:208 IAM Response AccessKeyId is already in use --- PASS: TestEmbeddedIamCreateAccessKeyRejectsCollision (0.00s) === RUN TestEmbeddedIamCreateAccessKeyRejectsPartialSupply E0824 22:18:53.004818 s3api_embedded_iam.go:2648 CreateAccessKey: AccessKeyId and SecretAccessKey must be supplied together E0824 22:18:53.004991 s3api_embedded_iam.go:208 IAM Response AccessKeyId and SecretAccessKey must be supplied together E0824 22:18:53.005445 s3api_embedded_iam.go:2648 CreateAccessKey: AccessKeyId and SecretAccessKey must be supplied together E0824 22:18:53.005560 s3api_embedded_iam.go:208 IAM Response AccessKeyId and SecretAccessKey must be supplied together --- PASS: TestEmbeddedIamCreateAccessKeyRejectsPartialSupply (0.00s) === RUN TestEmbeddedIamCreateAccessKeyBoundary E0824 22:18:53.007340 s3api_embedded_iam.go:2648 CreateAccessKey: AccessKeyId must be 4 to 128 alphanumeric characters E0824 22:18:53.007480 s3api_embedded_iam.go:208 IAM Response AccessKeyId must be 4 to 128 alphanumeric characters E0824 22:18:53.009018 s3api_embedded_iam.go:2648 CreateAccessKey: AccessKeyId must be 4 to 128 alphanumeric characters E0824 22:18:53.009924 s3api_embedded_iam.go:208 IAM Response AccessKeyId must be 4 to 128 alphanumeric characters E0824 22:18:53.010555 s3api_embedded_iam.go:2648 CreateAccessKey: SecretAccessKey must be between 8 and 128 characters E0824 22:18:53.010755 s3api_embedded_iam.go:208 IAM Response SecretAccessKey must be between 8 and 128 characters E0824 22:18:53.012663 s3api_embedded_iam.go:2648 CreateAccessKey: SecretAccessKey must be between 8 and 128 characters E0824 22:18:53.012872 s3api_embedded_iam.go:208 IAM Response SecretAccessKey must be between 8 and 128 characters --- PASS: TestEmbeddedIamCreateAccessKeyBoundary (0.01s) === RUN TestEmbeddedIamDeleteAccessKey --- PASS: TestEmbeddedIamDeleteAccessKey (0.00s) === RUN TestEmbeddedIamHandleImplicitUsername --- PASS: TestEmbeddedIamHandleImplicitUsername (0.00s) === RUN TestEmbeddedIamFullWorkflow === RUN TestEmbeddedIamFullWorkflow/CreateUser === RUN TestEmbeddedIamFullWorkflow/CreateAccessKey === RUN TestEmbeddedIamFullWorkflow/PutUserPolicy === RUN TestEmbeddedIamFullWorkflow/ListUsers === RUN TestEmbeddedIamFullWorkflow/DeleteUser --- PASS: TestEmbeddedIamFullWorkflow (0.02s) --- PASS: TestEmbeddedIamFullWorkflow/CreateUser (0.00s) --- PASS: TestEmbeddedIamFullWorkflow/CreateAccessKey (0.00s) --- PASS: TestEmbeddedIamFullWorkflow/PutUserPolicy (0.00s) --- PASS: TestEmbeddedIamFullWorkflow/ListUsers (0.01s) --- PASS: TestEmbeddedIamFullWorkflow/DeleteUser (0.00s) === RUN TestIamStringSlicesEqual === RUN TestIamStringSlicesEqual/both_empty === RUN TestIamStringSlicesEqual/both_nil === RUN TestIamStringSlicesEqual/same_elements_same_order === RUN TestIamStringSlicesEqual/same_elements_different_order === RUN TestIamStringSlicesEqual/different_lengths === RUN TestIamStringSlicesEqual/different_elements === RUN TestIamStringSlicesEqual/one_empty_one_not === RUN TestIamStringSlicesEqual/duplicates_same --- PASS: TestIamStringSlicesEqual (0.00s) --- PASS: TestIamStringSlicesEqual/both_empty (0.00s) --- PASS: TestIamStringSlicesEqual/both_nil (0.00s) --- PASS: TestIamStringSlicesEqual/same_elements_same_order (0.00s) --- PASS: TestIamStringSlicesEqual/same_elements_different_order (0.00s) --- PASS: TestIamStringSlicesEqual/different_lengths (0.00s) --- PASS: TestIamStringSlicesEqual/different_elements (0.00s) --- PASS: TestIamStringSlicesEqual/one_empty_one_not (0.00s) --- PASS: TestIamStringSlicesEqual/duplicates_same (0.00s) === RUN TestIamHash --- PASS: TestIamHash (0.00s) === RUN TestIamStringWithCharset --- PASS: TestIamStringWithCharset (0.00s) === RUN TestIamMapToStatementAction === RUN TestIamMapToStatementAction/* === RUN TestIamMapToStatementAction/Get* === RUN TestIamMapToStatementAction/Put* === RUN TestIamMapToStatementAction/List* === RUN TestIamMapToStatementAction/Tagging* === RUN TestIamMapToStatementAction/DeleteBucket* === RUN TestIamMapToStatementAction/PutBucketAcl === RUN TestIamMapToStatementAction/GetBucketAcl === RUN TestIamMapToStatementAction/InvalidAction --- PASS: TestIamMapToStatementAction (0.00s) --- PASS: TestIamMapToStatementAction/* (0.00s) --- PASS: TestIamMapToStatementAction/Get* (0.00s) --- PASS: TestIamMapToStatementAction/Put* (0.00s) --- PASS: TestIamMapToStatementAction/List* (0.00s) --- PASS: TestIamMapToStatementAction/Tagging* (0.00s) --- PASS: TestIamMapToStatementAction/DeleteBucket* (0.00s) --- PASS: TestIamMapToStatementAction/PutBucketAcl (0.00s) --- PASS: TestIamMapToStatementAction/GetBucketAcl (0.00s) --- PASS: TestIamMapToStatementAction/InvalidAction (0.00s) === RUN TestIamMapToIdentitiesAction === RUN TestIamMapToIdentitiesAction/Admin === RUN TestIamMapToIdentitiesAction/Read === RUN TestIamMapToIdentitiesAction/Write === RUN TestIamMapToIdentitiesAction/List === RUN TestIamMapToIdentitiesAction/Tagging === RUN TestIamMapToIdentitiesAction/Unknown --- PASS: TestIamMapToIdentitiesAction (0.00s) --- PASS: TestIamMapToIdentitiesAction/Admin (0.00s) --- PASS: TestIamMapToIdentitiesAction/Read (0.00s) --- PASS: TestIamMapToIdentitiesAction/Write (0.00s) --- PASS: TestIamMapToIdentitiesAction/List (0.00s) --- PASS: TestIamMapToIdentitiesAction/Tagging (0.00s) --- PASS: TestIamMapToIdentitiesAction/Unknown (0.00s) === RUN TestEmbeddedIamGetUserNotFound E0824 22:18:53.045656 s3api_embedded_iam.go:208 IAM Response the user with name NonExistentUser cannot be found. --- PASS: TestEmbeddedIamGetUserNotFound (0.00s) === RUN TestEmbeddedIamDeleteUserNotFound E0824 22:18:53.046858 s3api_embedded_iam.go:208 IAM Response the user with name NonExistentUser cannot be found. --- PASS: TestEmbeddedIamDeleteUserNotFound (0.00s) === RUN TestEmbeddedIamUpdateUserNotFound E0824 22:18:53.047802 s3api_embedded_iam.go:208 IAM Response the user with name NonExistentUser cannot be found. --- PASS: TestEmbeddedIamUpdateUserNotFound (0.00s) === RUN TestEmbeddedIamCreateAccessKeyForExistingUser --- PASS: TestEmbeddedIamCreateAccessKeyForExistingUser (0.00s) === RUN TestEmbeddedIamPutGetUserPolicyRoundTrip --- PASS: TestEmbeddedIamPutGetUserPolicyRoundTrip (0.00s) === RUN TestEmbeddedIamGetUserPolicyFallback --- PASS: TestEmbeddedIamGetUserPolicyFallback (0.00s) === RUN TestEmbeddedIamGetUserPolicyUserNotFound E0824 22:18:53.051051 s3api_embedded_iam.go:208 IAM Response the user with name NonExistentUser cannot be found. --- PASS: TestEmbeddedIamGetUserPolicyUserNotFound (0.00s) === RUN TestEmbeddedIamCreatePolicyMalformed E0824 22:18:53.052184 s3api_embedded_iam.go:2657 CreatePolicy: invalid character 'i' looking for beginning of value E0824 22:18:53.052318 s3api_embedded_iam.go:208 IAM Response invalid character 'i' looking for beginning of value --- PASS: TestEmbeddedIamCreatePolicyMalformed (0.00s) === RUN TestEmbeddedIamListAccessKeysForUser --- PASS: TestEmbeddedIamListAccessKeysForUser (0.00s) === RUN TestEmbeddedIamNotImplementedAction E0824 22:18:53.054593 s3api_embedded_iam.go:208 IAM Response A header you provided implies functionality that is not implemented --- PASS: TestEmbeddedIamNotImplementedAction (0.00s) === RUN TestGetPolicyDocument --- PASS: TestGetPolicyDocument (0.00s) === RUN TestEmbeddedIamGetActionsFromPolicy --- PASS: TestEmbeddedIamGetActionsFromPolicy (0.00s) === RUN TestEmbeddedIamGetActionsPrefixScopedResource --- PASS: TestEmbeddedIamGetActionsPrefixScopedResource (0.00s) === RUN TestEmbeddedIamPutUserPolicyAllResourceWildcard --- PASS: TestEmbeddedIamPutUserPolicyAllResourceWildcard (0.00s) === RUN TestEmbeddedIamSetUserStatus === RUN TestEmbeddedIamSetUserStatus/DisableUser === RUN TestEmbeddedIamSetUserStatus/EnableUser --- PASS: TestEmbeddedIamSetUserStatus (0.00s) --- PASS: TestEmbeddedIamSetUserStatus/DisableUser (0.00s) --- PASS: TestEmbeddedIamSetUserStatus/EnableUser (0.00s) === RUN TestEmbeddedIamSetUserStatusErrors === RUN TestEmbeddedIamSetUserStatusErrors/UserNotFound E0824 22:18:53.059046 s3api_embedded_iam.go:208 IAM Response the user with name NonExistentUser cannot be found. === RUN TestEmbeddedIamSetUserStatusErrors/InvalidStatus E0824 22:18:53.059689 s3api_embedded_iam.go:208 IAM Response Status must be 'Active' or 'Inactive' === RUN TestEmbeddedIamSetUserStatusErrors/MissingUserName E0824 22:18:53.060226 s3api_embedded_iam.go:208 IAM Response UserName is required === RUN TestEmbeddedIamSetUserStatusErrors/MissingStatus E0824 22:18:53.060804 s3api_embedded_iam.go:208 IAM Response Status parameter is required --- PASS: TestEmbeddedIamSetUserStatusErrors (0.00s) --- PASS: TestEmbeddedIamSetUserStatusErrors/UserNotFound (0.00s) --- PASS: TestEmbeddedIamSetUserStatusErrors/InvalidStatus (0.00s) --- PASS: TestEmbeddedIamSetUserStatusErrors/MissingUserName (0.00s) --- PASS: TestEmbeddedIamSetUserStatusErrors/MissingStatus (0.00s) === RUN TestEmbeddedIamUpdateAccessKey === RUN TestEmbeddedIamUpdateAccessKey/DeactivateAccessKey === RUN TestEmbeddedIamUpdateAccessKey/ActivateAccessKey --- PASS: TestEmbeddedIamUpdateAccessKey (0.00s) --- PASS: TestEmbeddedIamUpdateAccessKey/DeactivateAccessKey (0.00s) --- PASS: TestEmbeddedIamUpdateAccessKey/ActivateAccessKey (0.00s) === RUN TestEmbeddedIamUpdateAccessKeyErrors === RUN TestEmbeddedIamUpdateAccessKeyErrors/AccessKeyNotFound E0824 22:18:53.063148 s3api_embedded_iam.go:208 IAM Response the access key with id NONEXISTENT123 for user TestUser cannot be found === RUN TestEmbeddedIamUpdateAccessKeyErrors/InvalidStatus E0824 22:18:53.063843 s3api_embedded_iam.go:208 IAM Response Status must be 'Active' or 'Inactive' === RUN TestEmbeddedIamUpdateAccessKeyErrors/MissingUserName E0824 22:18:53.064495 s3api_embedded_iam.go:208 IAM Response UserName is required === RUN TestEmbeddedIamUpdateAccessKeyErrors/MissingAccessKeyId E0824 22:18:53.065105 s3api_embedded_iam.go:208 IAM Response AccessKeyId is required === RUN TestEmbeddedIamUpdateAccessKeyErrors/UserNotFound E0824 22:18:53.065748 s3api_embedded_iam.go:208 IAM Response the user with name NonExistentUser cannot be found. === RUN TestEmbeddedIamUpdateAccessKeyErrors/MissingStatus E0824 22:18:53.066327 s3api_embedded_iam.go:208 IAM Response Status parameter is required --- PASS: TestEmbeddedIamUpdateAccessKeyErrors (0.00s) --- PASS: TestEmbeddedIamUpdateAccessKeyErrors/AccessKeyNotFound (0.00s) --- PASS: TestEmbeddedIamUpdateAccessKeyErrors/InvalidStatus (0.00s) --- PASS: TestEmbeddedIamUpdateAccessKeyErrors/MissingUserName (0.00s) --- PASS: TestEmbeddedIamUpdateAccessKeyErrors/MissingAccessKeyId (0.00s) --- PASS: TestEmbeddedIamUpdateAccessKeyErrors/UserNotFound (0.00s) --- PASS: TestEmbeddedIamUpdateAccessKeyErrors/MissingStatus (0.00s) === RUN TestEmbeddedIamListAccessKeysShowsStatus --- PASS: TestEmbeddedIamListAccessKeysShowsStatus (0.00s) === RUN TestDisabledUserLookupFails --- PASS: TestDisabledUserLookupFails (0.00s) === RUN TestInactiveAccessKeyLookupFails --- PASS: TestInactiveAccessKeyLookupFails (0.00s) === RUN TestAuthIamAuthenticatesBeforeParseForm --- PASS: TestAuthIamAuthenticatesBeforeParseForm (0.00s) === RUN TestOldCodeOrderWouldFail s3api_embedded_iam_test.go:2776: This demonstrates the bug: ParseForm before auth causes SignatureDoesNotMatch --- PASS: TestOldCodeOrderWouldFail (0.00s) === RUN TestEmbeddedIamExecuteAction --- PASS: TestEmbeddedIamExecuteAction (0.00s) === RUN TestEmbeddedIamReadOnly E0824 22:18:53.073563 s3api_embedded_iam.go:208 IAM Response IAM write operations are disabled on this server --- PASS: TestEmbeddedIamReadOnly (0.00s) === RUN TestEmbeddedIamTagUser --- PASS: TestEmbeddedIamTagUser (0.00s) === RUN TestEmbeddedIamListUserTags --- PASS: TestEmbeddedIamListUserTags (0.00s) === RUN TestEmbeddedIamUntagUser --- PASS: TestEmbeddedIamUntagUser (0.00s) === RUN TestEmbeddedIamTagUserInvalidKeyLength E0824 22:18:53.096715 s3api_embedded_iam.go:208 IAM Response tag key "kkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkk" exceeds 128 characters --- PASS: TestEmbeddedIamTagUserInvalidKeyLength (0.00s) === RUN TestEmbeddedIamTagUserReplacesDuplicate --- PASS: TestEmbeddedIamTagUserReplacesDuplicate (0.00s) === RUN TestEmbeddedIamTagUserLimitExceeded E0824 22:18:53.105582 s3api_embedded_iam.go:208 IAM Response cannot exceed 50 tags per user --- PASS: TestEmbeddedIamTagUserLimitExceeded (0.00s) === RUN TestEmbeddedIamUntagUserNoOpForMissingKey --- PASS: TestEmbeddedIamUntagUserNoOpForMissingKey (0.00s) === RUN TestEmbeddedIamTagUserRejectsDuplicateKeyInRequest E0824 22:18:53.111136 s3api_embedded_iam.go:208 IAM Response duplicate tag key "env" --- PASS: TestEmbeddedIamTagUserRejectsDuplicateKeyInRequest (0.00s) === RUN TestEmbeddedIamUntagUserRejectsInvalidKeys === RUN TestEmbeddedIamUntagUserRejectsInvalidKeys/no_TagKeys_entries E0824 22:18:53.114682 s3api_embedded_iam.go:208 IAM Response at least one TagKeys entry is required === RUN TestEmbeddedIamUntagUserRejectsInvalidKeys/empty_TagKeys_entry E0824 22:18:53.117369 s3api_embedded_iam.go:208 IAM Response tag key cannot be empty === RUN TestEmbeddedIamUntagUserRejectsInvalidKeys/TagKeys_entry_over_MaxUserTagKeyLength E0824 22:18:53.119796 s3api_embedded_iam.go:208 IAM Response tag key "kkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkk" exceeds 128 characters --- PASS: TestEmbeddedIamUntagUserRejectsInvalidKeys (0.01s) --- PASS: TestEmbeddedIamUntagUserRejectsInvalidKeys/no_TagKeys_entries (0.00s) --- PASS: TestEmbeddedIamUntagUserRejectsInvalidKeys/empty_TagKeys_entry (0.00s) --- PASS: TestEmbeddedIamUntagUserRejectsInvalidKeys/TagKeys_entry_over_MaxUserTagKeyLength (0.00s) === RUN TestEmbeddedIamTagUserNotFound E0824 22:18:53.123642 s3api_embedded_iam.go:208 IAM Response the user with name ghost cannot be found. --- PASS: TestEmbeddedIamTagUserNotFound (0.00s) === RUN TestCreateDestinationChunkPreservesEncryption === RUN TestCreateDestinationChunkPreservesEncryption/Encrypted_and_compressed_chunk === RUN TestCreateDestinationChunkPreservesEncryption/Only_encrypted_chunk === RUN TestCreateDestinationChunkPreservesEncryption/Only_compressed_chunk === RUN TestCreateDestinationChunkPreservesEncryption/Unencrypted_and_uncompressed_chunk --- PASS: TestCreateDestinationChunkPreservesEncryption (0.00s) --- PASS: TestCreateDestinationChunkPreservesEncryption/Encrypted_and_compressed_chunk (0.00s) --- PASS: TestCreateDestinationChunkPreservesEncryption/Only_encrypted_chunk (0.00s) --- PASS: TestCreateDestinationChunkPreservesEncryption/Only_compressed_chunk (0.00s) --- PASS: TestCreateDestinationChunkPreservesEncryption/Unencrypted_and_uncompressed_chunk (0.00s) === RUN TestEncryptedVolumeCopyScenario === RUN TestEncryptedVolumeCopyScenario/Scenario:_Copy_file_on_encrypted_volume_with_multiple_chunks s3api_encrypted_volume_copy_test.go:173: ✓ All chunk metadata properly preserved for encrypted volume copy scenario --- PASS: TestEncryptedVolumeCopyScenario (0.01s) --- PASS: TestEncryptedVolumeCopyScenario/Scenario:_Copy_file_on_encrypted_volume_with_multiple_chunks (0.01s) === RUN TestCheckGovernanceBypassPermissionResourceGeneration === RUN TestCheckGovernanceBypassPermissionResourceGeneration/simple_object === RUN TestCheckGovernanceBypassPermissionResourceGeneration/object_with_leading_slash === RUN TestCheckGovernanceBypassPermissionResourceGeneration/nested_object === RUN TestCheckGovernanceBypassPermissionResourceGeneration/empty_object === RUN TestCheckGovernanceBypassPermissionResourceGeneration/root_object --- PASS: TestCheckGovernanceBypassPermissionResourceGeneration (0.00s) --- PASS: TestCheckGovernanceBypassPermissionResourceGeneration/simple_object (0.00s) --- PASS: TestCheckGovernanceBypassPermissionResourceGeneration/object_with_leading_slash (0.00s) --- PASS: TestCheckGovernanceBypassPermissionResourceGeneration/nested_object (0.00s) --- PASS: TestCheckGovernanceBypassPermissionResourceGeneration/empty_object (0.00s) --- PASS: TestCheckGovernanceBypassPermissionResourceGeneration/root_object (0.00s) === RUN TestCheckGovernanceBypassPermissionActionGeneration === RUN TestCheckGovernanceBypassPermissionActionGeneration/bypass_action_generation === RUN TestCheckGovernanceBypassPermissionActionGeneration/leading_slash_handling --- PASS: TestCheckGovernanceBypassPermissionActionGeneration (0.00s) --- PASS: TestCheckGovernanceBypassPermissionActionGeneration/bypass_action_generation (0.00s) --- PASS: TestCheckGovernanceBypassPermissionActionGeneration/leading_slash_handling (0.00s) === RUN TestCheckGovernanceBypassPermissionErrorHandling === RUN TestCheckGovernanceBypassPermissionErrorHandling/empty_bucket s3api_governance_permissions_test.go:169: Generated resource path for Empty bucket should be handled gracefully: /test-object.txt === RUN TestCheckGovernanceBypassPermissionErrorHandling/special_characters s3api_governance_permissions_test.go:169: Generated resource path for Objects with special characters should be handled: test-bucket/test object with spaces.txt === RUN TestCheckGovernanceBypassPermissionErrorHandling/unicode_characters s3api_governance_permissions_test.go:169: Generated resource path for Objects with unicode characters should be handled: test-bucket/测试文件.txt --- PASS: TestCheckGovernanceBypassPermissionErrorHandling (0.00s) --- PASS: TestCheckGovernanceBypassPermissionErrorHandling/empty_bucket (0.00s) --- PASS: TestCheckGovernanceBypassPermissionErrorHandling/special_characters (0.00s) --- PASS: TestCheckGovernanceBypassPermissionErrorHandling/unicode_characters (0.00s) === RUN TestCheckGovernanceBypassPermissionIntegrationBehavior s3api_governance_permissions_test.go:177: Documentation test - describes expected behavior with full IAM integration --- SKIP: TestCheckGovernanceBypassPermissionIntegrationBehavior (0.00s) === RUN TestGovernanceBypassWithIAMPermission s3api_governance_permissions_test.go:206: Integration test requires full IAM setup - demonstrates expected behavior --- SKIP: TestGovernanceBypassWithIAMPermission (0.00s) === RUN TestGovernancePermissionIntegration s3api_governance_permissions_test.go:224: Integration test requires full IAM setup - demonstrates expected behavior --- SKIP: TestGovernancePermissionIntegration (0.00s) === RUN TestGovernanceBypassHeader === RUN TestGovernanceBypassHeader/bypass_header_true === RUN TestGovernanceBypassHeader/bypass_header_false === RUN TestGovernanceBypassHeader/bypass_header_empty === RUN TestGovernanceBypassHeader/bypass_header_invalid --- PASS: TestGovernanceBypassHeader (0.00s) --- PASS: TestGovernanceBypassHeader/bypass_header_true (0.00s) --- PASS: TestGovernanceBypassHeader/bypass_header_false (0.00s) --- PASS: TestGovernanceBypassHeader/bypass_header_empty (0.00s) --- PASS: TestGovernanceBypassHeader/bypass_header_invalid (0.00s) === RUN TestGovernanceRetentionModeChecking === RUN TestGovernanceRetentionModeChecking/compliance_mode_cannot_bypass === RUN TestGovernanceRetentionModeChecking/governance_mode_without_bypass === RUN TestGovernanceRetentionModeChecking/governance_mode_with_bypass_no_permission === RUN TestGovernanceRetentionModeChecking/governance_mode_with_bypass_and_permission --- PASS: TestGovernanceRetentionModeChecking (0.00s) --- PASS: TestGovernanceRetentionModeChecking/compliance_mode_cannot_bypass (0.00s) --- PASS: TestGovernanceRetentionModeChecking/governance_mode_without_bypass (0.00s) --- PASS: TestGovernanceRetentionModeChecking/governance_mode_with_bypass_no_permission (0.00s) --- PASS: TestGovernanceRetentionModeChecking/governance_mode_with_bypass_and_permission (0.00s) === RUN TestGovernancePermissionActionGeneration === RUN TestGovernancePermissionActionGeneration/bucket_and_object_action === RUN TestGovernancePermissionActionGeneration/bucket_only_action === RUN TestGovernancePermissionActionGeneration/nested_object_action --- PASS: TestGovernancePermissionActionGeneration (0.00s) --- PASS: TestGovernancePermissionActionGeneration/bucket_and_object_action (0.00s) --- PASS: TestGovernancePermissionActionGeneration/bucket_only_action (0.00s) --- PASS: TestGovernancePermissionActionGeneration/nested_object_action (0.00s) === RUN TestGovernancePermissionEndToEnd s3api_governance_permissions_test.go:406: End-to-end testing requires full S3 API server setup - demonstrates expected behavior --- SKIP: TestGovernancePermissionEndToEnd (0.00s) === RUN TestGovernancePermissionHTTPFlow === RUN TestGovernancePermissionHTTPFlow/bypass_header_true === RUN TestGovernancePermissionHTTPFlow/bypass_header_false === RUN TestGovernancePermissionHTTPFlow/bypass_header_missing --- PASS: TestGovernancePermissionHTTPFlow (0.00s) --- PASS: TestGovernancePermissionHTTPFlow/bypass_header_true (0.00s) --- PASS: TestGovernancePermissionHTTPFlow/bypass_header_false (0.00s) --- PASS: TestGovernancePermissionHTTPFlow/bypass_header_missing (0.00s) === RUN TestGovernancePermissionMethodCalls === RUN TestGovernancePermissionMethodCalls/delete_object_handler_pattern s3api_governance_permissions_test.go:482: Extracted bucket: , object: === RUN TestGovernancePermissionMethodCalls/put_object_handler_pattern s3api_governance_permissions_test.go:503: Extracted bucket: , object: --- PASS: TestGovernancePermissionMethodCalls (0.00s) --- PASS: TestGovernancePermissionMethodCalls/delete_object_handler_pattern (0.00s) --- PASS: TestGovernancePermissionMethodCalls/put_object_handler_pattern (0.00s) === RUN TestGovernanceBypassNotPermittedError === RUN TestGovernanceBypassNotPermittedError/governance_bypass_without_permission === RUN TestGovernanceBypassNotPermittedError/governance_bypass_with_permission === RUN TestGovernanceBypassNotPermittedError/governance_no_bypass --- PASS: TestGovernanceBypassNotPermittedError (0.00s) --- PASS: TestGovernanceBypassNotPermittedError/governance_bypass_without_permission (0.00s) --- PASS: TestGovernanceBypassNotPermittedError/governance_bypass_with_permission (0.00s) --- PASS: TestGovernanceBypassNotPermittedError/governance_no_bypass (0.00s) === RUN TestListOpenIDConnectProviders --- PASS: TestListOpenIDConnectProviders (0.00s) === RUN TestGetOpenIDConnectProvider --- PASS: TestGetOpenIDConnectProvider (0.00s) === RUN TestGetOpenIDConnectProviderMissing --- PASS: TestGetOpenIDConnectProviderMissing (0.00s) === RUN TestGetOpenIDConnectProviderRequiresArn --- PASS: TestGetOpenIDConnectProviderRequiresArn (0.00s) === RUN TestReadOnlyAllowsOIDCList --- PASS: TestReadOnlyAllowsOIDCList (0.00s) === RUN TestReadOnlyDeniesOIDCMutations === RUN TestReadOnlyDeniesOIDCMutations/CreateOpenIDConnectProvider === RUN TestReadOnlyDeniesOIDCMutations/DeleteOpenIDConnectProvider === RUN TestReadOnlyDeniesOIDCMutations/AddClientIDToOpenIDConnectProvider === RUN TestReadOnlyDeniesOIDCMutations/RemoveClientIDFromOpenIDConnectProvider === RUN TestReadOnlyDeniesOIDCMutations/UpdateOpenIDConnectProviderThumbprint === RUN TestReadOnlyDeniesOIDCMutations/TagOpenIDConnectProvider === RUN TestReadOnlyDeniesOIDCMutations/UntagOpenIDConnectProvider --- PASS: TestReadOnlyDeniesOIDCMutations (0.00s) --- PASS: TestReadOnlyDeniesOIDCMutations/CreateOpenIDConnectProvider (0.00s) --- PASS: TestReadOnlyDeniesOIDCMutations/DeleteOpenIDConnectProvider (0.00s) --- PASS: TestReadOnlyDeniesOIDCMutations/AddClientIDToOpenIDConnectProvider (0.00s) --- PASS: TestReadOnlyDeniesOIDCMutations/RemoveClientIDFromOpenIDConnectProvider (0.00s) --- PASS: TestReadOnlyDeniesOIDCMutations/UpdateOpenIDConnectProviderThumbprint (0.00s) --- PASS: TestReadOnlyDeniesOIDCMutations/TagOpenIDConnectProvider (0.00s) --- PASS: TestReadOnlyDeniesOIDCMutations/UntagOpenIDConnectProvider (0.00s) === RUN TestCreateOpenIDConnectProvider I0824 22:18:53.170592 oidc_provider_audit.go:66 oidc-audit: {"type":"Create","arn":"arn:aws:iam::111122223333:oidc-provider/auth.example.com","url":"https://auth.example.com","occurredAt":"2026-08-24T22:18:53.169765856Z"} --- PASS: TestCreateOpenIDConnectProvider (0.00s) === RUN TestCreateOpenIDConnectProviderRejectsBadInput === RUN TestCreateOpenIDConnectProviderRejectsBadInput/missing_Url === RUN TestCreateOpenIDConnectProviderRejectsBadInput/missing_ClientIDList === RUN TestCreateOpenIDConnectProviderRejectsBadInput/invalid_thumbprint --- PASS: TestCreateOpenIDConnectProviderRejectsBadInput (0.00s) --- PASS: TestCreateOpenIDConnectProviderRejectsBadInput/missing_Url (0.00s) --- PASS: TestCreateOpenIDConnectProviderRejectsBadInput/missing_ClientIDList (0.00s) --- PASS: TestCreateOpenIDConnectProviderRejectsBadInput/invalid_thumbprint (0.00s) === RUN TestCreateOpenIDConnectProviderConflict --- PASS: TestCreateOpenIDConnectProviderConflict (0.00s) === RUN TestDeleteOpenIDConnectProvider I0824 22:18:53.179343 oidc_provider_audit.go:66 oidc-audit: {"type":"Delete","arn":"arn:aws:iam::111122223333:oidc-provider/accounts.google.com","occurredAt":"2026-08-24T22:18:53.179142153Z"} I0824 22:18:53.180259 oidc_provider_audit.go:66 oidc-audit: {"type":"Delete","arn":"arn:aws:iam::111122223333:oidc-provider/accounts.google.com","occurredAt":"2026-08-24T22:18:53.180164353Z"} --- PASS: TestDeleteOpenIDConnectProvider (0.00s) === RUN TestAddRemoveClientID I0824 22:18:53.182274 oidc_provider_audit.go:66 oidc-audit: {"type":"AddClientID","arn":"arn:aws:iam::111122223333:oidc-provider/accounts.google.com","url":"https://accounts.google.com","detail":{"clientId":"extra-client"},"occurredAt":"2026-08-24T22:18:53.182028992Z"} I0824 22:18:53.182781 oidc_provider_audit.go:66 oidc-audit: {"type":"RemoveClientID","arn":"arn:aws:iam::111122223333:oidc-provider/accounts.google.com","url":"https://accounts.google.com","detail":{"clientId":"extra-client"},"occurredAt":"2026-08-24T22:18:53.182716992Z"} --- PASS: TestAddRemoveClientID (0.00s) === RUN TestUpdateThumbprintAndTags I0824 22:18:53.185348 oidc_provider_audit.go:66 oidc-audit: {"type":"UpdateThumbprints","arn":"arn:aws:iam::111122223333:oidc-provider/accounts.google.com","url":"https://accounts.google.com","detail":{"count":"2"},"occurredAt":"2026-08-24T22:18:53.185055151Z"} I0824 22:18:53.186160 oidc_provider_audit.go:66 oidc-audit: {"type":"Tag","arn":"arn:aws:iam::111122223333:oidc-provider/accounts.google.com","url":"https://accounts.google.com","detail":{"count":"1"},"occurredAt":"2026-08-24T22:18:53.18599485Z"} I0824 22:18:53.186817 oidc_provider_audit.go:66 oidc-audit: {"type":"Untag","arn":"arn:aws:iam::111122223333:oidc-provider/accounts.google.com","url":"https://accounts.google.com","detail":{"count":"1"},"occurredAt":"2026-08-24T22:18:53.18672047Z"} --- PASS: TestUpdateThumbprintAndTags (0.00s) === RUN TestTablePrefixSessionPolicyScopesToTheTable --- PASS: TestTablePrefixSessionPolicyScopesToTheTable (0.00s) === RUN TestTablePrefixSessionPolicyRefusesAWholeBucket --- PASS: TestTablePrefixSessionPolicyRefusesAWholeBucket (0.00s) === RUN TestTablePrefixSessionPolicyLeavesBucketLocationUnconditioned --- PASS: TestTablePrefixSessionPolicyLeavesBucketLocationUnconditioned (0.00s) === RUN TestIcebergSessionNameIsBounded --- PASS: TestIcebergSessionNameIsBounded (0.00s) === RUN TestVendTableCredentialsDisabledWithoutRole --- PASS: TestVendTableCredentialsDisabledWithoutRole (0.00s) === RUN TestVendTableCredentialsNeedsSTS --- PASS: TestVendTableCredentialsNeedsSTS (0.00s) === RUN TestTablePrefixSessionPolicyRestrictsListing --- PASS: TestTablePrefixSessionPolicyRestrictsListing (0.00s) === RUN TestTablePrefixSessionPolicyRefusesWildcards --- PASS: TestTablePrefixSessionPolicyRefusesWildcards (0.00s) === RUN TestImplicitDirectoryBehaviorLogic === RUN TestImplicitDirectoryBehaviorLogic/Implicit_directory:_0-byte_file_with_children,_no_trailing_slash s3api_implicit_directory_test.go:143: ✓ Implicit directory: 0-byte file with children, no trailing slash: correctly returns 404 === RUN TestImplicitDirectoryBehaviorLogic/Implicit_directory:_actual_directory_with_children,_no_trailing_slash s3api_implicit_directory_test.go:143: ✓ Implicit directory: actual directory with children, no trailing slash: correctly returns 404 === RUN TestImplicitDirectoryBehaviorLogic/Explicit_directory_request:_trailing_slash s3api_implicit_directory_test.go:143: ✓ Explicit directory request: trailing slash: correctly returns 200 === RUN TestImplicitDirectoryBehaviorLogic/Empty_file:_0-byte_file_without_children s3api_implicit_directory_test.go:143: ✓ Empty file: 0-byte file without children: correctly returns 200 === RUN TestImplicitDirectoryBehaviorLogic/Empty_directory:_0-byte_directory_without_children s3api_implicit_directory_test.go:143: ✓ Empty directory: 0-byte directory without children: correctly returns 404 === RUN TestImplicitDirectoryBehaviorLogic/Regular_file:_non-zero_size s3api_implicit_directory_test.go:143: ✓ Regular file: non-zero size: correctly returns 200 === RUN TestImplicitDirectoryBehaviorLogic/Versioned_bucket:_implicit_directory_should_return_200 s3api_implicit_directory_test.go:143: ✓ Versioned bucket: implicit directory should return 200: correctly returns 200 === RUN TestImplicitDirectoryBehaviorLogic/PyArrow_directory_marker:_0-byte_with_children s3api_implicit_directory_test.go:143: ✓ PyArrow directory marker: 0-byte with children: correctly returns 404 --- PASS: TestImplicitDirectoryBehaviorLogic (0.00s) --- PASS: TestImplicitDirectoryBehaviorLogic/Implicit_directory:_0-byte_file_with_children,_no_trailing_slash (0.00s) --- PASS: TestImplicitDirectoryBehaviorLogic/Implicit_directory:_actual_directory_with_children,_no_trailing_slash (0.00s) --- PASS: TestImplicitDirectoryBehaviorLogic/Explicit_directory_request:_trailing_slash (0.00s) --- PASS: TestImplicitDirectoryBehaviorLogic/Empty_file:_0-byte_file_without_children (0.00s) --- PASS: TestImplicitDirectoryBehaviorLogic/Empty_directory:_0-byte_directory_without_children (0.00s) --- PASS: TestImplicitDirectoryBehaviorLogic/Regular_file:_non-zero_size (0.00s) --- PASS: TestImplicitDirectoryBehaviorLogic/Versioned_bucket:_implicit_directory_should_return_200 (0.00s) --- PASS: TestImplicitDirectoryBehaviorLogic/PyArrow_directory_marker:_0-byte_with_children (0.00s) === RUN TestHasChildrenLogic === RUN TestHasChildrenLogic/Directory_with_children s3api_implicit_directory_test.go:218: ✓ Directory with children: correctly returns true === RUN TestHasChildrenLogic/Empty_directory s3api_implicit_directory_test.go:218: ✓ Empty directory: correctly returns false === RUN TestHasChildrenLogic/Directory_with_leading_slash_in_prefix s3api_implicit_directory_test.go:218: ✓ Directory with leading slash in prefix: correctly returns true --- PASS: TestHasChildrenLogic (0.00s) --- PASS: TestHasChildrenLogic/Directory_with_children (0.00s) --- PASS: TestHasChildrenLogic/Empty_directory (0.00s) --- PASS: TestHasChildrenLogic/Directory_with_leading_slash_in_prefix (0.00s) === RUN TestImplicitDirectoryEdgeCases === RUN TestImplicitDirectoryEdgeCases/PyArrow_write_dataset_creates_0-byte_files s3api_implicit_directory_test.go:265: Scenario: PyArrow creates 'dataset' as 0-byte file, then writes 'dataset/file.parquet' s3api_implicit_directory_test.go:266: Expected: HEAD dataset → 404 (has children), s3fs uses LIST → correctly identifies as directory === RUN TestImplicitDirectoryEdgeCases/Filer_creates_actual_directories s3api_implicit_directory_test.go:265: Scenario: Filer creates 'dataset' as actual directory with IsDirectory=true s3api_implicit_directory_test.go:266: Expected: HEAD dataset → 404 (has children), s3fs uses LIST → correctly identifies as directory === RUN TestImplicitDirectoryEdgeCases/Empty_file_edge_case s3api_implicit_directory_test.go:265: Scenario: User creates 'empty.txt' as 0-byte file with no children s3api_implicit_directory_test.go:266: Expected: HEAD empty.txt → 200 (no children), s3fs correctly reports as file === RUN TestImplicitDirectoryEdgeCases/Explicit_directory_request s3api_implicit_directory_test.go:265: Scenario: User requests 'dataset/' with trailing slash s3api_implicit_directory_test.go:266: Expected: HEAD dataset/ → 200 (explicit directory request), normal directory behavior === RUN TestImplicitDirectoryEdgeCases/Versioned_bucket s3api_implicit_directory_test.go:265: Scenario: Bucket has versioning enabled s3api_implicit_directory_test.go:266: Expected: HEAD dataset → 200 (skip implicit dir check), versioned semantics apply === RUN TestImplicitDirectoryEdgeCases/AWS_S3_compatibility s3api_implicit_directory_test.go:265: Scenario: Only 'dataset/file.txt' exists, no marker at 'dataset' s3api_implicit_directory_test.go:266: Expected: HEAD dataset → 404 (object doesn't exist), matches AWS S3 behavior --- PASS: TestImplicitDirectoryEdgeCases (0.00s) --- PASS: TestImplicitDirectoryEdgeCases/PyArrow_write_dataset_creates_0-byte_files (0.00s) --- PASS: TestImplicitDirectoryEdgeCases/Filer_creates_actual_directories (0.00s) --- PASS: TestImplicitDirectoryEdgeCases/Empty_file_edge_case (0.00s) --- PASS: TestImplicitDirectoryEdgeCases/Explicit_directory_request (0.00s) --- PASS: TestImplicitDirectoryEdgeCases/Versioned_bucket (0.00s) --- PASS: TestImplicitDirectoryEdgeCases/AWS_S3_compatibility (0.00s) === RUN TestImplicitDirectoryIntegration s3api_implicit_directory_test.go:278: Integration test - run manually with: cd test/s3/parquet && make test-implicit-dir-with-server --- SKIP: TestImplicitDirectoryIntegration (0.00s) === RUN TestUserInlinePolicySourceIpCondition_Denies --- PASS: TestUserInlinePolicySourceIpCondition_Denies (0.00s) === RUN TestUserInlinePolicySourceIpCondition_Allows --- PASS: TestUserInlinePolicySourceIpCondition_Allows (0.00s) === RUN TestGroupInlinePolicy_PutAndEnforce --- PASS: TestGroupInlinePolicy_PutAndEnforce (0.01s) === RUN TestUserInlinePolicy_ConditionDiscriminatesAllowVsDeny --- PASS: TestUserInlinePolicy_ConditionDiscriminatesAllowVsDeny (0.01s) === RUN TestUserInlinePolicy_ReloadFromStore --- PASS: TestUserInlinePolicy_ReloadFromStore (0.00s) === RUN TestComputeEntryIdentity_BasicFields --- PASS: TestComputeEntryIdentity_BasicFields (0.00s) === RUN TestComputeEntryIdentity_NilSafeMissingChunks --- PASS: TestComputeEntryIdentity_NilSafeMissingChunks (0.00s) === RUN TestHashExtended_OrderStable --- PASS: TestHashExtended_OrderStable (0.00s) === RUN TestHashExtended_DelimiterCollisionResistant --- PASS: TestHashExtended_DelimiterCollisionResistant (0.00s) === RUN TestHashExtended_NilEqualsEmpty --- PASS: TestHashExtended_NilEqualsEmpty (0.00s) === RUN TestIdentityMatches_NilWantTreatedAsMatch --- PASS: TestIdentityMatches_NilWantTreatedAsMatch (0.00s) === RUN TestIdentityMatches_NilLiveDoesNotMatch --- PASS: TestIdentityMatches_NilLiveDoesNotMatch (0.00s) === RUN TestIdentityMatches_AllFieldsCompared === RUN TestIdentityMatches_AllFieldsCompared/identical === RUN TestIdentityMatches_AllFieldsCompared/mtime-drift === RUN TestIdentityMatches_AllFieldsCompared/size-drift === RUN TestIdentityMatches_AllFieldsCompared/fid-drift === RUN TestIdentityMatches_AllFieldsCompared/extended-drift --- PASS: TestIdentityMatches_AllFieldsCompared (0.00s) --- PASS: TestIdentityMatches_AllFieldsCompared/identical (0.00s) --- PASS: TestIdentityMatches_AllFieldsCompared/mtime-drift (0.00s) --- PASS: TestIdentityMatches_AllFieldsCompared/size-drift (0.00s) --- PASS: TestIdentityMatches_AllFieldsCompared/fid-drift (0.00s) --- PASS: TestIdentityMatches_AllFieldsCompared/extended-drift (0.00s) === RUN TestLifecycleDelete_RejectsEmptyRequest --- PASS: TestLifecycleDelete_RejectsEmptyRequest (0.00s) === RUN TestLifecycleAbortMPU_RejectsTraversalUploadIDs === RUN TestLifecycleAbortMPU_RejectsTraversalUploadIDs/#00 === RUN TestLifecycleAbortMPU_RejectsTraversalUploadIDs/.uploads === RUN TestLifecycleAbortMPU_RejectsTraversalUploadIDs/.uploads/ === RUN TestLifecycleAbortMPU_RejectsTraversalUploadIDs/.uploads/. === RUN TestLifecycleAbortMPU_RejectsTraversalUploadIDs/.uploads/.. === RUN TestLifecycleAbortMPU_RejectsTraversalUploadIDs/.uploads/u1/extra --- PASS: TestLifecycleAbortMPU_RejectsTraversalUploadIDs (0.00s) --- PASS: TestLifecycleAbortMPU_RejectsTraversalUploadIDs/#00 (0.00s) --- PASS: TestLifecycleAbortMPU_RejectsTraversalUploadIDs/.uploads (0.00s) --- PASS: TestLifecycleAbortMPU_RejectsTraversalUploadIDs/.uploads/ (0.00s) --- PASS: TestLifecycleAbortMPU_RejectsTraversalUploadIDs/.uploads/. (0.00s) --- PASS: TestLifecycleAbortMPU_RejectsTraversalUploadIDs/.uploads/.. (0.00s) --- PASS: TestLifecycleAbortMPU_RejectsTraversalUploadIDs/.uploads/u1/extra (0.00s) === RUN TestLifecycleDispatch_AbortMPUAfterFetchIsBlocked --- PASS: TestLifecycleDispatch_AbortMPUAfterFetchIsBlocked (0.00s) === RUN TestLifecycleDispatch_UnknownActionKindIsBlocked --- PASS: TestLifecycleDispatch_UnknownActionKindIsBlocked (0.00s) === RUN TestLifecycleDispatch_NoncurrentRequiresVersionID === RUN TestLifecycleDispatch_NoncurrentRequiresVersionID/NONCURRENT_DAYS === RUN TestLifecycleDispatch_NoncurrentRequiresVersionID/NEWER_NONCURRENT === RUN TestLifecycleDispatch_NoncurrentRequiresVersionID/EXPIRED_DELETE_MARKER --- PASS: TestLifecycleDispatch_NoncurrentRequiresVersionID (0.00s) --- PASS: TestLifecycleDispatch_NoncurrentRequiresVersionID/NONCURRENT_DAYS (0.00s) --- PASS: TestLifecycleDispatch_NoncurrentRequiresVersionID/NEWER_NONCURRENT (0.00s) --- PASS: TestLifecycleDispatch_NoncurrentRequiresVersionID/EXPIRED_DELETE_MARKER (0.00s) === RUN TestEntryUsesMetadataOnlyDelete === RUN TestEntryUsesMetadataOnlyDelete/nil_entry === RUN TestEntryUsesMetadataOnlyDelete/nil_attributes === RUN TestEntryUsesMetadataOnlyDelete/TtlSec=0_(no_per-write_stamp) === RUN TestEntryUsesMetadataOnlyDelete/TtlSec>0_(PR_9377_stamped_a_fast-path_TTL) === RUN TestEntryUsesMetadataOnlyDelete/TtlSec<0_should_not_happen_but_must_not_flip_the_path_on --- PASS: TestEntryUsesMetadataOnlyDelete (0.00s) --- PASS: TestEntryUsesMetadataOnlyDelete/nil_entry (0.00s) --- PASS: TestEntryUsesMetadataOnlyDelete/nil_attributes (0.00s) --- PASS: TestEntryUsesMetadataOnlyDelete/TtlSec=0_(no_per-write_stamp) (0.00s) --- PASS: TestEntryUsesMetadataOnlyDelete/TtlSec>0_(PR_9377_stamped_a_fast-path_TTL) (0.00s) --- PASS: TestEntryUsesMetadataOnlyDelete/TtlSec<0_should_not_happen_but_must_not_flip_the_path_on (0.00s) === RUN TestRecordMetadataOnlyIf_OnlyFiresWhenOn --- PASS: TestRecordMetadataOnlyIf_OnlyFiresWhenOn (0.00s) === RUN TestRecordMetadataOnlyIf_NilRequestSafe --- PASS: TestRecordMetadataOnlyIf_NilRequestSafe (0.00s) === RUN TestRecordMetadataOnlyIf_EmptyRuleHashCollapsesToEmptyLabel --- PASS: TestRecordMetadataOnlyIf_EmptyRuleHashCollapsesToEmptyLabel (0.00s) === RUN TestPrefixNormalizationInList === RUN TestPrefixNormalizationInList/simple_prefix === RUN TestPrefixNormalizationInList/leading_slash === RUN TestPrefixNormalizationInList/duplicate_slashes === RUN TestPrefixNormalizationInList/backslashes --- PASS: TestPrefixNormalizationInList (0.00s) --- PASS: TestPrefixNormalizationInList/simple_prefix (0.00s) --- PASS: TestPrefixNormalizationInList/leading_slash (0.00s) --- PASS: TestPrefixNormalizationInList/duplicate_slashes (0.00s) --- PASS: TestPrefixNormalizationInList/backslashes (0.00s) === RUN TestListPrefixConsistency --- PASS: TestListPrefixConsistency (0.00s) === RUN TestProbeReadable === RUN TestProbeReadable/readable_local_copy === RUN TestProbeReadable/final_byte_with_trailing_EOF_counts_as_readable === RUN TestProbeReadable/zero-byte_EOF_is_unreadable === RUN TestProbeReadable/zero-byte_read_without_error_is_unreadable === RUN TestProbeReadable/read_error_surfaces === RUN TestProbeReadable/stuck_volume_trips_the_timeout_instead_of_blocking --- PASS: TestProbeReadable (0.06s) --- PASS: TestProbeReadable/readable_local_copy (0.00s) --- PASS: TestProbeReadable/final_byte_with_trailing_EOF_counts_as_readable (0.00s) --- PASS: TestProbeReadable/zero-byte_EOF_is_unreadable (0.00s) --- PASS: TestProbeReadable/zero-byte_read_without_error_is_unreadable (0.00s) --- PASS: TestProbeReadable/read_error_surfaces (0.00s) --- PASS: TestProbeReadable/stuck_volume_trips_the_timeout_instead_of_blocking (0.05s) === RUN TestShouldFallBackToRemote === RUN TestShouldFallBackToRemote/matching_remote === RUN TestShouldFallBackToRemote/null_versionId_is_treated_as_unversioned === RUN TestShouldFallBackToRemote/versioned_read_cannot_use_the_unversioned_remote_key === RUN TestShouldFallBackToRemote/latest_read_that_resolves_to_a_version_cannot_use_the_unversioned_key === RUN TestShouldFallBackToRemote/no_remote_entry === RUN TestShouldFallBackToRemote/size_mismatch_is_not_served_as_identical_bytes --- PASS: TestShouldFallBackToRemote (0.12s) --- PASS: TestShouldFallBackToRemote/matching_remote (0.00s) --- PASS: TestShouldFallBackToRemote/null_versionId_is_treated_as_unversioned (0.00s) --- PASS: TestShouldFallBackToRemote/versioned_read_cannot_use_the_unversioned_remote_key (0.00s) --- PASS: TestShouldFallBackToRemote/latest_read_that_resolves_to_a_version_cannot_use_the_unversioned_key (0.00s) --- PASS: TestShouldFallBackToRemote/no_remote_entry (0.00s) --- PASS: TestShouldFallBackToRemote/size_mismatch_is_not_served_as_identical_bytes (0.00s) === RUN TestS3CachedReadFallsBackToRemote === RUN TestS3CachedReadFallsBackToRemote/unreadable_local_copy_is_served_from_the_mounted_remote E0824 22:18:53.537068 reader_at.go:249 fetching chunk &{FileId:1,0123456789ab OffsetInChunk:0 ViewSize:10 ViewOffset:0 ChunkSize:10 CipherKey:[] IsGzipped:false ModifiedTsNs:0}: operation LookupFileId 1,0123456789ab failed, err: volume 1 not found for fileId 1,0123456789ab: provider lookup failed: all 1 filer(s) failed with non-retryable error: filer.LookupVolume failed: rpc error: code = Unimplemented desc = method LookupVolume not implemented volume 1 not found === RUN TestS3CachedReadFallsBackToRemote/range_fallback_serves_the_requested_window_from_the_remote E0824 22:18:53.680058 reader_at.go:249 fetching chunk &{FileId:1,0123456789ab OffsetInChunk:2 ViewSize:4 ViewOffset:2 ChunkSize:10 CipherKey:[] IsGzipped:false ModifiedTsNs:0}: operation LookupFileId 1,0123456789ab failed, err: volume 1 not found for fileId 1,0123456789ab: provider lookup failed: all 1 filer(s) failed with non-retryable error: filer.LookupVolume failed: rpc error: code = Unimplemented desc = method LookupVolume not implemented volume 1 not found === RUN TestS3CachedReadFallsBackToRemote/a_local-only_object_keeps_the_existing_error E0824 22:18:53.731540 reader_at.go:249 fetching chunk &{FileId:1,0123456789ab OffsetInChunk:0 ViewSize:10 ViewOffset:0 ChunkSize:10 CipherKey:[] IsGzipped:false ModifiedTsNs:0}: operation LookupFileId 1,0123456789ab failed, err: volume 1 not found for fileId 1,0123456789ab: provider lookup failed: all 1 filer(s) failed with non-retryable error: filer.LookupVolume failed: rpc error: code = Unimplemented desc = method LookupVolume not implemented volume 1 not found E0824 22:18:53.732513 s3api_object_handlers.go:1201 streamFromVolumeServers: streamFn failed after writing 0 bytes: operation LookupFileId 1,0123456789ab failed, err: volume 1 not found for fileId 1,0123456789ab: provider lookup failed: all 1 filer(s) failed with non-retryable error: filer.LookupVolume failed: rpc error: code = Unimplemented desc = method LookupVolume not implemented volume 1 not found === RUN TestS3CachedReadFallsBackToRemote/versioned_read_does_not_fall_back_to_the_unversioned_remote_key E0824 22:18:53.764526 reader_at.go:249 fetching chunk &{FileId:1,0123456789ab OffsetInChunk:0 ViewSize:10 ViewOffset:0 ChunkSize:10 CipherKey:[] IsGzipped:false ModifiedTsNs:0}: operation LookupFileId 1,0123456789ab failed, err: volume 1 not found for fileId 1,0123456789ab: provider lookup failed: all 1 filer(s) failed with non-retryable error: filer.LookupVolume failed: rpc error: code = Unimplemented desc = method LookupVolume not implemented volume 1 not found E0824 22:18:53.765505 s3api_object_handlers.go:1201 streamFromVolumeServers: streamFn failed after writing 0 bytes: operation LookupFileId 1,0123456789ab failed, err: volume 1 not found for fileId 1,0123456789ab: provider lookup failed: all 1 filer(s) failed with non-retryable error: filer.LookupVolume failed: rpc error: code = Unimplemented desc = method LookupVolume not implemented volume 1 not found --- PASS: TestS3CachedReadFallsBackToRemote (0.32s) --- PASS: TestS3CachedReadFallsBackToRemote/unreadable_local_copy_is_served_from_the_mounted_remote (0.18s) --- PASS: TestS3CachedReadFallsBackToRemote/range_fallback_serves_the_requested_window_from_the_remote (0.07s) --- PASS: TestS3CachedReadFallsBackToRemote/a_local-only_object_keeps_the_existing_error (0.03s) --- PASS: TestS3CachedReadFallsBackToRemote/versioned_read_does_not_fall_back_to_the_unversioned_remote_key (0.03s) === RUN TestStreamRangeToClientFinishesFromRemote === RUN TestStreamRangeToClientFinishesFromRemote/later_chunk_failure_is_finished_from_the_remote === RUN TestStreamRangeToClientFinishesFromRemote/a_short_local_read_reported_as_clean_EOF_is_not_served_truncated === RUN TestStreamRangeToClientFinishesFromRemote/range_read_resumes_at_the_absolute_offset === RUN TestStreamRangeToClientFinishesFromRemote/a_remote_overwritten_since_caching_is_not_spliced_onto_the_local_prefix W0824 22:18:53.909135 s3api_object_handlers.go:1313 streamFromVolumeServers: origin stream mybucket/dir/obj.bin: remote mybucket/dir/obj.bin changed since it was cached === RUN TestStreamRangeToClientFinishesFromRemote/a_local-only_object_keeps_the_mid-stream_error === RUN TestStreamRangeToClientFinishesFromRemote/a_failed_write_to_the_client_is_not_refetched_from_the_remote --- PASS: TestStreamRangeToClientFinishesFromRemote (0.15s) --- PASS: TestStreamRangeToClientFinishesFromRemote/later_chunk_failure_is_finished_from_the_remote (0.04s) --- PASS: TestStreamRangeToClientFinishesFromRemote/a_short_local_read_reported_as_clean_EOF_is_not_served_truncated (0.03s) --- PASS: TestStreamRangeToClientFinishesFromRemote/range_read_resumes_at_the_absolute_offset (0.03s) --- PASS: TestStreamRangeToClientFinishesFromRemote/a_remote_overwritten_since_caching_is_not_spliced_onto_the_local_prefix (0.04s) --- PASS: TestStreamRangeToClientFinishesFromRemote/a_local-only_object_keeps_the_mid-stream_error (0.00s) --- PASS: TestStreamRangeToClientFinishesFromRemote/a_failed_write_to_the_client_is_not_refetched_from_the_remote (0.00s) === RUN TestCheckUploadIDRequiresGeneratedFormat === RUN TestCheckUploadIDRequiresGeneratedFormat/legacy_hash === RUN TestCheckUploadIDRequiresGeneratedFormat/current_format === RUN TestCheckUploadIDRequiresGeneratedFormat/wrong_object_hash E0824 22:18:53.923914 s3api_object_handlers_multipart.go:535 object dir/object and uploadID 0000000000000000000000000000000000000000_aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa are not matched === RUN TestCheckUploadIDRequiresGeneratedFormat/arbitrary_suffix E0824 22:18:53.925589 s3api_object_handlers_multipart.go:535 object dir/object and uploadID 0fb17d4690a62965c850aca2ac98bd432bdd2331_suffix are not matched === RUN TestCheckUploadIDRequiresGeneratedFormat/uppercase_suffix E0824 22:18:53.927237 s3api_object_handlers_multipart.go:535 object dir/object and uploadID 0fb17d4690a62965c850aca2ac98bd432bdd2331_AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA are not matched === RUN TestCheckUploadIDRequiresGeneratedFormat/short_suffix E0824 22:18:53.928469 s3api_object_handlers_multipart.go:535 object dir/object and uploadID 0fb17d4690a62965c850aca2ac98bd432bdd2331_aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa are not matched === RUN TestCheckUploadIDRequiresGeneratedFormat/long_suffix E0824 22:18:53.929415 s3api_object_handlers_multipart.go:535 object dir/object and uploadID 0fb17d4690a62965c850aca2ac98bd432bdd2331_aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa are not matched === RUN TestCheckUploadIDRequiresGeneratedFormat/slash_traversal E0824 22:18:53.930215 s3api_object_handlers_multipart.go:535 object dir/object and uploadID 0fb17d4690a62965c850aca2ac98bd432bdd2331/../../../victim-bucket are not matched === RUN TestCheckUploadIDRequiresGeneratedFormat/suffix_traversal E0824 22:18:53.930858 s3api_object_handlers_multipart.go:535 object dir/object and uploadID 0fb17d4690a62965c850aca2ac98bd432bdd2331_aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa/../../../victim-bucket are not matched === RUN TestCheckUploadIDRequiresGeneratedFormat/backslash_traversal E0824 22:18:53.931676 s3api_object_handlers_multipart.go:535 object dir/object and uploadID 0fb17d4690a62965c850aca2ac98bd432bdd2331\..\..\victim-bucket are not matched === RUN TestCheckUploadIDRequiresGeneratedFormat/nul_suffix E0824 22:18:53.932214 s3api_object_handlers_multipart.go:535 object dir/object and uploadID 0fb17d4690a62965c850aca2ac98bd432bdd2331_abcdef are not matched --- PASS: TestCheckUploadIDRequiresGeneratedFormat (0.01s) --- PASS: TestCheckUploadIDRequiresGeneratedFormat/legacy_hash (0.00s) --- PASS: TestCheckUploadIDRequiresGeneratedFormat/current_format (0.00s) --- PASS: TestCheckUploadIDRequiresGeneratedFormat/wrong_object_hash (0.00s) --- PASS: TestCheckUploadIDRequiresGeneratedFormat/arbitrary_suffix (0.00s) --- PASS: TestCheckUploadIDRequiresGeneratedFormat/uppercase_suffix (0.00s) --- PASS: TestCheckUploadIDRequiresGeneratedFormat/short_suffix (0.00s) --- PASS: TestCheckUploadIDRequiresGeneratedFormat/long_suffix (0.00s) --- PASS: TestCheckUploadIDRequiresGeneratedFormat/slash_traversal (0.00s) --- PASS: TestCheckUploadIDRequiresGeneratedFormat/suffix_traversal (0.00s) --- PASS: TestCheckUploadIDRequiresGeneratedFormat/backslash_traversal (0.00s) --- PASS: TestCheckUploadIDRequiresGeneratedFormat/nul_suffix (0.00s) === RUN TestBuildMultipartSSEKMSReader_RejectsBadIVBeforeAnyFetch === RUN TestBuildMultipartSSEKMSReader_RejectsBadIVBeforeAnyFetch/missing_IV === RUN TestBuildMultipartSSEKMSReader_RejectsBadIVBeforeAnyFetch/empty_IV === RUN TestBuildMultipartSSEKMSReader_RejectsBadIVBeforeAnyFetch/short_IV === RUN TestBuildMultipartSSEKMSReader_RejectsBadIVBeforeAnyFetch/long_IV --- PASS: TestBuildMultipartSSEKMSReader_RejectsBadIVBeforeAnyFetch (0.02s) --- PASS: TestBuildMultipartSSEKMSReader_RejectsBadIVBeforeAnyFetch/missing_IV (0.01s) --- PASS: TestBuildMultipartSSEKMSReader_RejectsBadIVBeforeAnyFetch/empty_IV (0.00s) --- PASS: TestBuildMultipartSSEKMSReader_RejectsBadIVBeforeAnyFetch/short_IV (0.00s) --- PASS: TestBuildMultipartSSEKMSReader_RejectsBadIVBeforeAnyFetch/long_IV (0.00s) === RUN TestBuildMultipartSSEKMSReader_RejectsMissingMetadataBeforeAnyFetch --- PASS: TestBuildMultipartSSEKMSReader_RejectsMissingMetadataBeforeAnyFetch (0.00s) === RUN TestBuildMultipartSSEKMSReader_RejectsUnparseableMetadataBeforeAnyFetch --- PASS: TestBuildMultipartSSEKMSReader_RejectsUnparseableMetadataBeforeAnyFetch (0.00s) === RUN TestBuildMultipartSSEKMSReader_SortsByOffset --- PASS: TestBuildMultipartSSEKMSReader_SortsByOffset (0.00s) === RUN TestNextMarkerWithNestedPrefix === RUN TestNextMarkerWithNestedPrefix/nested_prefix_with_both_requestDir_and_prefix === RUN TestNextMarkerWithNestedPrefix/only_requestDir,_no_prefix === RUN TestNextMarkerWithNestedPrefix/no_requestDir,_only_prefix === RUN TestNextMarkerWithNestedPrefix/deeply_nested_prefix --- PASS: TestNextMarkerWithNestedPrefix (0.00s) --- PASS: TestNextMarkerWithNestedPrefix/nested_prefix_with_both_requestDir_and_prefix (0.00s) --- PASS: TestNextMarkerWithNestedPrefix/only_requestDir,_no_prefix (0.00s) --- PASS: TestNextMarkerWithNestedPrefix/no_requestDir,_only_prefix (0.00s) --- PASS: TestNextMarkerWithNestedPrefix/deeply_nested_prefix (0.00s) === RUN TestNextMarkerWithCommonPrefix === RUN TestNextMarkerWithCommonPrefix/nested_prefix_with_CommonPrefix === RUN TestNextMarkerWithCommonPrefix/only_requestDir_with_CommonPrefix === RUN TestNextMarkerWithCommonPrefix/no_requestDir_with_CommonPrefix --- PASS: TestNextMarkerWithCommonPrefix (0.00s) --- PASS: TestNextMarkerWithCommonPrefix/nested_prefix_with_CommonPrefix (0.00s) --- PASS: TestNextMarkerWithCommonPrefix/only_requestDir_with_CommonPrefix (0.00s) --- PASS: TestNextMarkerWithCommonPrefix/no_requestDir_with_CommonPrefix (0.00s) === RUN TestObjectMetadataUpdateDirectory === RUN TestObjectMetadataUpdateDirectory/non-versioned_nested_key === RUN TestObjectMetadataUpdateDirectory/non-versioned_root_key === RUN TestObjectMetadataUpdateDirectory/non-versioned_deep_key === RUN TestObjectMetadataUpdateDirectory/null_version_nested_key === RUN TestObjectMetadataUpdateDirectory/explicit_null_version_nested_key === RUN TestObjectMetadataUpdateDirectory/empty_version_nested_key === RUN TestObjectMetadataUpdateDirectory/specific_version_nested_key === RUN TestObjectMetadataUpdateDirectory/latest_versioned_nested_key --- PASS: TestObjectMetadataUpdateDirectory (0.00s) --- PASS: TestObjectMetadataUpdateDirectory/non-versioned_nested_key (0.00s) --- PASS: TestObjectMetadataUpdateDirectory/non-versioned_root_key (0.00s) --- PASS: TestObjectMetadataUpdateDirectory/non-versioned_deep_key (0.00s) --- PASS: TestObjectMetadataUpdateDirectory/null_version_nested_key (0.00s) --- PASS: TestObjectMetadataUpdateDirectory/explicit_null_version_nested_key (0.00s) --- PASS: TestObjectMetadataUpdateDirectory/empty_version_nested_key (0.00s) --- PASS: TestObjectMetadataUpdateDirectory/specific_version_nested_key (0.00s) --- PASS: TestObjectMetadataUpdateDirectory/latest_versioned_nested_key (0.00s) === RUN TestDownloadChunkData_AllocationBound s3api_object_handlers_copy_alloc_test.go:105: allocated=17160216 bytes, bound=25165824, output=16777216 --- PASS: TestDownloadChunkData_AllocationBound (0.87s) === RUN TestApplyDestChecksumHeaderToCopyRequest --- PASS: TestApplyDestChecksumHeaderToCopyRequest (0.00s) === RUN TestUploadEntryHasChecksum --- PASS: TestUploadEntryHasChecksum (0.00s) === RUN TestBuildCopyPartResult === RUN TestBuildCopyPartResult/CRC32 === RUN TestBuildCopyPartResult/CRC32C === RUN TestBuildCopyPartResult/CRC64NVME === RUN TestBuildCopyPartResult/SHA1 === RUN TestBuildCopyPartResult/SHA256 === RUN TestBuildCopyPartResult/Unknown --- PASS: TestBuildCopyPartResult (0.01s) --- PASS: TestBuildCopyPartResult/CRC32 (0.00s) --- PASS: TestBuildCopyPartResult/CRC32C (0.00s) --- PASS: TestBuildCopyPartResult/CRC64NVME (0.00s) --- PASS: TestBuildCopyPartResult/SHA1 (0.00s) --- PASS: TestBuildCopyPartResult/SHA256 (0.00s) --- PASS: TestBuildCopyPartResult/Unknown (0.00s) === RUN TestNewChunkUploadOption_AvoidsBytePool === RUN TestNewChunkUploadOption_AvoidsBytePool/empty_chunk === RUN TestNewChunkUploadOption_AvoidsBytePool/small_chunk === RUN TestNewChunkUploadOption_AvoidsBytePool/typical_8_MiB_chunk === RUN TestNewChunkUploadOption_AvoidsBytePool/large_64_MiB_chunk --- PASS: TestNewChunkUploadOption_AvoidsBytePool (0.05s) --- PASS: TestNewChunkUploadOption_AvoidsBytePool/empty_chunk (0.00s) --- PASS: TestNewChunkUploadOption_AvoidsBytePool/small_chunk (0.00s) --- PASS: TestNewChunkUploadOption_AvoidsBytePool/typical_8_MiB_chunk (0.02s) --- PASS: TestNewChunkUploadOption_AvoidsBytePool/large_64_MiB_chunk (0.02s) === RUN TestNewChunkUploadOption_PerCallIsolation --- PASS: TestNewChunkUploadOption_PerCallIsolation (0.00s) === RUN TestCopyDestinationPathResolvesBucketCollection --- PASS: TestCopyDestinationPathResolvesBucketCollection (0.00s) === RUN TestMultipartStorageRuleFollowsDestinationObject --- PASS: TestMultipartStorageRuleFollowsDestinationObject (0.02s) === RUN TestCopyEntryETagPrefersStoredExtendedETag --- PASS: TestCopyEntryETagPrefersStoredExtendedETag (0.00s) === RUN TestCopyEntryETagFallsBackToFilerETag --- PASS: TestCopyEntryETagFallsBackToFilerETag (0.00s) === RUN TestValidateConditionalCopyHeadersUsesStoredExtendedETag --- PASS: TestValidateConditionalCopyHeadersUsesStoredExtendedETag (0.00s) === RUN TestResolveDestinationMime === RUN TestResolveDestinationMime/COPY_keeps_source_mime === RUN TestResolveDestinationMime/COPY_without_request_CT === RUN TestResolveDestinationMime/COPY_with_empty_source_mime === RUN TestResolveDestinationMime/REPLACE_with_request_CT_wins === RUN TestResolveDestinationMime/REPLACE_without_request_CT_uses_default === RUN TestResolveDestinationMime/REPLACE_with_request_CT_and_empty_source === RUN TestResolveDestinationMime/REPLACE_without_request_CT_and_empty_source --- PASS: TestResolveDestinationMime (0.00s) --- PASS: TestResolveDestinationMime/COPY_keeps_source_mime (0.00s) --- PASS: TestResolveDestinationMime/COPY_without_request_CT (0.00s) --- PASS: TestResolveDestinationMime/COPY_with_empty_source_mime (0.00s) --- PASS: TestResolveDestinationMime/REPLACE_with_request_CT_wins (0.00s) --- PASS: TestResolveDestinationMime/REPLACE_without_request_CT_uses_default (0.00s) --- PASS: TestResolveDestinationMime/REPLACE_with_request_CT_and_empty_source (0.00s) --- PASS: TestResolveDestinationMime/REPLACE_without_request_CT_and_empty_source (0.00s) === RUN TestIsValidDirective === RUN TestIsValidDirective/#00 === RUN TestIsValidDirective/COPY === RUN TestIsValidDirective/REPLACE === RUN TestIsValidDirective/copy === RUN TestIsValidDirective/replace === RUN TestIsValidDirective/FOO === RUN TestIsValidDirective/REPLACE_ --- PASS: TestIsValidDirective (0.00s) --- PASS: TestIsValidDirective/#00 (0.00s) --- PASS: TestIsValidDirective/COPY (0.00s) --- PASS: TestIsValidDirective/REPLACE (0.00s) --- PASS: TestIsValidDirective/copy (0.00s) --- PASS: TestIsValidDirective/replace (0.00s) --- PASS: TestIsValidDirective/FOO (0.00s) --- PASS: TestIsValidDirective/REPLACE_ (0.00s) === RUN TestProcessMetadataBytes_ReplaceSystemHeaders --- PASS: TestProcessMetadataBytes_ReplaceSystemHeaders (0.00s) === RUN TestIsManagedCopyMetadataKey_CoversSystemHeaders --- PASS: TestIsManagedCopyMetadataKey_CoversSystemHeaders (0.00s) === RUN TestIsManagedCopyMetadataKey_CaseInsensitive === RUN TestIsManagedCopyMetadataKey_CaseInsensitive/cache-control === RUN TestIsManagedCopyMetadataKey_CaseInsensitive/CACHE-CONTROL === RUN TestIsManagedCopyMetadataKey_CaseInsensitive/content-encoding === RUN TestIsManagedCopyMetadataKey_CaseInsensitive/x-amz-meta-owner === RUN TestIsManagedCopyMetadataKey_CaseInsensitive/X-AMZ-META-OWNER === RUN TestIsManagedCopyMetadataKey_CaseInsensitive/x-amz-tagging-env --- PASS: TestIsManagedCopyMetadataKey_CaseInsensitive (0.01s) --- PASS: TestIsManagedCopyMetadataKey_CaseInsensitive/cache-control (0.00s) --- PASS: TestIsManagedCopyMetadataKey_CaseInsensitive/CACHE-CONTROL (0.00s) --- PASS: TestIsManagedCopyMetadataKey_CaseInsensitive/content-encoding (0.00s) --- PASS: TestIsManagedCopyMetadataKey_CaseInsensitive/x-amz-meta-owner (0.00s) --- PASS: TestIsManagedCopyMetadataKey_CaseInsensitive/X-AMZ-META-OWNER (0.00s) --- PASS: TestIsManagedCopyMetadataKey_CaseInsensitive/x-amz-tagging-env (0.00s) === RUN TestProcessMetadataBytes_CopyAcceptsLowercaseSourceKeys --- PASS: TestProcessMetadataBytes_CopyAcceptsLowercaseSourceKeys (0.00s) === RUN TestProcessMetadataBytes_CopyCanonicalSystemHeaderWinsOverLegacy --- PASS: TestProcessMetadataBytes_CopyCanonicalSystemHeaderWinsOverLegacy (0.00s) === RUN TestProcessMetadataBytes_CopyCanonicalTagWinsOverLegacy --- PASS: TestProcessMetadataBytes_CopyCanonicalTagWinsOverLegacy (0.00s) === RUN TestProcessMetadataBytes_CopyAcceptsLowercaseTagKeys --- PASS: TestProcessMetadataBytes_CopyAcceptsLowercaseTagKeys (0.00s) === RUN TestMergeCopyMetadata_ReplaceDropsStaleSystemHeader --- PASS: TestMergeCopyMetadata_ReplaceDropsStaleSystemHeader (0.00s) === RUN TestMergeCopyMetadata_ReplaceTaggingDropsStaleTags --- PASS: TestMergeCopyMetadata_ReplaceTaggingDropsStaleTags (0.00s) === RUN TestProcessMetadataBytes_CopyInheritsSystemHeaders --- PASS: TestProcessMetadataBytes_CopyInheritsSystemHeaders (0.00s) === RUN TestProcessMetadataBytes_CopyAppliesRequestedSSES3Header --- PASS: TestProcessMetadataBytes_CopyAppliesRequestedSSES3Header (0.00s) === RUN TestProcessMetadataBytes_CopyPreservesSourceSSEWhenRequestOmitsHeader --- PASS: TestProcessMetadataBytes_CopyPreservesSourceSSEWhenRequestOmitsHeader (0.00s) === RUN TestCopyReplacesSourceEntry === RUN TestCopyReplacesSourceEntry/no_versioning_replaces_the_bare_key === RUN TestCopyReplacesSourceEntry/a_copy_to_another_key_writes_its_own_entry === RUN TestCopyReplacesSourceEntry/versioning_enabled_writes_a_new_version_file === RUN TestCopyReplacesSourceEntry/suspended_writes_the_null_version_beside_live_versions === RUN TestCopyReplacesSourceEntry/pinned_source_version_outlives_the_copy === RUN TestCopyReplacesSourceEntry/pinned_null_source_version --- PASS: TestCopyReplacesSourceEntry (0.01s) --- PASS: TestCopyReplacesSourceEntry/no_versioning_replaces_the_bare_key (0.00s) --- PASS: TestCopyReplacesSourceEntry/a_copy_to_another_key_writes_its_own_entry (0.00s) --- PASS: TestCopyReplacesSourceEntry/versioning_enabled_writes_a_new_version_file (0.00s) --- PASS: TestCopyReplacesSourceEntry/suspended_writes_the_null_version_beside_live_versions (0.00s) --- PASS: TestCopyReplacesSourceEntry/pinned_source_version_outlives_the_copy (0.00s) --- PASS: TestCopyReplacesSourceEntry/pinned_null_source_version (0.00s) === RUN TestValidateDeleteObjectIdentifier === RUN TestValidateDeleteObjectIdentifier/clean_key === RUN TestValidateDeleteObjectIdentifier/clean_version === RUN TestValidateDeleteObjectIdentifier/key_traversal === RUN TestValidateDeleteObjectIdentifier/encoded_traversal_already_decoded === RUN TestValidateDeleteObjectIdentifier/backslash_traversal === RUN TestValidateDeleteObjectIdentifier/version_traversal === RUN TestValidateDeleteObjectIdentifier/version_backslash --- PASS: TestValidateDeleteObjectIdentifier (0.00s) --- PASS: TestValidateDeleteObjectIdentifier/clean_key (0.00s) --- PASS: TestValidateDeleteObjectIdentifier/clean_version (0.00s) --- PASS: TestValidateDeleteObjectIdentifier/key_traversal (0.00s) --- PASS: TestValidateDeleteObjectIdentifier/encoded_traversal_already_decoded (0.00s) --- PASS: TestValidateDeleteObjectIdentifier/backslash_traversal (0.00s) --- PASS: TestValidateDeleteObjectIdentifier/version_traversal (0.00s) --- PASS: TestValidateDeleteObjectIdentifier/version_backslash (0.00s) === RUN TestGetSpecificObjectVersionRejectsUnsafeVersionID --- PASS: TestGetSpecificObjectVersionRejectsUnsafeVersionID (0.00s) === RUN TestDeleteUnversionedObjectWithClient_MetadataOnlySkipsChunkDelete --- PASS: TestDeleteUnversionedObjectWithClient_MetadataOnlySkipsChunkDelete (0.00s) === RUN TestDeleteUnversionedObjectWithClient_FullDeletePreservesIsDeleteData --- PASS: TestDeleteUnversionedObjectWithClient_FullDeletePreservesIsDeleteData (0.00s) === RUN TestDeleteUnversionedObjectWithClient_FullPathFromBucketsRoot --- PASS: TestDeleteUnversionedObjectWithClient_FullPathFromBucketsRoot (0.00s) === RUN TestDeleteUnversionedObjectWithClientRejectsTraversal --- PASS: TestDeleteUnversionedObjectWithClientRejectsTraversal (0.00s) === RUN TestDeleteUnversionedObjectWithClient_PropagatesEntryAttributesIrrelevant --- PASS: TestDeleteUnversionedObjectWithClient_PropagatesEntryAttributesIrrelevant (0.00s) === RUN TestServeDirectoryContentContentType === RUN TestServeDirectoryContentContentType/bare_directory_answers_the_directory_marker_type === RUN TestServeDirectoryContentContentType/promoted_file_with_data_keeps_octet-stream === RUN TestServeDirectoryContentContentType/stored_mime_is_echoed_verbatim --- PASS: TestServeDirectoryContentContentType (0.00s) --- PASS: TestServeDirectoryContentContentType/bare_directory_answers_the_directory_marker_type (0.00s) --- PASS: TestServeDirectoryContentContentType/promoted_file_with_data_keeps_octet-stream (0.00s) --- PASS: TestServeDirectoryContentContentType/stored_mime_is_echoed_verbatim (0.00s) === RUN TestDeletedPrefixIsNotACommonPrefix --- PASS: TestDeletedPrefixIsNotACommonPrefix (0.00s) === RUN TestLivePrefixIsStillACommonPrefix --- PASS: TestLivePrefixIsStillACommonPrefix (0.00s) === RUN TestDeletedPrefixWithNullObjectIsNotACommonPrefix --- PASS: TestDeletedPrefixWithNullObjectIsNotACommonPrefix (0.00s) === RUN TestDeletedPrefixGetsNoDirectoryMarker --- PASS: TestDeletedPrefixGetsNoDirectoryMarker (0.00s) === RUN TestEmptyDirectoryStaysACommonPrefix --- PASS: TestEmptyDirectoryStaysACommonPrefix (0.00s) === RUN TestDirectoryKeyObjectListedDespiteDeletedChildren --- PASS: TestDirectoryKeyObjectListedDespiteDeletedChildren (0.00s) === RUN TestUnstampedVersionKeepsThePrefix --- PASS: TestUnstampedVersionKeepsThePrefix (0.00s) === RUN TestLiveVersionStampKeepsThePrefix --- PASS: TestLiveVersionStampKeepsThePrefix (0.00s) === RUN TestBucketWithoutVersioningSkipsTheProbe --- PASS: TestBucketWithoutVersioningSkipsTheProbe (0.00s) === RUN TestDeletedPrefixesDoNotConsumeMaxKeys --- PASS: TestDeletedPrefixesDoNotConsumeMaxKeys (0.00s) === RUN TestDeletedDirectoryMarkerIsNotListed --- PASS: TestDeletedDirectoryMarkerIsNotListed (0.00s) === RUN TestLiveDirectoryMarkerIsListed --- PASS: TestLiveDirectoryMarkerIsListed (0.00s) === RUN TestDeletedDirectoryMarkerKeepsLiveChildren --- PASS: TestDeletedDirectoryMarkerKeepsLiveChildren (0.00s) === RUN TestStaleOwnVersionsIsNotAKey --- PASS: TestStaleOwnVersionsIsNotAKey (0.00s) === RUN TestDirectoryMarkersCostNoLookups --- PASS: TestDirectoryMarkersCostNoLookups (0.00s) === RUN TestDirectoryListedAsCommonPrefix --- PASS: TestDirectoryListedAsCommonPrefix (0.00s) === RUN TestEmptyDirectorySurfacedAsMarker --- PASS: TestEmptyDirectorySurfacedAsMarker (0.00s) === RUN TestNonEmptyDirectoryGetsNoPhantomMarker --- PASS: TestNonEmptyDirectoryGetsNoPhantomMarker (0.00s) === RUN TestEmptyDirectoryHiddenInFlatListing --- PASS: TestEmptyDirectoryHiddenInFlatListing (0.00s) === RUN TestListObjectsHandler --- PASS: TestListObjectsHandler (0.00s) === RUN TestListObjectsV1NamespaceResponse --- PASS: TestListObjectsV1NamespaceResponse (0.00s) === RUN Test_normalizePrefixMarker === RUN Test_normalizePrefixMarker/bucket_root_listing_with_delimiter === RUN Test_normalizePrefixMarker/prefix_is_a_directory === RUN Test_normalizePrefixMarker/normal_case === RUN Test_normalizePrefixMarker/empty_prefix === RUN Test_normalizePrefixMarker/empty_directory === RUN Test_normalizePrefixMarker/partial_name_prefix,_marker_resumes_inside_a_matching_subdirectory === RUN Test_normalizePrefixMarker/partial_name_prefix,_marker_resumes_inside_a_matching_sibling_directory === RUN Test_normalizePrefixMarker/top-level_partial_name_prefix,_marker_resumes_inside_a_matching_subdirectory --- PASS: Test_normalizePrefixMarker (0.01s) --- PASS: Test_normalizePrefixMarker/bucket_root_listing_with_delimiter (0.00s) --- PASS: Test_normalizePrefixMarker/prefix_is_a_directory (0.00s) --- PASS: Test_normalizePrefixMarker/normal_case (0.00s) --- PASS: Test_normalizePrefixMarker/empty_prefix (0.00s) --- PASS: Test_normalizePrefixMarker/empty_directory (0.00s) --- PASS: Test_normalizePrefixMarker/partial_name_prefix,_marker_resumes_inside_a_matching_subdirectory (0.00s) --- PASS: Test_normalizePrefixMarker/partial_name_prefix,_marker_resumes_inside_a_matching_sibling_directory (0.00s) --- PASS: Test_normalizePrefixMarker/top-level_partial_name_prefix,_marker_resumes_inside_a_matching_subdirectory (0.00s) === RUN TestBuildTruncatedNextMarker === RUN TestBuildTruncatedNextMarker/does_not_duplicate_prefix_segment_in_next_continuation_token === RUN TestBuildTruncatedNextMarker/keeps_common_prefix_marker_trailing_slash === RUN TestBuildTruncatedNextMarker/keeps_common_prefix_marker_when_request_dir_is_empty === RUN TestBuildTruncatedNextMarker/does_not_fold_a_partial_name_prefix_into_the_common_prefix_marker --- PASS: TestBuildTruncatedNextMarker (0.00s) --- PASS: TestBuildTruncatedNextMarker/does_not_duplicate_prefix_segment_in_next_continuation_token (0.00s) --- PASS: TestBuildTruncatedNextMarker/keeps_common_prefix_marker_trailing_slash (0.00s) --- PASS: TestBuildTruncatedNextMarker/keeps_common_prefix_marker_when_request_dir_is_empty (0.00s) --- PASS: TestBuildTruncatedNextMarker/does_not_fold_a_partial_name_prefix_into_the_common_prefix_marker (0.00s) === RUN TestAllowUnorderedParameterValidation === RUN TestAllowUnorderedParameterValidation/getListObjectsV1Args_with_allow-unordered === RUN TestAllowUnorderedParameterValidation/getListObjectsV2Args_with_allow-unordered --- PASS: TestAllowUnorderedParameterValidation (0.02s) --- PASS: TestAllowUnorderedParameterValidation/getListObjectsV1Args_with_allow-unordered (0.00s) --- PASS: TestAllowUnorderedParameterValidation/getListObjectsV2Args_with_allow-unordered (0.00s) === RUN TestDoListFilerEntries_BucketRootPrefixSlashDelimiterSlash_ListsDirectories --- PASS: TestDoListFilerEntries_BucketRootPrefixSlashDelimiterSlash_ListsDirectories (0.00s) === RUN TestDoListFilerEntries_ExclusiveStartSkipsMarkerEcho --- PASS: TestDoListFilerEntries_ExclusiveStartSkipsMarkerEcho (0.00s) === RUN TestDoListFilerEntries_ExclusiveStartSkipsMarkerEchoWithSubsequentEntries --- PASS: TestDoListFilerEntries_ExclusiveStartSkipsMarkerEchoWithSubsequentEntries (0.00s) === RUN TestDoListFilerEntries_EmptyDirectoriesDoNotHideLaterEntries --- PASS: TestDoListFilerEntries_EmptyDirectoriesDoNotHideLaterEntries (0.00s) === RUN TestDoListFilerEntries_EmptyDirectoriesInSubdirectoryDoNotHideSiblings --- PASS: TestDoListFilerEntries_EmptyDirectoriesInSubdirectoryDoNotHideSiblings (0.00s) === RUN TestDoListFilerEntries_TruncationAcrossEmptyDirectories --- PASS: TestDoListFilerEntries_TruncationAcrossEmptyDirectories (0.00s) === RUN TestAllowUnorderedWithDelimiterValidation === RUN TestAllowUnorderedWithDelimiterValidation/should_return_error_when_allow-unordered=true_and_delimiter_are_both_present === RUN TestAllowUnorderedWithDelimiterValidation/should_allow_allow-unordered=true_without_delimiter === RUN TestAllowUnorderedWithDelimiterValidation/should_allow_delimiter_without_allow-unordered --- PASS: TestAllowUnorderedWithDelimiterValidation (0.02s) --- PASS: TestAllowUnorderedWithDelimiterValidation/should_return_error_when_allow-unordered=true_and_delimiter_are_both_present (0.01s) --- PASS: TestAllowUnorderedWithDelimiterValidation/should_allow_allow-unordered=true_without_delimiter (0.00s) --- PASS: TestAllowUnorderedWithDelimiterValidation/should_allow_delimiter_without_allow-unordered (0.00s) === RUN TestSanitizeV1MarkerEcho_NoProgressGuard --- PASS: TestSanitizeV1MarkerEcho_NoProgressGuard (0.00s) === RUN TestMaxKeysParameterValidation === RUN TestMaxKeysParameterValidation/valid_max-keys_values_should_work === RUN TestMaxKeysParameterValidation/invalid_max-keys_values_should_return_error === RUN TestMaxKeysParameterValidation/empty_max-keys_should_use_default --- PASS: TestMaxKeysParameterValidation (0.00s) --- PASS: TestMaxKeysParameterValidation/valid_max-keys_values_should_work (0.00s) --- PASS: TestMaxKeysParameterValidation/invalid_max-keys_values_should_return_error (0.00s) --- PASS: TestMaxKeysParameterValidation/empty_max-keys_should_use_default (0.00s) === RUN TestDelimiterWithDirectoryKeyObjects === RUN TestDelimiterWithDirectoryKeyObjects/directory_key_object_should_be_grouped_into_common_prefix_with_delimiter === RUN TestDelimiterWithDirectoryKeyObjects/directory_key_object_without_delimiter_should_be_individual_key --- PASS: TestDelimiterWithDirectoryKeyObjects (0.00s) --- PASS: TestDelimiterWithDirectoryKeyObjects/directory_key_object_should_be_grouped_into_common_prefix_with_delimiter (0.00s) --- PASS: TestDelimiterWithDirectoryKeyObjects/directory_key_object_without_delimiter_should_be_individual_key (0.00s) === RUN TestObjectLevelListPermissions === RUN TestObjectLevelListPermissions/Identity_CanDo_Object_Level_Permissions === RUN TestObjectLevelListPermissions/Identity_CanDo_Object_Level_Permissions/allowed_prefix_exact_match === RUN TestObjectLevelListPermissions/Identity_CanDo_Object_Level_Permissions/allowed_prefix_subdirectory === RUN TestObjectLevelListPermissions/Identity_CanDo_Object_Level_Permissions/denied_different_prefix === RUN TestObjectLevelListPermissions/Identity_CanDo_Object_Level_Permissions/denied_different_bucket === RUN TestObjectLevelListPermissions/Identity_CanDo_Object_Level_Permissions/denied_root_level === RUN TestObjectLevelListPermissions/Bucket_Level_Permissions_Still_Work === RUN TestObjectLevelListPermissions/Empty_Object_With_Prefix_Logic === RUN TestObjectLevelListPermissions/Empty_Object_With_Prefix_Logic/empty_object_with_prefix === RUN TestObjectLevelListPermissions/Empty_Object_With_Prefix_Logic/slash_object_with_prefix === RUN TestObjectLevelListPermissions/Empty_Object_With_Prefix_Logic/object_already_set === RUN TestObjectLevelListPermissions/Empty_Object_With_Prefix_Logic/no_prefix_provided === RUN TestObjectLevelListPermissions/Issue_7039_Scenario === NAME TestObjectLevelListPermissions s3api_object_handlers_list_test.go:876: This test validates the fix for issue #7039 s3api_object_handlers_list_test.go:877: Object-level List permissions like 'List:bucket/prefix/*' now work correctly s3api_object_handlers_list_test.go:878: Middleware properly extracts prefix for permission validation --- PASS: TestObjectLevelListPermissions (0.01s) --- PASS: TestObjectLevelListPermissions/Identity_CanDo_Object_Level_Permissions (0.00s) --- PASS: TestObjectLevelListPermissions/Identity_CanDo_Object_Level_Permissions/allowed_prefix_exact_match (0.00s) --- PASS: TestObjectLevelListPermissions/Identity_CanDo_Object_Level_Permissions/allowed_prefix_subdirectory (0.00s) --- PASS: TestObjectLevelListPermissions/Identity_CanDo_Object_Level_Permissions/denied_different_prefix (0.00s) --- PASS: TestObjectLevelListPermissions/Identity_CanDo_Object_Level_Permissions/denied_different_bucket (0.00s) --- PASS: TestObjectLevelListPermissions/Identity_CanDo_Object_Level_Permissions/denied_root_level (0.00s) --- PASS: TestObjectLevelListPermissions/Bucket_Level_Permissions_Still_Work (0.00s) --- PASS: TestObjectLevelListPermissions/Empty_Object_With_Prefix_Logic (0.00s) --- PASS: TestObjectLevelListPermissions/Empty_Object_With_Prefix_Logic/empty_object_with_prefix (0.00s) --- PASS: TestObjectLevelListPermissions/Empty_Object_With_Prefix_Logic/slash_object_with_prefix (0.00s) --- PASS: TestObjectLevelListPermissions/Empty_Object_With_Prefix_Logic/object_already_set (0.00s) --- PASS: TestObjectLevelListPermissions/Empty_Object_With_Prefix_Logic/no_prefix_provided (0.00s) --- PASS: TestObjectLevelListPermissions/Issue_7039_Scenario (0.00s) === RUN TestListObjectsV2_Regression --- PASS: TestListObjectsV2_Regression (0.00s) === RUN TestListObjectsV2_Regression_Sorting --- PASS: TestListObjectsV2_Regression_Sorting (0.00s) === RUN TestListObjectsV2_PrefixEndingWithSlash_DoesNotMatchSiblings --- PASS: TestListObjectsV2_PrefixEndingWithSlash_DoesNotMatchSiblings (0.00s) === RUN TestListObjectsV2_PrefixEndingWithSlash_WithDelimiter --- PASS: TestListObjectsV2_PrefixEndingWithSlash_WithDelimiter (0.00s) === RUN TestListObjectsWithVersionedObjects === RUN TestListObjectsWithVersionedObjects/List_all_objects_including_versioned_(no_delimiter) === RUN TestListObjectsWithVersionedObjects/List_bucket_root_with_delimiter --- PASS: TestListObjectsWithVersionedObjects (0.00s) --- PASS: TestListObjectsWithVersionedObjects/List_all_objects_including_versioned_(no_delimiter) (0.00s) --- PASS: TestListObjectsWithVersionedObjects/List_bucket_root_with_delimiter (0.00s) === RUN TestVersionedObjectsNoDuplication --- PASS: TestVersionedObjectsNoDuplication (0.00s) === RUN TestVersionedObjectsWithDeleteMarker --- PASS: TestVersionedObjectsWithDeleteMarker (0.00s) === RUN TestVersionedObjectsMaxKeys --- PASS: TestVersionedObjectsMaxKeys (0.00s) === RUN TestVersionsDirectoryNotTraversed --- PASS: TestVersionsDirectoryNotTraversed (0.00s) === RUN TestListObjectVersionsResult_XMLInterleaving --- PASS: TestListObjectVersionsResult_XMLInterleaving (0.00s) === RUN TestListObjectVersionsResult_XMLInterleavingMultipleKeys --- PASS: TestListObjectVersionsResult_XMLInterleavingMultipleKeys (0.00s) === RUN TestListObjectVersionsResult_XMLCommonPrefixes --- PASS: TestListObjectVersionsResult_XMLCommonPrefixes (0.00s) === RUN TestListObjectVersions_PrefixWithLeadingSlash === RUN TestListObjectVersions_PrefixWithLeadingSlash/Prefix_without_leading_slash_matches_file === RUN TestListObjectVersions_PrefixWithLeadingSlash/Prefix_with_leading_slash_(bug_fix_test)_-_normalized_and_matches_file === RUN TestListObjectVersions_PrefixWithLeadingSlash/Normalized_prefix_matches_subdirectory_file === RUN TestListObjectVersions_PrefixWithLeadingSlash/Normalized_prefix_does_not_match_different_path === RUN TestListObjectVersions_PrefixWithLeadingSlash/Prefix_with_leading_slash_allows_descending_into_directory === RUN TestListObjectVersions_PrefixWithLeadingSlash/Prefix_with_leading_slash_matches_directory_with_trailing_slash --- PASS: TestListObjectVersions_PrefixWithLeadingSlash (0.00s) --- PASS: TestListObjectVersions_PrefixWithLeadingSlash/Prefix_without_leading_slash_matches_file (0.00s) --- PASS: TestListObjectVersions_PrefixWithLeadingSlash/Prefix_with_leading_slash_(bug_fix_test)_-_normalized_and_matches_file (0.00s) --- PASS: TestListObjectVersions_PrefixWithLeadingSlash/Normalized_prefix_matches_subdirectory_file (0.00s) --- PASS: TestListObjectVersions_PrefixWithLeadingSlash/Normalized_prefix_does_not_match_different_path (0.00s) --- PASS: TestListObjectVersions_PrefixWithLeadingSlash/Prefix_with_leading_slash_allows_descending_into_directory (0.00s) --- PASS: TestListObjectVersions_PrefixWithLeadingSlash/Prefix_with_leading_slash_matches_directory_with_trailing_slash (0.00s) === RUN TestComputeStartFrom === RUN TestComputeStartFrom/empty_marker === RUN TestComputeStartFrom/empty_marker_with_path === RUN TestComputeStartFrom/root_level_file === RUN TestComputeStartFrom/root_level_with_subpath === RUN TestComputeStartFrom/matching_subdir === RUN TestComputeStartFrom/deeper_subdir === RUN TestComputeStartFrom/at_leaf_level === RUN TestComputeStartFrom/unrelated_directory === RUN TestComputeStartFrom/marker_equals_relativePath === RUN TestComputeStartFrom/marker_before_directory --- PASS: TestComputeStartFrom (0.00s) --- PASS: TestComputeStartFrom/empty_marker (0.00s) --- PASS: TestComputeStartFrom/empty_marker_with_path (0.00s) --- PASS: TestComputeStartFrom/root_level_file (0.00s) --- PASS: TestComputeStartFrom/root_level_with_subpath (0.00s) --- PASS: TestComputeStartFrom/matching_subdir (0.00s) --- PASS: TestComputeStartFrom/deeper_subdir (0.00s) --- PASS: TestComputeStartFrom/at_leaf_level (0.00s) --- PASS: TestComputeStartFrom/unrelated_directory (0.00s) --- PASS: TestComputeStartFrom/marker_equals_relativePath (0.00s) --- PASS: TestComputeStartFrom/marker_before_directory (0.00s) === RUN TestProcessDirectorySkipsBeforeMarker === RUN TestProcessDirectorySkipsBeforeMarker/no_marker === RUN TestProcessDirectorySkipsBeforeMarker/dir_before_marker === RUN TestProcessDirectorySkipsBeforeMarker/marker_descends_into_dir === RUN TestProcessDirectorySkipsBeforeMarker/dir_after_marker === RUN TestProcessDirectorySkipsBeforeMarker/same_prefix_different_suffix --- PASS: TestProcessDirectorySkipsBeforeMarker (0.00s) --- PASS: TestProcessDirectorySkipsBeforeMarker/no_marker (0.00s) --- PASS: TestProcessDirectorySkipsBeforeMarker/dir_before_marker (0.00s) --- PASS: TestProcessDirectorySkipsBeforeMarker/marker_descends_into_dir (0.00s) --- PASS: TestProcessDirectorySkipsBeforeMarker/dir_after_marker (0.00s) --- PASS: TestProcessDirectorySkipsBeforeMarker/same_prefix_different_suffix (0.00s) === RUN TestVersionedListSkipsMarkerObject --- PASS: TestVersionedListSkipsMarkerObject (0.00s) === RUN TestVersionedListSkipsEchoedVersionsMarker --- PASS: TestVersionedListSkipsEchoedVersionsMarker (0.00s) === RUN TestPostPolicyKeyNormalization === RUN TestPostPolicyKeyNormalization/key_without_leading_slash === RUN TestPostPolicyKeyNormalization/key_with_leading_slash === RUN TestPostPolicyKeyNormalization/key_with_path_without_leading_slash === RUN TestPostPolicyKeyNormalization/key_with_path_with_leading_slash === RUN TestPostPolicyKeyNormalization/simple_filename === RUN TestPostPolicyKeyNormalization/key_with_duplicate_slashes === RUN TestPostPolicyKeyNormalization/key_with_leading_duplicate_slashes === RUN TestPostPolicyKeyNormalization/key_with_trailing_slash --- PASS: TestPostPolicyKeyNormalization (0.00s) --- PASS: TestPostPolicyKeyNormalization/key_without_leading_slash (0.00s) --- PASS: TestPostPolicyKeyNormalization/key_with_leading_slash (0.00s) --- PASS: TestPostPolicyKeyNormalization/key_with_path_without_leading_slash (0.00s) --- PASS: TestPostPolicyKeyNormalization/key_with_path_with_leading_slash (0.00s) --- PASS: TestPostPolicyKeyNormalization/simple_filename (0.00s) --- PASS: TestPostPolicyKeyNormalization/key_with_duplicate_slashes (0.00s) --- PASS: TestPostPolicyKeyNormalization/key_with_leading_duplicate_slashes (0.00s) --- PASS: TestPostPolicyKeyNormalization/key_with_trailing_slash (0.00s) === RUN TestNormalizeObjectKey === RUN TestNormalizeObjectKey/empty_string === RUN TestNormalizeObjectKey/simple_file === RUN TestNormalizeObjectKey/with_leading_slash === RUN TestNormalizeObjectKey/path_without_slash === RUN TestNormalizeObjectKey/path_with_slash === RUN TestNormalizeObjectKey/duplicate_slashes === RUN TestNormalizeObjectKey/leading_duplicates === RUN TestNormalizeObjectKey/all_duplicates === RUN TestNormalizeObjectKey/just_slashes === RUN TestNormalizeObjectKey/trailing_slash === RUN TestNormalizeObjectKey/backslash_to_forward_slash === RUN TestNormalizeObjectKey/windows_path === RUN TestNormalizeObjectKey/mixed_slashes --- PASS: TestNormalizeObjectKey (0.01s) --- PASS: TestNormalizeObjectKey/empty_string (0.00s) --- PASS: TestNormalizeObjectKey/simple_file (0.00s) --- PASS: TestNormalizeObjectKey/with_leading_slash (0.00s) --- PASS: TestNormalizeObjectKey/path_without_slash (0.00s) --- PASS: TestNormalizeObjectKey/path_with_slash (0.00s) --- PASS: TestNormalizeObjectKey/duplicate_slashes (0.00s) --- PASS: TestNormalizeObjectKey/leading_duplicates (0.00s) --- PASS: TestNormalizeObjectKey/all_duplicates (0.00s) --- PASS: TestNormalizeObjectKey/just_slashes (0.00s) --- PASS: TestNormalizeObjectKey/trailing_slash (0.00s) --- PASS: TestNormalizeObjectKey/backslash_to_forward_slash (0.00s) --- PASS: TestNormalizeObjectKey/windows_path (0.00s) --- PASS: TestNormalizeObjectKey/mixed_slashes (0.00s) === RUN TestPostPolicyFilenameSubstitution === RUN TestPostPolicyFilenameSubstitution/filename_at_end === RUN TestPostPolicyFilenameSubstitution/filename_in_middle === RUN TestPostPolicyFilenameSubstitution/no_substitution_needed === RUN TestPostPolicyFilenameSubstitution/filename_only --- PASS: TestPostPolicyFilenameSubstitution (0.00s) --- PASS: TestPostPolicyFilenameSubstitution/filename_at_end (0.00s) --- PASS: TestPostPolicyFilenameSubstitution/filename_in_middle (0.00s) --- PASS: TestPostPolicyFilenameSubstitution/no_substitution_needed (0.00s) --- PASS: TestPostPolicyFilenameSubstitution/filename_only (0.00s) === RUN TestExtractPostPolicyFormValues === RUN TestExtractPostPolicyFormValues/basic_upload === RUN TestExtractPostPolicyFormValues/upload_with_path_key --- PASS: TestExtractPostPolicyFormValues (0.00s) --- PASS: TestExtractPostPolicyFormValues/basic_upload (0.00s) --- PASS: TestExtractPostPolicyFormValues/upload_with_path_key (0.00s) === RUN TestPostPolicyPathConstruction === RUN TestPostPolicyPathConstruction/simple_key_without_slash_-_the_bug_case === RUN TestPostPolicyPathConstruction/simple_key_with_slash === RUN TestPostPolicyPathConstruction/nested_path_without_leading_slash === RUN TestPostPolicyPathConstruction/nested_path_with_leading_slash === RUN TestPostPolicyPathConstruction/key_with_duplicate_slashes --- PASS: TestPostPolicyPathConstruction (0.01s) --- PASS: TestPostPolicyPathConstruction/simple_key_without_slash_-_the_bug_case (0.00s) --- PASS: TestPostPolicyPathConstruction/simple_key_with_slash (0.00s) --- PASS: TestPostPolicyPathConstruction/nested_path_without_leading_slash (0.00s) --- PASS: TestPostPolicyPathConstruction/nested_path_with_leading_slash (0.00s) --- PASS: TestPostPolicyPathConstruction/key_with_duplicate_slashes (0.00s) === RUN TestApplyPostPolicyFormHeaders_ForwardsAcl --- PASS: TestApplyPostPolicyFormHeaders_ForwardsAcl (0.00s) === RUN TestApplyPostPolicyFormHeaders_ForwardsContentHeaders === RUN TestApplyPostPolicyFormHeaders_ForwardsContentHeaders/Content-Encoding === RUN TestApplyPostPolicyFormHeaders_ForwardsContentHeaders/Content-Language --- PASS: TestApplyPostPolicyFormHeaders_ForwardsContentHeaders (0.00s) --- PASS: TestApplyPostPolicyFormHeaders_ForwardsContentHeaders/Content-Encoding (0.00s) --- PASS: TestApplyPostPolicyFormHeaders_ForwardsContentHeaders/Content-Language (0.00s) === RUN TestApplyPostPolicyFormHeaders_ForwardsXAmzHeaders --- PASS: TestApplyPostPolicyFormHeaders_ForwardsXAmzHeaders (0.00s) === RUN TestApplyPostPolicyFormHeaders_SkipsReserved --- PASS: TestApplyPostPolicyFormHeaders_SkipsReserved (0.00s) === RUN TestApplyPostPolicyFormHeaders_KeepsExistingCacheControl --- PASS: TestApplyPostPolicyFormHeaders_KeepsExistingCacheControl (0.00s) === RUN TestApplyPostPolicyFormHeaders_IgnoresContentType --- PASS: TestApplyPostPolicyFormHeaders_IgnoresContentType (0.00s) === RUN TestPostPolicyBucketHandlerKeyExtraction === RUN TestPostPolicyBucketHandlerKeyExtraction/key_without_leading_slash === RUN TestPostPolicyBucketHandlerKeyExtraction/key_with_leading_slash === RUN TestPostPolicyBucketHandlerKeyExtraction/key_with_duplicate_slashes --- PASS: TestPostPolicyBucketHandlerKeyExtraction (0.00s) --- PASS: TestPostPolicyBucketHandlerKeyExtraction/key_without_leading_slash (0.00s) --- PASS: TestPostPolicyBucketHandlerKeyExtraction/key_with_leading_slash (0.00s) --- PASS: TestPostPolicyBucketHandlerKeyExtraction/key_with_duplicate_slashes (0.00s) === RUN TestPostPolicyBucketHandlerRejectsTraversalKey --- PASS: TestPostPolicyBucketHandlerRejectsTraversalKey (0.00s) === RUN TestPostPolicyBucketHandler_PolicyViolationReturns403 --- PASS: TestPostPolicyBucketHandler_PolicyViolationReturns403 (0.00s) === RUN TestDetectRequestedChecksumAlgorithm === RUN TestDetectRequestedChecksumAlgorithm/sdk_algorithm_header === RUN TestDetectRequestedChecksumAlgorithm/algorithm_header === RUN TestDetectRequestedChecksumAlgorithm/presigned_url_hoists_sdk_algorithm_to_query === RUN TestDetectRequestedChecksumAlgorithm/presigned_url_hoists_checksum-algorithm_to_query_(lowercase) === RUN TestDetectRequestedChecksumAlgorithm/presigned_url_hoists_individual_checksum_value === RUN TestDetectRequestedChecksumAlgorithm/unsupported_in_query_returns_error W0824 22:18:55.161500 s3api_object_handlers_put.go:1081 unsupported checksum algorithm in X-Amz-Sdk-Checksum-Algorithm: "MD5" === RUN TestDetectRequestedChecksumAlgorithm/no_checksum --- PASS: TestDetectRequestedChecksumAlgorithm (0.00s) --- PASS: TestDetectRequestedChecksumAlgorithm/sdk_algorithm_header (0.00s) --- PASS: TestDetectRequestedChecksumAlgorithm/algorithm_header (0.00s) --- PASS: TestDetectRequestedChecksumAlgorithm/presigned_url_hoists_sdk_algorithm_to_query (0.00s) --- PASS: TestDetectRequestedChecksumAlgorithm/presigned_url_hoists_checksum-algorithm_to_query_(lowercase) (0.00s) --- PASS: TestDetectRequestedChecksumAlgorithm/presigned_url_hoists_individual_checksum_value (0.00s) --- PASS: TestDetectRequestedChecksumAlgorithm/unsupported_in_query_returns_error (0.00s) --- PASS: TestDetectRequestedChecksumAlgorithm/no_checksum (0.00s) === RUN TestLookupHeaderOrQueryCaseInsensitive --- PASS: TestLookupHeaderOrQueryCaseInsensitive (0.00s) === RUN TestUploadChunkSizeFollowsFilerMaxMB === RUN TestUploadChunkSizeFollowsFilerMaxMB/filer_maxMB_unset === RUN TestUploadChunkSizeFollowsFilerMaxMB/filer_maxMB_negative === RUN TestUploadChunkSizeFollowsFilerMaxMB/filer_default === RUN TestUploadChunkSizeFollowsFilerMaxMB/filer_maxMB_32 === RUN TestUploadChunkSizeFollowsFilerMaxMB/clamped_to_int32_range --- PASS: TestUploadChunkSizeFollowsFilerMaxMB (0.00s) --- PASS: TestUploadChunkSizeFollowsFilerMaxMB/filer_maxMB_unset (0.00s) --- PASS: TestUploadChunkSizeFollowsFilerMaxMB/filer_maxMB_negative (0.00s) --- PASS: TestUploadChunkSizeFollowsFilerMaxMB/filer_default (0.00s) --- PASS: TestUploadChunkSizeFollowsFilerMaxMB/filer_maxMB_32 (0.00s) --- PASS: TestUploadChunkSizeFollowsFilerMaxMB/clamped_to_int32_range (0.00s) === RUN TestMapChunkedUploadErrorToS3Error === RUN TestMapChunkedUploadErrorToS3Error/truncated_source_read_maps_to_IncompleteBody === RUN TestMapChunkedUploadErrorToS3Error/volume_upload_unexpected_EOF_maps_to_InternalError === RUN TestMapChunkedUploadErrorToS3Error/payload_checksum_mismatch_maps_to_InvalidDigest === RUN TestMapChunkedUploadErrorToS3Error/other_errors_map_to_InternalError --- PASS: TestMapChunkedUploadErrorToS3Error (0.00s) --- PASS: TestMapChunkedUploadErrorToS3Error/truncated_source_read_maps_to_IncompleteBody (0.00s) --- PASS: TestMapChunkedUploadErrorToS3Error/volume_upload_unexpected_EOF_maps_to_InternalError (0.00s) --- PASS: TestMapChunkedUploadErrorToS3Error/payload_checksum_mismatch_maps_to_InvalidDigest (0.00s) --- PASS: TestMapChunkedUploadErrorToS3Error/other_errors_map_to_InternalError (0.00s) === RUN TestMapChunkedUploadErrorToS3ErrorClientDisconnect === RUN TestMapChunkedUploadErrorToS3ErrorClientDisconnect/peer_gone_maps_to_ClientDisconnected === RUN TestMapChunkedUploadErrorToS3ErrorClientDisconnect/peer_still_connected_stays_IncompleteBody === RUN TestMapChunkedUploadErrorToS3ErrorClientDisconnect/expired_deadline_stays_IncompleteBody === RUN TestMapChunkedUploadErrorToS3ErrorClientDisconnect/server_fault_under_a_canceled_context_stays_InternalError --- PASS: TestMapChunkedUploadErrorToS3ErrorClientDisconnect (0.00s) --- PASS: TestMapChunkedUploadErrorToS3ErrorClientDisconnect/peer_gone_maps_to_ClientDisconnected (0.00s) --- PASS: TestMapChunkedUploadErrorToS3ErrorClientDisconnect/peer_still_connected_stays_IncompleteBody (0.00s) --- PASS: TestMapChunkedUploadErrorToS3ErrorClientDisconnect/expired_deadline_stays_IncompleteBody (0.00s) --- PASS: TestMapChunkedUploadErrorToS3ErrorClientDisconnect/server_fault_under_a_canceled_context_stays_InternalError (0.00s) === RUN TestClientDisconnectedAPIError --- PASS: TestClientDisconnectedAPIError (0.00s) === RUN TestRenameSourceCandidates === RUN TestRenameSourceCandidates/bare_key === RUN TestRenameSourceCandidates/bare_key_with_leading_slash === RUN TestRenameSourceCandidates/bare_key_with_prefix === RUN TestRenameSourceCandidates/bucket_qualified === RUN TestRenameSourceCandidates/bucket_qualified_without_leading_slash === RUN TestRenameSourceCandidates/key_whose_first_segment_is_another_bucket === RUN TestRenameSourceCandidates/percent_encoded === RUN TestRenameSourceCandidates/plus_stays_literal === RUN TestRenameSourceCandidates/duplicate_slashes_collapse === RUN TestRenameSourceCandidates/bucket_name_alone_is_a_key === RUN TestRenameSourceCandidates/bucket_prefix_with_empty_key === RUN TestRenameSourceCandidates/missing_header === RUN TestRenameSourceCandidates/parent_traversal === RUN TestRenameSourceCandidates/encoded_parent_traversal === RUN TestRenameSourceCandidates/parent_traversal_behind_the_bucket --- PASS: TestRenameSourceCandidates (0.01s) --- PASS: TestRenameSourceCandidates/bare_key (0.00s) --- PASS: TestRenameSourceCandidates/bare_key_with_leading_slash (0.00s) --- PASS: TestRenameSourceCandidates/bare_key_with_prefix (0.00s) --- PASS: TestRenameSourceCandidates/bucket_qualified (0.00s) --- PASS: TestRenameSourceCandidates/bucket_qualified_without_leading_slash (0.00s) --- PASS: TestRenameSourceCandidates/key_whose_first_segment_is_another_bucket (0.00s) --- PASS: TestRenameSourceCandidates/percent_encoded (0.00s) --- PASS: TestRenameSourceCandidates/plus_stays_literal (0.00s) --- PASS: TestRenameSourceCandidates/duplicate_slashes_collapse (0.00s) --- PASS: TestRenameSourceCandidates/bucket_name_alone_is_a_key (0.00s) --- PASS: TestRenameSourceCandidates/bucket_prefix_with_empty_key (0.00s) --- PASS: TestRenameSourceCandidates/missing_header (0.00s) --- PASS: TestRenameSourceCandidates/parent_traversal (0.00s) --- PASS: TestRenameSourceCandidates/encoded_parent_traversal (0.00s) --- PASS: TestRenameSourceCandidates/parent_traversal_behind_the_bucket (0.00s) === RUN TestRenameSourceConditionalHeaders === RUN TestRenameSourceConditionalHeaders/if-match_hit === RUN TestRenameSourceConditionalHeaders/if-match_miss === RUN TestRenameSourceConditionalHeaders/if-match_star === RUN TestRenameSourceConditionalHeaders/if-none-match_miss === RUN TestRenameSourceConditionalHeaders/if-none-match_hit === RUN TestRenameSourceConditionalHeaders/if-none-match_star === RUN TestRenameSourceConditionalHeaders/modified_since_older === RUN TestRenameSourceConditionalHeaders/modified_since_newer === RUN TestRenameSourceConditionalHeaders/unmodified_since_newer === RUN TestRenameSourceConditionalHeaders/unmodified_since_older === RUN TestRenameSourceConditionalHeaders/unparsable_date === RUN TestRenameSourceConditionalHeaders/no_headers --- PASS: TestRenameSourceConditionalHeaders (0.02s) --- PASS: TestRenameSourceConditionalHeaders/if-match_hit (0.01s) --- PASS: TestRenameSourceConditionalHeaders/if-match_miss (0.00s) --- PASS: TestRenameSourceConditionalHeaders/if-match_star (0.00s) --- PASS: TestRenameSourceConditionalHeaders/if-none-match_miss (0.00s) --- PASS: TestRenameSourceConditionalHeaders/if-none-match_hit (0.00s) --- PASS: TestRenameSourceConditionalHeaders/if-none-match_star (0.00s) --- PASS: TestRenameSourceConditionalHeaders/modified_since_older (0.00s) --- PASS: TestRenameSourceConditionalHeaders/modified_since_newer (0.00s) --- PASS: TestRenameSourceConditionalHeaders/unmodified_since_newer (0.00s) --- PASS: TestRenameSourceConditionalHeaders/unmodified_since_older (0.00s) --- PASS: TestRenameSourceConditionalHeaders/unparsable_date (0.00s) --- PASS: TestRenameSourceConditionalHeaders/no_headers (0.00s) === RUN TestSourceConditionalHeaderPrecedence === RUN TestSourceConditionalHeaderPrecedence/X-Amz-Copy-Source-If-Match === RUN TestSourceConditionalHeaderPrecedence/X-Amz-Copy-Source-If-Match/matched_if-match_outranks_a_failing_if-unmodified-since === RUN TestSourceConditionalHeaderPrecedence/X-Amz-Copy-Source-If-Match/passed_if-none-match_outranks_a_failing_if-modified-since === RUN TestSourceConditionalHeaderPrecedence/X-Amz-Copy-Source-If-Match/a_failing_if-match_still_loses === RUN TestSourceConditionalHeaderPrecedence/X-Amz-Rename-Source-If-Match === RUN TestSourceConditionalHeaderPrecedence/X-Amz-Rename-Source-If-Match/matched_if-match_outranks_a_failing_if-unmodified-since === RUN TestSourceConditionalHeaderPrecedence/X-Amz-Rename-Source-If-Match/passed_if-none-match_outranks_a_failing_if-modified-since === RUN TestSourceConditionalHeaderPrecedence/X-Amz-Rename-Source-If-Match/a_failing_if-match_still_loses --- PASS: TestSourceConditionalHeaderPrecedence (0.00s) --- PASS: TestSourceConditionalHeaderPrecedence/X-Amz-Copy-Source-If-Match (0.00s) --- PASS: TestSourceConditionalHeaderPrecedence/X-Amz-Copy-Source-If-Match/matched_if-match_outranks_a_failing_if-unmodified-since (0.00s) --- PASS: TestSourceConditionalHeaderPrecedence/X-Amz-Copy-Source-If-Match/passed_if-none-match_outranks_a_failing_if-modified-since (0.00s) --- PASS: TestSourceConditionalHeaderPrecedence/X-Amz-Copy-Source-If-Match/a_failing_if-match_still_loses (0.00s) --- PASS: TestSourceConditionalHeaderPrecedence/X-Amz-Rename-Source-If-Match (0.00s) --- PASS: TestSourceConditionalHeaderPrecedence/X-Amz-Rename-Source-If-Match/matched_if-match_outranks_a_failing_if-unmodified-since (0.00s) --- PASS: TestSourceConditionalHeaderPrecedence/X-Amz-Rename-Source-If-Match/passed_if-none-match_outranks_a_failing_if-modified-since (0.00s) --- PASS: TestSourceConditionalHeaderPrecedence/X-Amz-Rename-Source-If-Match/a_failing_if-match_still_loses (0.00s) === RUN TestRouting_RenameObject I0824 22:18:55.202831 config_loader.go:73 Using explicit credential store: memory --- PASS: TestRouting_RenameObject (0.04s) === RUN TestNewLifecycleTTLResolver_NilOnEmpty --- PASS: TestNewLifecycleTTLResolver_NilOnEmpty (0.00s) === RUN TestNewLifecycleTTLResolver_NilOnVersionedBucket --- PASS: TestNewLifecycleTTLResolver_NilOnVersionedBucket (0.00s) === RUN TestNewLifecycleTTLResolver_DropsTagFilteredRules --- PASS: TestNewLifecycleTTLResolver_DropsTagFilteredRules (0.00s) === RUN TestNewLifecycleTTLResolver_DropsDisabledAndNonExpirationDays --- PASS: TestNewLifecycleTTLResolver_DropsDisabledAndNonExpirationDays (0.00s) === RUN TestResolve_PrefixMatch --- PASS: TestResolve_PrefixMatch (0.00s) === RUN TestResolve_OverlappingRulesShorterExpirationWins --- PASS: TestResolve_OverlappingRulesShorterExpirationWins (0.00s) === RUN TestResolve_OverflowDefersToWorker --- PASS: TestResolve_OverflowDefersToWorker (0.00s) === RUN TestResolve_OverflowSkipsButShorterStillFires --- PASS: TestResolve_OverflowSkipsButShorterStillFires (0.00s) === RUN TestResolve_SizeFilter --- PASS: TestResolve_SizeFilter (0.00s) === RUN TestResolve_NilReceiverReturnsZero --- PASS: TestResolve_NilReceiverReturnsZero (0.00s) === RUN TestNewBucketConfigFromEntry_RefreshesLifecycleTTL --- PASS: TestNewBucketConfigFromEntry_RefreshesLifecycleTTL (0.01s) === RUN TestNewBucketConfigFromEntry_ObjectLockTreatedAsVersioned --- PASS: TestNewBucketConfigFromEntry_ObjectLockTreatedAsVersioned (0.00s) === RUN TestNewBucketConfigFromEntry_TtlFastPathOptIn --- PASS: TestNewBucketConfigFromEntry_TtlFastPathOptIn (0.00s) === RUN TestVeeamObjectLockBugFix === RUN TestVeeamObjectLockBugFix/Bug_case:_bucket_with_no_extended_attributes === RUN TestVeeamObjectLockBugFix/Fix_verification:_bucket_with_Object_Lock_enabled_via_boolean_flag === RUN TestVeeamObjectLockBugFix/Fix_verification:_bucket_with_Object_Lock_enabled_via_Enabled_constant --- PASS: TestVeeamObjectLockBugFix (0.00s) --- PASS: TestVeeamObjectLockBugFix/Bug_case:_bucket_with_no_extended_attributes (0.00s) --- PASS: TestVeeamObjectLockBugFix/Fix_verification:_bucket_with_Object_Lock_enabled_via_boolean_flag (0.00s) --- PASS: TestVeeamObjectLockBugFix/Fix_verification:_bucket_with_Object_Lock_enabled_via_Enabled_constant (0.00s) === RUN TestExtractObjectLockMetadataFromRequest === RUN TestExtractObjectLockMetadataFromRequest/Extract_COMPLIANCE_mode_and_retention_date === RUN TestExtractObjectLockMetadataFromRequest/Extract_GOVERNANCE_mode_and_retention_date === RUN TestExtractObjectLockMetadataFromRequest/Extract_legal_hold_ON === RUN TestExtractObjectLockMetadataFromRequest/Extract_legal_hold_OFF === RUN TestExtractObjectLockMetadataFromRequest/Handle_all_object_lock_headers_together === RUN TestExtractObjectLockMetadataFromRequest/Handle_no_object_lock_headers === RUN TestExtractObjectLockMetadataFromRequest/Handle_invalid_retention_date_-_should_return_error E0824 22:18:55.276232 s3api_object_handlers_put.go:1690 extractObjectLockMetadataFromRequest: failed to parse retention until date, expected format: 2006-01-02T15:04:05Z07:00, error: parsing time "invalid-date" as "2006-01-02T15:04:05Z07:00": cannot parse "invalid-date" as "2006" === RUN TestExtractObjectLockMetadataFromRequest/Handle_invalid_legal_hold_value_-_should_return_error E0824 22:18:55.278007 s3api_object_handlers_put.go:1704 extractObjectLockMetadataFromRequest: unexpected legal hold value provided, expected 'ON' or 'OFF' --- PASS: TestExtractObjectLockMetadataFromRequest (0.01s) --- PASS: TestExtractObjectLockMetadataFromRequest/Extract_COMPLIANCE_mode_and_retention_date (0.00s) --- PASS: TestExtractObjectLockMetadataFromRequest/Extract_GOVERNANCE_mode_and_retention_date (0.00s) --- PASS: TestExtractObjectLockMetadataFromRequest/Extract_legal_hold_ON (0.00s) --- PASS: TestExtractObjectLockMetadataFromRequest/Extract_legal_hold_OFF (0.00s) --- PASS: TestExtractObjectLockMetadataFromRequest/Handle_all_object_lock_headers_together (0.00s) --- PASS: TestExtractObjectLockMetadataFromRequest/Handle_no_object_lock_headers (0.00s) --- PASS: TestExtractObjectLockMetadataFromRequest/Handle_invalid_retention_date_-_should_return_error (0.00s) --- PASS: TestExtractObjectLockMetadataFromRequest/Handle_invalid_legal_hold_value_-_should_return_error (0.00s) === RUN TestAddObjectLockHeadersToResponse === RUN TestAddObjectLockHeadersToResponse/Add_COMPLIANCE_mode_and_retention_date_to_response === RUN TestAddObjectLockHeadersToResponse/Add_GOVERNANCE_mode_to_response === RUN TestAddObjectLockHeadersToResponse/Add_legal_hold_ON_to_response === RUN TestAddObjectLockHeadersToResponse/Add_legal_hold_OFF_to_response === RUN TestAddObjectLockHeadersToResponse/Add_all_object_lock_headers_to_response === RUN TestAddObjectLockHeadersToResponse/Handle_entry_with_no_object_lock_metadata === RUN TestAddObjectLockHeadersToResponse/Handle_entry_with_object_lock_mode_but_no_legal_hold_-_should_default_to_OFF === RUN TestAddObjectLockHeadersToResponse/Handle_entry_with_retention_date_but_no_legal_hold_-_should_default_to_OFF === RUN TestAddObjectLockHeadersToResponse/Handle_nil_entry_gracefully === RUN TestAddObjectLockHeadersToResponse/Handle_entry_with_nil_Extended_map_gracefully === RUN TestAddObjectLockHeadersToResponse/Handle_invalid_retention_timestamp_gracefully E0824 22:18:55.284177 s3api_object_handlers.go:2665 addObjectLockHeadersToResponse: failed to parse retention until date from stored metadata (dateStr: invalid-timestamp): strconv.ParseInt: parsing "invalid-timestamp": invalid syntax --- PASS: TestAddObjectLockHeadersToResponse (0.01s) --- PASS: TestAddObjectLockHeadersToResponse/Add_COMPLIANCE_mode_and_retention_date_to_response (0.00s) --- PASS: TestAddObjectLockHeadersToResponse/Add_GOVERNANCE_mode_to_response (0.00s) --- PASS: TestAddObjectLockHeadersToResponse/Add_legal_hold_ON_to_response (0.00s) --- PASS: TestAddObjectLockHeadersToResponse/Add_legal_hold_OFF_to_response (0.00s) --- PASS: TestAddObjectLockHeadersToResponse/Add_all_object_lock_headers_to_response (0.00s) --- PASS: TestAddObjectLockHeadersToResponse/Handle_entry_with_no_object_lock_metadata (0.00s) --- PASS: TestAddObjectLockHeadersToResponse/Handle_entry_with_object_lock_mode_but_no_legal_hold_-_should_default_to_OFF (0.00s) --- PASS: TestAddObjectLockHeadersToResponse/Handle_entry_with_retention_date_but_no_legal_hold_-_should_default_to_OFF (0.00s) --- PASS: TestAddObjectLockHeadersToResponse/Handle_nil_entry_gracefully (0.00s) --- PASS: TestAddObjectLockHeadersToResponse/Handle_entry_with_nil_Extended_map_gracefully (0.00s) --- PASS: TestAddObjectLockHeadersToResponse/Handle_invalid_retention_timestamp_gracefully (0.00s) === RUN TestObjectLockHeaderRoundTrip === RUN TestObjectLockHeaderRoundTrip/Complete_round_trip_for_COMPLIANCE_mode === RUN TestObjectLockHeaderRoundTrip/Complete_round_trip_for_GOVERNANCE_mode --- PASS: TestObjectLockHeaderRoundTrip (0.00s) --- PASS: TestObjectLockHeaderRoundTrip/Complete_round_trip_for_COMPLIANCE_mode (0.00s) --- PASS: TestObjectLockHeaderRoundTrip/Complete_round_trip_for_GOVERNANCE_mode (0.00s) === RUN TestValidateObjectLockHeaders === RUN TestValidateObjectLockHeaders/Valid_COMPLIANCE_mode_with_retention_date_on_versioned_bucket === RUN TestValidateObjectLockHeaders/Valid_GOVERNANCE_mode_with_retention_date_on_versioned_bucket === RUN TestValidateObjectLockHeaders/Valid_legal_hold_ON_on_versioned_bucket === RUN TestValidateObjectLockHeaders/Valid_legal_hold_OFF_on_versioned_bucket === RUN TestValidateObjectLockHeaders/Invalid_object_lock_mode === RUN TestValidateObjectLockHeaders/Invalid_legal_hold_status === RUN TestValidateObjectLockHeaders/Object_lock_headers_on_non-versioned_bucket === RUN TestValidateObjectLockHeaders/Invalid_retention_date_format === RUN TestValidateObjectLockHeaders/Retention_date_in_the_past === RUN TestValidateObjectLockHeaders/Mode_without_retention_date === RUN TestValidateObjectLockHeaders/Retention_date_without_mode === RUN TestValidateObjectLockHeaders/Governance_bypass_header_on_non-versioned_bucket === RUN TestValidateObjectLockHeaders/Governance_bypass_header_on_versioned_bucket_should_pass === RUN TestValidateObjectLockHeaders/No_object_lock_headers_should_pass === RUN TestValidateObjectLockHeaders/Mixed_valid_headers_should_pass --- PASS: TestValidateObjectLockHeaders (0.01s) --- PASS: TestValidateObjectLockHeaders/Valid_COMPLIANCE_mode_with_retention_date_on_versioned_bucket (0.00s) --- PASS: TestValidateObjectLockHeaders/Valid_GOVERNANCE_mode_with_retention_date_on_versioned_bucket (0.00s) --- PASS: TestValidateObjectLockHeaders/Valid_legal_hold_ON_on_versioned_bucket (0.00s) --- PASS: TestValidateObjectLockHeaders/Valid_legal_hold_OFF_on_versioned_bucket (0.00s) --- PASS: TestValidateObjectLockHeaders/Invalid_object_lock_mode (0.00s) --- PASS: TestValidateObjectLockHeaders/Invalid_legal_hold_status (0.00s) --- PASS: TestValidateObjectLockHeaders/Object_lock_headers_on_non-versioned_bucket (0.00s) --- PASS: TestValidateObjectLockHeaders/Invalid_retention_date_format (0.00s) --- PASS: TestValidateObjectLockHeaders/Retention_date_in_the_past (0.00s) --- PASS: TestValidateObjectLockHeaders/Mode_without_retention_date (0.00s) --- PASS: TestValidateObjectLockHeaders/Retention_date_without_mode (0.00s) --- PASS: TestValidateObjectLockHeaders/Governance_bypass_header_on_non-versioned_bucket (0.00s) --- PASS: TestValidateObjectLockHeaders/Governance_bypass_header_on_versioned_bucket_should_pass (0.00s) --- PASS: TestValidateObjectLockHeaders/No_object_lock_headers_should_pass (0.00s) --- PASS: TestValidateObjectLockHeaders/Mixed_valid_headers_should_pass (0.00s) === RUN TestMapValidationErrorToS3Error === RUN TestMapValidationErrorToS3Error/ErrObjectLockVersioningRequired === RUN TestMapValidationErrorToS3Error/ErrInvalidObjectLockMode === RUN TestMapValidationErrorToS3Error/ErrInvalidLegalHoldStatus === RUN TestMapValidationErrorToS3Error/ErrInvalidRetentionDateFormat === RUN TestMapValidationErrorToS3Error/ErrRetentionDateMustBeFuture === RUN TestMapValidationErrorToS3Error/ErrObjectLockModeRequiresDate === RUN TestMapValidationErrorToS3Error/ErrRetentionDateRequiresMode === RUN TestMapValidationErrorToS3Error/ErrGovernanceBypassVersioningRequired === RUN TestMapValidationErrorToS3Error/Unknown_error_defaults_to_ErrInvalidRequest --- PASS: TestMapValidationErrorToS3Error (0.01s) --- PASS: TestMapValidationErrorToS3Error/ErrObjectLockVersioningRequired (0.00s) --- PASS: TestMapValidationErrorToS3Error/ErrInvalidObjectLockMode (0.00s) --- PASS: TestMapValidationErrorToS3Error/ErrInvalidLegalHoldStatus (0.00s) --- PASS: TestMapValidationErrorToS3Error/ErrInvalidRetentionDateFormat (0.00s) --- PASS: TestMapValidationErrorToS3Error/ErrRetentionDateMustBeFuture (0.00s) --- PASS: TestMapValidationErrorToS3Error/ErrObjectLockModeRequiresDate (0.00s) --- PASS: TestMapValidationErrorToS3Error/ErrRetentionDateRequiresMode (0.00s) --- PASS: TestMapValidationErrorToS3Error/ErrGovernanceBypassVersioningRequired (0.00s) --- PASS: TestMapValidationErrorToS3Error/Unknown_error_defaults_to_ErrInvalidRequest (0.00s) === RUN TestObjectLockPermissionLogic === RUN TestObjectLockPermissionLogic/Non-versioned_bucket_PUT_operation_logic s3api_object_lock_headers_test.go:626: For non-versioned buckets: s3api_object_lock_headers_test.go:627: - PUT operations overwrite existing objects s3api_object_lock_headers_test.go:628: - Must check existing object lock protections before allowing overwrite s3api_object_lock_headers_test.go:629: - Governance bypass headers can be used to override GOVERNANCE mode retention s3api_object_lock_headers_test.go:630: - COMPLIANCE mode retention and legal holds cannot be bypassed === RUN TestObjectLockPermissionLogic/Versioned_bucket_PUT_operation_logic s3api_object_lock_headers_test.go:643: For versioned buckets: s3api_object_lock_headers_test.go:644: - PUT operations create new versions without overwriting existing objects s3api_object_lock_headers_test.go:645: - No need to check existing object lock protections s3api_object_lock_headers_test.go:646: - Only validate object lock headers for the new version being created s3api_object_lock_headers_test.go:647: - Each version has independent object lock settings === RUN TestObjectLockPermissionLogic/Governance_bypass_header_validation s3api_object_lock_headers_test.go:656: Governance bypass behavior: s3api_object_lock_headers_test.go:657: - Only valid on versioned buckets (header validation) s3api_object_lock_headers_test.go:658: - For non-versioned buckets: Allows overwriting objects under GOVERNANCE retention s3api_object_lock_headers_test.go:659: - For versioned buckets: Not typically needed for PUT operations s3api_object_lock_headers_test.go:660: - Must have s3:BypassGovernanceRetention permission --- PASS: TestObjectLockPermissionLogic (0.00s) --- PASS: TestObjectLockPermissionLogic/Non-versioned_bucket_PUT_operation_logic (0.00s) --- PASS: TestObjectLockPermissionLogic/Versioned_bucket_PUT_operation_logic (0.00s) --- PASS: TestObjectLockPermissionLogic/Governance_bypass_header_validation (0.00s) === RUN TestSetObjectOwnerFromRequest === RUN TestSetObjectOwnerFromRequest/BucketOwnerEnforced_-_use_bucket_owner === RUN TestSetObjectOwnerFromRequest/ObjectWriter_-_use_uploader === RUN TestSetObjectOwnerFromRequest/BucketOwnerPreferred_-_use_uploader === RUN TestSetObjectOwnerFromRequest/BucketOwnerEnforced_but_owner_is_nil_-_fallback_to_uploader === RUN TestSetObjectOwnerFromRequest/BucketOwnerEnforced_but_owner_ID_is_nil_-_fallback_to_uploader === RUN TestSetObjectOwnerFromRequest/Bucket_metadata_error_-_fallback_to_uploader === RUN TestSetObjectOwnerFromRequest/Bucket_registry_is_nil_-_fallback_to_uploader === RUN TestSetObjectOwnerFromRequest/Empty_uploader_account_ID_-_no_owner_set --- PASS: TestSetObjectOwnerFromRequest (0.01s) --- PASS: TestSetObjectOwnerFromRequest/BucketOwnerEnforced_-_use_bucket_owner (0.00s) --- PASS: TestSetObjectOwnerFromRequest/ObjectWriter_-_use_uploader (0.00s) --- PASS: TestSetObjectOwnerFromRequest/BucketOwnerPreferred_-_use_uploader (0.00s) --- PASS: TestSetObjectOwnerFromRequest/BucketOwnerEnforced_but_owner_is_nil_-_fallback_to_uploader (0.00s) --- PASS: TestSetObjectOwnerFromRequest/BucketOwnerEnforced_but_owner_ID_is_nil_-_fallback_to_uploader (0.00s) --- PASS: TestSetObjectOwnerFromRequest/Bucket_metadata_error_-_fallback_to_uploader (0.00s) --- PASS: TestSetObjectOwnerFromRequest/Bucket_registry_is_nil_-_fallback_to_uploader (0.00s) --- PASS: TestSetObjectOwnerFromRequest/Empty_uploader_account_ID_-_no_owner_set (0.00s) === RUN TestParseAndValidateRangeRejectsMalformedNumericOffsets === RUN TestParseAndValidateRangeRejectsMalformedNumericOffsets/invalid_start === RUN TestParseAndValidateRangeRejectsMalformedNumericOffsets/invalid_end === RUN TestParseAndValidateRangeRejectsMalformedNumericOffsets/invalid_open_ended_start --- PASS: TestParseAndValidateRangeRejectsMalformedNumericOffsets (0.01s) --- PASS: TestParseAndValidateRangeRejectsMalformedNumericOffsets/invalid_start (0.00s) --- PASS: TestParseAndValidateRangeRejectsMalformedNumericOffsets/invalid_end (0.00s) --- PASS: TestParseAndValidateRangeRejectsMalformedNumericOffsets/invalid_open_ended_start (0.00s) === RUN TestValidateRetention === RUN TestValidateRetention/Valid_GOVERNANCE_retention === RUN TestValidateRetention/Valid_COMPLIANCE_retention === RUN TestValidateRetention/Missing_Mode === RUN TestValidateRetention/Missing_RetainUntilDate === RUN TestValidateRetention/Invalid_Mode === RUN TestValidateRetention/Past_RetainUntilDate === RUN TestValidateRetention/Empty_retention --- PASS: TestValidateRetention (0.01s) --- PASS: TestValidateRetention/Valid_GOVERNANCE_retention (0.00s) --- PASS: TestValidateRetention/Valid_COMPLIANCE_retention (0.00s) --- PASS: TestValidateRetention/Missing_Mode (0.00s) --- PASS: TestValidateRetention/Missing_RetainUntilDate (0.00s) --- PASS: TestValidateRetention/Invalid_Mode (0.00s) --- PASS: TestValidateRetention/Past_RetainUntilDate (0.00s) --- PASS: TestValidateRetention/Empty_retention (0.00s) === RUN TestValidateLegalHold === RUN TestValidateLegalHold/Valid_ON_status === RUN TestValidateLegalHold/Valid_OFF_status === RUN TestValidateLegalHold/Invalid_status === RUN TestValidateLegalHold/Empty_status === RUN TestValidateLegalHold/Lowercase_on === RUN TestValidateLegalHold/Lowercase_off --- PASS: TestValidateLegalHold (0.00s) --- PASS: TestValidateLegalHold/Valid_ON_status (0.00s) --- PASS: TestValidateLegalHold/Valid_OFF_status (0.00s) --- PASS: TestValidateLegalHold/Invalid_status (0.00s) --- PASS: TestValidateLegalHold/Empty_status (0.00s) --- PASS: TestValidateLegalHold/Lowercase_on (0.00s) --- PASS: TestValidateLegalHold/Lowercase_off (0.00s) === RUN TestParseObjectRetention === RUN TestParseObjectRetention/Valid_retention_XML === RUN TestParseObjectRetention/Valid_compliance_retention_XML === RUN TestParseObjectRetention/Valid_retention_XML_without_namespace_(Veeam_compatibility) === RUN TestParseObjectRetention/Valid_compliance_retention_XML_without_namespace_(Veeam_compatibility) === RUN TestParseObjectRetention/Empty_XML_body === RUN TestParseObjectRetention/Invalid_XML === RUN TestParseObjectRetention/Malformed_XML === RUN TestParseObjectRetention/Missing_Mode === RUN TestParseObjectRetention/Missing_RetainUntilDate --- PASS: TestParseObjectRetention (0.01s) --- PASS: TestParseObjectRetention/Valid_retention_XML (0.00s) --- PASS: TestParseObjectRetention/Valid_compliance_retention_XML (0.00s) --- PASS: TestParseObjectRetention/Valid_retention_XML_without_namespace_(Veeam_compatibility) (0.00s) --- PASS: TestParseObjectRetention/Valid_compliance_retention_XML_without_namespace_(Veeam_compatibility) (0.00s) --- PASS: TestParseObjectRetention/Empty_XML_body (0.00s) --- PASS: TestParseObjectRetention/Invalid_XML (0.00s) --- PASS: TestParseObjectRetention/Malformed_XML (0.00s) --- PASS: TestParseObjectRetention/Missing_Mode (0.00s) --- PASS: TestParseObjectRetention/Missing_RetainUntilDate (0.00s) === RUN TestParseObjectLegalHold === RUN TestParseObjectLegalHold/Valid_legal_hold_ON === RUN TestParseObjectLegalHold/Valid_legal_hold_OFF === RUN TestParseObjectLegalHold/Valid_legal_hold_ON_without_namespace === RUN TestParseObjectLegalHold/Valid_legal_hold_OFF_without_namespace === RUN TestParseObjectLegalHold/Empty_XML_body === RUN TestParseObjectLegalHold/Invalid_XML === RUN TestParseObjectLegalHold/Missing_Status --- PASS: TestParseObjectLegalHold (0.01s) --- PASS: TestParseObjectLegalHold/Valid_legal_hold_ON (0.00s) --- PASS: TestParseObjectLegalHold/Valid_legal_hold_OFF (0.00s) --- PASS: TestParseObjectLegalHold/Valid_legal_hold_ON_without_namespace (0.00s) --- PASS: TestParseObjectLegalHold/Valid_legal_hold_OFF_without_namespace (0.00s) --- PASS: TestParseObjectLegalHold/Empty_XML_body (0.00s) --- PASS: TestParseObjectLegalHold/Invalid_XML (0.00s) --- PASS: TestParseObjectLegalHold/Missing_Status (0.00s) === RUN TestParseObjectLockConfiguration === RUN TestParseObjectLockConfiguration/Valid_object_lock_configuration === RUN TestParseObjectLockConfiguration/Valid_object_lock_configuration_with_rule === RUN TestParseObjectLockConfiguration/Valid_object_lock_configuration_without_namespace === RUN TestParseObjectLockConfiguration/Valid_object_lock_configuration_with_rule_without_namespace === RUN TestParseObjectLockConfiguration/Empty_XML_body === RUN TestParseObjectLockConfiguration/Invalid_XML --- PASS: TestParseObjectLockConfiguration (0.01s) --- PASS: TestParseObjectLockConfiguration/Valid_object_lock_configuration (0.00s) --- PASS: TestParseObjectLockConfiguration/Valid_object_lock_configuration_with_rule (0.00s) --- PASS: TestParseObjectLockConfiguration/Valid_object_lock_configuration_without_namespace (0.00s) --- PASS: TestParseObjectLockConfiguration/Valid_object_lock_configuration_with_rule_without_namespace (0.00s) --- PASS: TestParseObjectLockConfiguration/Empty_XML_body (0.00s) --- PASS: TestParseObjectLockConfiguration/Invalid_XML (0.00s) === RUN TestValidateObjectLockConfiguration === RUN TestValidateObjectLockConfiguration/Valid_config_with_ObjectLockEnabled_only === RUN TestValidateObjectLockConfiguration/Missing_ObjectLockEnabled === RUN TestValidateObjectLockConfiguration/Valid_config_with_rule_and_days === RUN TestValidateObjectLockConfiguration/Valid_config_with_rule_and_years === RUN TestValidateObjectLockConfiguration/Invalid_ObjectLockEnabled_value === RUN TestValidateObjectLockConfiguration/Invalid_rule_-_missing_mode === RUN TestValidateObjectLockConfiguration/Invalid_rule_-_both_days_and_years === RUN TestValidateObjectLockConfiguration/Invalid_rule_-_neither_days_nor_years === RUN TestValidateObjectLockConfiguration/Invalid_rule_-_invalid_mode === RUN TestValidateObjectLockConfiguration/Invalid_rule_-_days_out_of_range === RUN TestValidateObjectLockConfiguration/Invalid_rule_-_years_out_of_range === RUN TestValidateObjectLockConfiguration/Invalid_rule_-_missing_DefaultRetention --- PASS: TestValidateObjectLockConfiguration (0.01s) --- PASS: TestValidateObjectLockConfiguration/Valid_config_with_ObjectLockEnabled_only (0.00s) --- PASS: TestValidateObjectLockConfiguration/Missing_ObjectLockEnabled (0.00s) --- PASS: TestValidateObjectLockConfiguration/Valid_config_with_rule_and_days (0.00s) --- PASS: TestValidateObjectLockConfiguration/Valid_config_with_rule_and_years (0.00s) --- PASS: TestValidateObjectLockConfiguration/Invalid_ObjectLockEnabled_value (0.00s) --- PASS: TestValidateObjectLockConfiguration/Invalid_rule_-_missing_mode (0.00s) --- PASS: TestValidateObjectLockConfiguration/Invalid_rule_-_both_days_and_years (0.00s) --- PASS: TestValidateObjectLockConfiguration/Invalid_rule_-_neither_days_nor_years (0.00s) --- PASS: TestValidateObjectLockConfiguration/Invalid_rule_-_invalid_mode (0.00s) --- PASS: TestValidateObjectLockConfiguration/Invalid_rule_-_days_out_of_range (0.00s) --- PASS: TestValidateObjectLockConfiguration/Invalid_rule_-_years_out_of_range (0.00s) --- PASS: TestValidateObjectLockConfiguration/Invalid_rule_-_missing_DefaultRetention (0.00s) === RUN TestValidateDefaultRetention === RUN TestValidateDefaultRetention/Valid_retention_with_days === RUN TestValidateDefaultRetention/Valid_retention_with_years === RUN TestValidateDefaultRetention/Missing_mode === RUN TestValidateDefaultRetention/Invalid_mode === RUN TestValidateDefaultRetention/Both_days_and_years_specified === RUN TestValidateDefaultRetention/Neither_days_nor_years_specified === RUN TestValidateDefaultRetention/Zero_years --- PASS: TestValidateDefaultRetention (0.00s) --- PASS: TestValidateDefaultRetention/Valid_retention_with_days (0.00s) --- PASS: TestValidateDefaultRetention/Valid_retention_with_years (0.00s) --- PASS: TestValidateDefaultRetention/Missing_mode (0.00s) --- PASS: TestValidateDefaultRetention/Invalid_mode (0.00s) --- PASS: TestValidateDefaultRetention/Both_days_and_years_specified (0.00s) --- PASS: TestValidateDefaultRetention/Neither_days_nor_years_specified (0.00s) --- PASS: TestValidateDefaultRetention/Zero_years (0.00s) === RUN TestStoreObjectLockConfigurationClearsStaleYears --- PASS: TestStoreObjectLockConfigurationClearsStaleYears (0.00s) === RUN TestOwnerRecentlyUnreachable --- PASS: TestOwnerRecentlyUnreachable (0.00s) === RUN TestBuildWriteCondition === RUN TestBuildWriteCondition/no_headers_is_unconditional === RUN TestBuildWriteCondition/If-None-Match_*_to_IF_NOT_EXISTS === RUN TestBuildWriteCondition/If-Match_*_to_IF_EXISTS === RUN TestBuildWriteCondition/If-Match_strong_etag_to_IF_ETAG_MATCH === RUN TestBuildWriteCondition/If-None-Match_strong_etag_to_IF_ETAG_NOT_MATCH === RUN TestBuildWriteCondition/weak_etag_falls_back === RUN TestBuildWriteCondition/etag_list_falls_back === RUN TestBuildWriteCondition/both_match_and_none-match_falls_back === RUN TestBuildWriteCondition/time-based_falls_back --- PASS: TestBuildWriteCondition (0.01s) --- PASS: TestBuildWriteCondition/no_headers_is_unconditional (0.00s) --- PASS: TestBuildWriteCondition/If-None-Match_*_to_IF_NOT_EXISTS (0.00s) --- PASS: TestBuildWriteCondition/If-Match_*_to_IF_EXISTS (0.00s) --- PASS: TestBuildWriteCondition/If-Match_strong_etag_to_IF_ETAG_MATCH (0.00s) --- PASS: TestBuildWriteCondition/If-None-Match_strong_etag_to_IF_ETAG_NOT_MATCH (0.00s) --- PASS: TestBuildWriteCondition/weak_etag_falls_back (0.00s) --- PASS: TestBuildWriteCondition/etag_list_falls_back (0.00s) --- PASS: TestBuildWriteCondition/both_match_and_none-match_falls_back (0.00s) --- PASS: TestBuildWriteCondition/time-based_falls_back (0.00s) === RUN TestParseConditionalHeadersAcceptsHTTPDateFormats === RUN TestParseConditionalHeadersAcceptsHTTPDateFormats/If-Modified-Since_RFC850 === RUN TestParseConditionalHeadersAcceptsHTTPDateFormats/If-Unmodified-Since_ANSIC === RUN TestParseConditionalHeadersAcceptsHTTPDateFormats/If-Modified-Since_RFC1123_UTC_zone --- PASS: TestParseConditionalHeadersAcceptsHTTPDateFormats (0.00s) --- PASS: TestParseConditionalHeadersAcceptsHTTPDateFormats/If-Modified-Since_RFC850 (0.00s) --- PASS: TestParseConditionalHeadersAcceptsHTTPDateFormats/If-Unmodified-Since_ANSIC (0.00s) --- PASS: TestParseConditionalHeadersAcceptsHTTPDateFormats/If-Modified-Since_RFC1123_UTC_zone (0.00s) === RUN TestValidateConditionalCopyHeadersAcceptsHTTPDateFormats === RUN TestValidateConditionalCopyHeadersAcceptsHTTPDateFormats/X-Amz-Copy-Source-If-Modified-Since_RFC850 === RUN TestValidateConditionalCopyHeadersAcceptsHTTPDateFormats/X-Amz-Copy-Source-If-Unmodified-Since_ANSIC --- PASS: TestValidateConditionalCopyHeadersAcceptsHTTPDateFormats (0.00s) --- PASS: TestValidateConditionalCopyHeadersAcceptsHTTPDateFormats/X-Amz-Copy-Source-If-Modified-Since_RFC850 (0.00s) --- PASS: TestValidateConditionalCopyHeadersAcceptsHTTPDateFormats/X-Amz-Copy-Source-If-Unmodified-Since_ANSIC (0.00s) === RUN TestBuildDeleteCondition === RUN TestBuildDeleteCondition/no_If-Match_is_unconditional === RUN TestBuildDeleteCondition/If-Match_*_to_IF_EXISTS === RUN TestBuildDeleteCondition/If-Match_etag_to_IF_ETAG_MATCH === RUN TestBuildDeleteCondition/weak_etag_falls_back --- PASS: TestBuildDeleteCondition (0.00s) --- PASS: TestBuildDeleteCondition/no_If-Match_is_unconditional (0.00s) --- PASS: TestBuildDeleteCondition/If-Match_*_to_IF_EXISTS (0.00s) --- PASS: TestBuildDeleteCondition/If-Match_etag_to_IF_ETAG_MATCH (0.00s) --- PASS: TestBuildDeleteCondition/weak_etag_falls_back (0.00s) === RUN TestSingleStrongETag --- PASS: TestSingleStrongETag (0.00s) === RUN TestObjectTxnFailsOverStaleOwner --- PASS: TestObjectTxnFailsOverStaleOwner (0.04s) === RUN TestRouteWriteCondition --- PASS: TestRouteWriteCondition (0.00s) === RUN TestSelectLatestVersion_MixedFormats --- PASS: TestSelectLatestVersion_MixedFormats (0.00s) === RUN TestSelectLatestVersion_PromotesNewestDeleteMarker --- PASS: TestSelectLatestVersion_PromotesNewestDeleteMarker (0.00s) === RUN TestSelectLatestVersion_ContentWinsWhenNewer --- PASS: TestSelectLatestVersion_ContentWinsWhenNewer (0.00s) === RUN TestSelectLatestVersion_OnlyDeleteMarkers --- PASS: TestSelectLatestVersion_OnlyDeleteMarkers (0.00s) === RUN TestSelectLatestVersion_EmptyOrUntagged --- PASS: TestSelectLatestVersion_EmptyOrUntagged (0.00s) === RUN TestVersionIdFromEntry --- PASS: TestVersionIdFromEntry (0.00s) === RUN TestSelectLatestVersion_FilenameFallback --- PASS: TestSelectLatestVersion_FilenameFallback (0.00s) === RUN TestSelectLatestVersion_FilenameFallbackMixedWithTagged --- PASS: TestSelectLatestVersion_FilenameFallbackMixedWithTagged (0.00s) === RUN TestSelectLatestVersion_FilenameFallbackDeleteMarker --- PASS: TestSelectLatestVersion_FilenameFallbackDeleteMarker (0.00s) === RUN TestPartBoundaryJsonCompatibility --- PASS: TestPartBoundaryJsonCompatibility (0.00s) === RUN TestBuildObjectAttributesPartsPrefersOffsets --- PASS: TestBuildObjectAttributesPartsPrefersOffsets (0.00s) === RUN TestBuildObjectAttributesPartsLegacyChunkIndexes --- PASS: TestBuildObjectAttributesPartsLegacyChunkIndexes (0.00s) === RUN TestValidateRequestPath_RejectsTraversal === RUN TestValidateRequestPath_RejectsTraversal/clean_path_passes === RUN TestValidateRequestPath_RejectsTraversal/bucket_only_passes === RUN TestValidateRequestPath_RejectsTraversal/trailing_slash_passes === RUN TestValidateRequestPath_RejectsTraversal/leading_dotdot_rejected === RUN TestValidateRequestPath_RejectsTraversal/nested_dotdot_rejected === RUN TestValidateRequestPath_RejectsTraversal/backslash_dotdot_rejected === RUN TestValidateRequestPath_RejectsTraversal/percent-encoded_dotdot_rejected === RUN TestValidateRequestPath_RejectsTraversal/bare_dot_object_rejected === RUN TestValidateRequestPath_RejectsTraversal/dotdot_bucket_rejected --- PASS: TestValidateRequestPath_RejectsTraversal (0.02s) --- PASS: TestValidateRequestPath_RejectsTraversal/clean_path_passes (0.00s) --- PASS: TestValidateRequestPath_RejectsTraversal/bucket_only_passes (0.00s) --- PASS: TestValidateRequestPath_RejectsTraversal/trailing_slash_passes (0.00s) --- PASS: TestValidateRequestPath_RejectsTraversal/leading_dotdot_rejected (0.00s) --- PASS: TestValidateRequestPath_RejectsTraversal/nested_dotdot_rejected (0.00s) --- PASS: TestValidateRequestPath_RejectsTraversal/backslash_dotdot_rejected (0.00s) --- PASS: TestValidateRequestPath_RejectsTraversal/percent-encoded_dotdot_rejected (0.00s) --- PASS: TestValidateRequestPath_RejectsTraversal/bare_dot_object_rejected (0.00s) --- PASS: TestValidateRequestPath_RejectsTraversal/dotdot_bucket_rejected (0.00s) === RUN TestValidateRequestPath_RejectsEmptyCapturedVars === RUN TestValidateRequestPath_RejectsEmptyCapturedVars/empty_bucket === RUN TestValidateRequestPath_RejectsEmptyCapturedVars/empty_object --- PASS: TestValidateRequestPath_RejectsEmptyCapturedVars (0.00s) --- PASS: TestValidateRequestPath_RejectsEmptyCapturedVars/empty_bucket (0.00s) --- PASS: TestValidateRequestPath_RejectsEmptyCapturedVars/empty_object (0.00s) === RUN TestValidateRequestPath_RejectsUnsafePathQueryValues === RUN TestValidateRequestPath_RejectsUnsafePathQueryValues/clean_version_ID === RUN TestValidateRequestPath_RejectsUnsafePathQueryValues/clean_upload_ID === RUN TestValidateRequestPath_RejectsUnsafePathQueryValues/empty_values === RUN TestValidateRequestPath_RejectsUnsafePathQueryValues/version_ID_encoded_slash === RUN TestValidateRequestPath_RejectsUnsafePathQueryValues/version_ID_backslash === RUN TestValidateRequestPath_RejectsUnsafePathQueryValues/upload_ID_traversal === RUN TestValidateRequestPath_RejectsUnsafePathQueryValues/unsafe_repeated_value === RUN TestValidateRequestPath_RejectsUnsafePathQueryValues/encoded_version_ID_name === RUN TestValidateRequestPath_RejectsUnsafePathQueryValues/encoded_upload_ID_name === RUN TestValidateRequestPath_RejectsUnsafePathQueryValues/unrelated_query --- PASS: TestValidateRequestPath_RejectsUnsafePathQueryValues (0.01s) --- PASS: TestValidateRequestPath_RejectsUnsafePathQueryValues/clean_version_ID (0.00s) --- PASS: TestValidateRequestPath_RejectsUnsafePathQueryValues/clean_upload_ID (0.00s) --- PASS: TestValidateRequestPath_RejectsUnsafePathQueryValues/empty_values (0.00s) --- PASS: TestValidateRequestPath_RejectsUnsafePathQueryValues/version_ID_encoded_slash (0.00s) --- PASS: TestValidateRequestPath_RejectsUnsafePathQueryValues/version_ID_backslash (0.00s) --- PASS: TestValidateRequestPath_RejectsUnsafePathQueryValues/upload_ID_traversal (0.00s) --- PASS: TestValidateRequestPath_RejectsUnsafePathQueryValues/unsafe_repeated_value (0.00s) --- PASS: TestValidateRequestPath_RejectsUnsafePathQueryValues/encoded_version_ID_name (0.00s) --- PASS: TestValidateRequestPath_RejectsUnsafePathQueryValues/encoded_upload_ID_name (0.00s) --- PASS: TestValidateRequestPath_RejectsUnsafePathQueryValues/unrelated_query (0.00s) === RUN TestHasPathSegmentQuery_CommonPathDoesNotAllocate --- PASS: TestHasPathSegmentQuery_CommonPathDoesNotAllocate (0.04s) === RUN TestGetRequestDataReader_ChunkedEncodingWithoutIAM I0824 22:18:55.549522 config_loader.go:73 Using explicit credential store: memory === RUN TestGetRequestDataReader_ChunkedEncodingWithoutIAM/RegularRequest s3api_put_object_helper_test.go:82: Test case: RegularRequest - Regular requests without chunked encoding should pass through unchanged === RUN TestGetRequestDataReader_ChunkedEncodingWithoutIAM/StreamingSignedWithoutIAM s3api_put_object_helper_test.go:82: Test case: StreamingSignedWithoutIAM - Streaming signed requests should fail when IAM is disabled === RUN TestGetRequestDataReader_ChunkedEncodingWithoutIAM/StreamingUnsignedWithoutIAM s3api_put_object_helper_test.go:82: Test case: StreamingUnsignedWithoutIAM - Streaming unsigned requests should be processed even when IAM is disabled --- PASS: TestGetRequestDataReader_ChunkedEncodingWithoutIAM (0.00s) --- PASS: TestGetRequestDataReader_ChunkedEncodingWithoutIAM/RegularRequest (0.00s) --- PASS: TestGetRequestDataReader_ChunkedEncodingWithoutIAM/StreamingSignedWithoutIAM (0.00s) --- PASS: TestGetRequestDataReader_ChunkedEncodingWithoutIAM/StreamingUnsignedWithoutIAM (0.00s) === RUN TestGetRequestDataReader_AuthTypeDetection I0824 22:18:55.551409 config_loader.go:73 Using explicit credential store: memory === RUN TestGetRequestDataReader_AuthTypeDetection/ChunkedDataWithChecksum --- PASS: TestGetRequestDataReader_AuthTypeDetection (0.00s) --- PASS: TestGetRequestDataReader_AuthTypeDetection/ChunkedDataWithChecksum (0.00s) === RUN TestGetRequestDataReader_IAMEnabled I0824 22:18:55.551862 config_loader.go:73 Using explicit credential store: memory === RUN TestGetRequestDataReader_IAMEnabled/StreamingUnsignedWithIAMEnabled --- PASS: TestGetRequestDataReader_IAMEnabled (0.00s) --- PASS: TestGetRequestDataReader_IAMEnabled/StreamingUnsignedWithIAMEnabled (0.00s) === RUN TestAuthTypeDetection === RUN TestAuthTypeDetection/StreamingUnsigned === RUN TestAuthTypeDetection/StreamingSigned === RUN TestAuthTypeDetection/Regular --- PASS: TestAuthTypeDetection (0.00s) --- PASS: TestAuthTypeDetection/StreamingUnsigned (0.00s) --- PASS: TestAuthTypeDetection/StreamingSigned (0.00s) --- PASS: TestAuthTypeDetection/Regular (0.00s) === RUN TestIsInRemoteOnly === RUN TestIsInRemoteOnly/remote-only_entry_with_no_chunks === RUN TestIsInRemoteOnly/remote_entry_with_chunks_(cached) === RUN TestIsInRemoteOnly/local_file_with_chunks_(not_remote) === RUN TestIsInRemoteOnly/empty_remote_entry_(size_0) === RUN TestIsInRemoteOnly/no_chunks_but_nil_RemoteEntry --- PASS: TestIsInRemoteOnly (0.00s) --- PASS: TestIsInRemoteOnly/remote-only_entry_with_no_chunks (0.00s) --- PASS: TestIsInRemoteOnly/remote_entry_with_chunks_(cached) (0.00s) --- PASS: TestIsInRemoteOnly/local_file_with_chunks_(not_remote) (0.00s) --- PASS: TestIsInRemoteOnly/empty_remote_entry_(size_0) (0.00s) --- PASS: TestIsInRemoteOnly/no_chunks_but_nil_RemoteEntry (0.00s) === RUN TestRemoteOnlyEntryDetection === RUN TestRemoteOnlyEntryDetection/remote-only_entry_(no_chunks,_has_remote_entry) === RUN TestRemoteOnlyEntryDetection/data_integrity_error_(no_chunks,_no_remote,_has_size) === RUN TestRemoteOnlyEntryDetection/empty_local_file_(no_chunks,_no_remote,_size_0) === RUN TestRemoteOnlyEntryDetection/normal_file_with_chunks --- PASS: TestRemoteOnlyEntryDetection (0.00s) --- PASS: TestRemoteOnlyEntryDetection/remote-only_entry_(no_chunks,_has_remote_entry) (0.00s) --- PASS: TestRemoteOnlyEntryDetection/data_integrity_error_(no_chunks,_no_remote,_has_size) (0.00s) --- PASS: TestRemoteOnlyEntryDetection/empty_local_file_(no_chunks,_no_remote,_size_0) (0.00s) --- PASS: TestRemoteOnlyEntryDetection/normal_file_with_chunks (0.00s) === RUN TestVersionedRemoteObjectPathBuilding === RUN TestVersionedRemoteObjectPathBuilding/non-versioned_object_(empty_versionId) === RUN TestVersionedRemoteObjectPathBuilding/null_version === RUN TestVersionedRemoteObjectPathBuilding/specific_version === RUN TestVersionedRemoteObjectPathBuilding/nested_object_with_version === RUN TestVersionedRemoteObjectPathBuilding/object_with_leading_slash_and_version --- PASS: TestVersionedRemoteObjectPathBuilding (0.00s) --- PASS: TestVersionedRemoteObjectPathBuilding/non-versioned_object_(empty_versionId) (0.00s) --- PASS: TestVersionedRemoteObjectPathBuilding/null_version (0.00s) --- PASS: TestVersionedRemoteObjectPathBuilding/specific_version (0.00s) --- PASS: TestVersionedRemoteObjectPathBuilding/nested_object_with_version (0.00s) --- PASS: TestVersionedRemoteObjectPathBuilding/object_with_leading_slash_and_version (0.00s) === RUN TestResolvedSourceVersionId === RUN TestResolvedSourceVersionId/explicit_request_versionId_wins === RUN TestResolvedSourceVersionId/empty_request_falls_back_to_entry_version_(latest_in_versioned_bucket) === RUN TestResolvedSourceVersionId/empty_request_and_pre-versioning_entry_stays_empty === RUN TestResolvedSourceVersionId/empty_request_and_nil_Extended_stays_empty === RUN TestResolvedSourceVersionId/nil_entry_tolerated_when_no_request_version --- PASS: TestResolvedSourceVersionId (0.00s) --- PASS: TestResolvedSourceVersionId/explicit_request_versionId_wins (0.00s) --- PASS: TestResolvedSourceVersionId/empty_request_falls_back_to_entry_version_(latest_in_versioned_bucket) (0.00s) --- PASS: TestResolvedSourceVersionId/empty_request_and_pre-versioning_entry_stays_empty (0.00s) --- PASS: TestResolvedSourceVersionId/empty_request_and_nil_Extended_stays_empty (0.00s) --- PASS: TestResolvedSourceVersionId/nil_entry_tolerated_when_no_request_version (0.00s) === RUN TestCachedEntryHasLocalData === RUN TestCachedEntryHasLocalData/nil_entry_is_not_a_hit === RUN TestCachedEntryHasLocalData/entry_with_chunks_is_a_hit === RUN TestCachedEntryHasLocalData/entry_with_inline_content_is_a_hit === RUN TestCachedEntryHasLocalData/entry_with_neither_chunks_nor_content_is_not_a_hit --- PASS: TestCachedEntryHasLocalData (0.00s) --- PASS: TestCachedEntryHasLocalData/nil_entry_is_not_a_hit (0.00s) --- PASS: TestCachedEntryHasLocalData/entry_with_chunks_is_a_hit (0.00s) --- PASS: TestCachedEntryHasLocalData/entry_with_inline_content_is_a_hit (0.00s) --- PASS: TestCachedEntryHasLocalData/entry_with_neither_chunks_nor_content_is_not_a_hit (0.00s) === RUN TestCopyObjectRemoteOnlySourceDetection === RUN TestCopyObjectRemoteOnlySourceDetection/remote-only_object_with_size_and_no_chunks/content === RUN TestCopyObjectRemoteOnlySourceDetection/local_file_with_chunks_-_copy_works_fine,_fix_does_not_engage === RUN TestCopyObjectRemoteOnlySourceDetection/small_inline_file_(no_chunks,_has_Content)_-_hits_inline_branch_but_not_broken === RUN TestCopyObjectRemoteOnlySourceDetection/remote_entry_already_cached_(has_chunks)_-_fix_does_not_engage === RUN TestCopyObjectRemoteOnlySourceDetection/zero-byte_remote_object_-_fix_does_not_engage,_inline_branch_handles_it --- PASS: TestCopyObjectRemoteOnlySourceDetection (0.00s) --- PASS: TestCopyObjectRemoteOnlySourceDetection/remote-only_object_with_size_and_no_chunks/content (0.00s) --- PASS: TestCopyObjectRemoteOnlySourceDetection/local_file_with_chunks_-_copy_works_fine,_fix_does_not_engage (0.00s) --- PASS: TestCopyObjectRemoteOnlySourceDetection/small_inline_file_(no_chunks,_has_Content)_-_hits_inline_branch_but_not_broken (0.00s) --- PASS: TestCopyObjectRemoteOnlySourceDetection/remote_entry_already_cached_(has_chunks)_-_fix_does_not_engage (0.00s) --- PASS: TestCopyObjectRemoteOnlySourceDetection/zero-byte_remote_object_-_fix_does_not_engage,_inline_branch_handles_it (0.00s) === RUN TestCacheRemoteObjectForStreamingTimeout --- PASS: TestCacheRemoteObjectForStreamingTimeout (0.21s) === RUN TestCacheRemoteObjectForStreamingCached --- PASS: TestCacheRemoteObjectForStreamingCached (0.02s) === RUN TestS3ColdReadStreamsFromOrigin === RUN TestS3ColdReadStreamsFromOrigin/full_read === RUN TestS3ColdReadStreamsFromOrigin/range_read === RUN TestS3ColdReadStreamsFromOrigin/short_origin_read_fails_instead_of_silently_truncating === RUN TestS3ColdReadStreamsFromOrigin/version-specific_read_keeps_503_retry === RUN TestS3ColdReadStreamsFromOrigin/latest_read_of_a_versioned_entry_keeps_503_retry === RUN TestS3ColdReadStreamsFromOrigin/local_cache_failure_falls_back_to_origin === RUN TestS3ColdReadStreamsFromOrigin/entry_not_found_stays_404 === RUN TestS3ColdReadStreamsFromOrigin/entry_not_found_stays_404/canonical_status E0824 22:18:56.063381 s3api_object_handlers.go:1029 streamFromVolumeServers: entry not found while caching mybucket/dir/obj.bin: rpc error: code = NotFound desc = entry vanished === RUN TestS3ColdReadStreamsFromOrigin/entry_not_found_stays_404/raw_sentinel E0824 22:18:56.084992 s3api_object_handlers.go:1029 streamFromVolumeServers: entry not found while caching mybucket/dir/obj.bin: rpc error: code = Unknown desc = filer: no entry is found in filer store --- PASS: TestS3ColdReadStreamsFromOrigin (0.29s) --- PASS: TestS3ColdReadStreamsFromOrigin/full_read (0.05s) --- PASS: TestS3ColdReadStreamsFromOrigin/range_read (0.05s) --- PASS: TestS3ColdReadStreamsFromOrigin/short_origin_read_fails_instead_of_silently_truncating (0.05s) --- PASS: TestS3ColdReadStreamsFromOrigin/version-specific_read_keeps_503_retry (0.02s) --- PASS: TestS3ColdReadStreamsFromOrigin/latest_read_of_a_versioned_entry_keeps_503_retry (0.03s) --- PASS: TestS3ColdReadStreamsFromOrigin/local_cache_failure_falls_back_to_origin (0.05s) --- PASS: TestS3ColdReadStreamsFromOrigin/entry_not_found_stays_404 (0.04s) --- PASS: TestS3ColdReadStreamsFromOrigin/entry_not_found_stays_404/canonical_status (0.02s) --- PASS: TestS3ColdReadStreamsFromOrigin/entry_not_found_stays_404/raw_sentinel (0.02s) === RUN TestAdvertisedHost === RUN TestAdvertisedHost/advertised_ip_wins_over_wildcard_bind === RUN TestAdvertisedHost/advertised_ip_wins_over_specific_bind === RUN TestAdvertisedHost/bind_ip_used_when_no_advertised_ip === RUN TestAdvertisedHost/ipv6_bind_ip_used_when_no_advertised_ip === RUN TestAdvertisedHost/wildcard_bind_falls_back_to_detected === RUN TestAdvertisedHost/ipv6_wildcard_bind_falls_back_to_detected === RUN TestAdvertisedHost/bracketed_ipv6_wildcard_bind_falls_back_to_detected === RUN TestAdvertisedHost/expanded_ipv6_wildcard_bind_falls_back_to_detected === RUN TestAdvertisedHost/wildcard_advertised_ip_falls_back_to_bind_ip === RUN TestAdvertisedHost/wildcard_advertised_ip_and_bind_fall_back_to_detected === RUN TestAdvertisedHost/empty_falls_back_to_detected --- PASS: TestAdvertisedHost (0.08s) --- PASS: TestAdvertisedHost/advertised_ip_wins_over_wildcard_bind (0.00s) --- PASS: TestAdvertisedHost/advertised_ip_wins_over_specific_bind (0.00s) --- PASS: TestAdvertisedHost/bind_ip_used_when_no_advertised_ip (0.00s) --- PASS: TestAdvertisedHost/ipv6_bind_ip_used_when_no_advertised_ip (0.00s) --- PASS: TestAdvertisedHost/wildcard_bind_falls_back_to_detected (0.01s) --- PASS: TestAdvertisedHost/ipv6_wildcard_bind_falls_back_to_detected (0.01s) --- PASS: TestAdvertisedHost/bracketed_ipv6_wildcard_bind_falls_back_to_detected (0.01s) --- PASS: TestAdvertisedHost/expanded_ipv6_wildcard_bind_falls_back_to_detected (0.02s) --- PASS: TestAdvertisedHost/wildcard_advertised_ip_falls_back_to_bind_ip (0.00s) --- PASS: TestAdvertisedHost/wildcard_advertised_ip_and_bind_fall_back_to_detected (0.01s) --- PASS: TestAdvertisedHost/empty_falls_back_to_detected (0.01s) === RUN TestRouting_STSWithQueryParams I0824 22:18:56.175029 config_loader.go:73 Using explicit credential store: memory --- PASS: TestRouting_STSWithQueryParams (0.04s) === RUN TestRouting_STSWithBodyParams I0824 22:18:56.211795 config_loader.go:73 Using explicit credential store: memory --- PASS: TestRouting_STSWithBodyParams (0.03s) === RUN TestRouting_GetFederationTokenWithQueryParams I0824 22:18:56.239925 config_loader.go:73 Using explicit credential store: memory --- PASS: TestRouting_GetFederationTokenWithQueryParams (0.03s) === RUN TestRouting_GetFederationTokenAuthenticatedBody I0824 22:18:56.269334 config_loader.go:73 Using explicit credential store: memory --- PASS: TestRouting_GetFederationTokenAuthenticatedBody (0.03s) === RUN TestRouting_AuthenticatedIAM I0824 22:18:56.297996 config_loader.go:73 Using explicit credential store: memory --- PASS: TestRouting_AuthenticatedIAM (0.03s) === RUN TestRouting_IAMMatcherLogic === RUN TestRouting_IAMMatcherLogic/No_auth_-_anonymous I0824 22:18:56.325048 config_loader.go:73 Using explicit credential store: memory === RUN TestRouting_IAMMatcherLogic/AWS4_signature I0824 22:18:56.348749 config_loader.go:73 Using explicit credential store: memory === RUN TestRouting_IAMMatcherLogic/AWS2_signature I0824 22:18:56.374972 config_loader.go:73 Using explicit credential store: memory W0824 22:18:56.402779 auth_signature_v2.go:137 InvalidAccessKeyId (V2 signed): attempted key 'AKIA...' not found. Available keys: 1, Auth enabled: true === RUN TestRouting_IAMMatcherLogic/Presigned_V4 I0824 22:18:56.405555 config_loader.go:73 Using explicit credential store: memory === RUN TestRouting_IAMMatcherLogic/Presigned_V2 I0824 22:18:56.432266 config_loader.go:73 Using explicit credential store: memory === RUN TestRouting_IAMMatcherLogic/AWS4_signature_with_STS_action_in_body I0824 22:18:56.460123 config_loader.go:73 Using explicit credential store: memory --- PASS: TestRouting_IAMMatcherLogic (0.16s) --- PASS: TestRouting_IAMMatcherLogic/No_auth_-_anonymous (0.02s) --- PASS: TestRouting_IAMMatcherLogic/AWS4_signature (0.02s) --- PASS: TestRouting_IAMMatcherLogic/AWS2_signature (0.03s) --- PASS: TestRouting_IAMMatcherLogic/Presigned_V4 (0.03s) --- PASS: TestRouting_IAMMatcherLogic/Presigned_V2 (0.03s) --- PASS: TestRouting_IAMMatcherLogic/AWS4_signature_with_STS_action_in_body (0.03s) === RUN TestRouting_BucketRouteMatchesAnyHost === RUN TestRouting_BucketRouteMatchesAnyHost/virtual-host_domain_configured,_request_via_IP I0824 22:18:56.486565 config_loader.go:73 Using explicit credential store: memory === RUN TestRouting_BucketRouteMatchesAnyHost/virtual-host_domain_configured,_request_via_unrelated_host I0824 22:18:56.521988 config_loader.go:73 Using explicit credential store: memory === RUN TestRouting_BucketRouteMatchesAnyHost/path-style_+_virtual-host_configured,_request_via_IP I0824 22:18:56.558074 config_loader.go:73 Using explicit credential store: memory === RUN TestRouting_BucketRouteMatchesAnyHost/domain_configured,_request_via_the_bare_configured_host_(no_bucket_subdomain) I0824 22:18:56.608510 config_loader.go:73 Using explicit credential store: memory --- PASS: TestRouting_BucketRouteMatchesAnyHost (0.16s) --- PASS: TestRouting_BucketRouteMatchesAnyHost/virtual-host_domain_configured,_request_via_IP (0.03s) --- PASS: TestRouting_BucketRouteMatchesAnyHost/virtual-host_domain_configured,_request_via_unrelated_host (0.04s) --- PASS: TestRouting_BucketRouteMatchesAnyHost/path-style_+_virtual-host_configured,_request_via_IP (0.05s) --- PASS: TestRouting_BucketRouteMatchesAnyHost/domain_configured,_request_via_the_bare_configured_host_(no_bucket_subdomain) (0.03s) === RUN TestSSECDecryptChunkView_NoOffsetAdjustment s3api_sse_decrypt_test.go:63: ✓ Correct decryption (using stored IV directly) successful s3api_sse_decrypt_test.go:91: ✓ Verified: Offset-adjusted IV produces corrupted data (as expected for SSE-C) s3api_sse_decrypt_test.go:96: Plaintext: "This is a test messa" s3api_sse_decrypt_test.go:101: Corrupted: "\xcdѸL<\x1f\x8c\xe7K\xfb\xdd\a\xf5)˰\x1c-\xbe~" --- PASS: TestSSECDecryptChunkView_NoOffsetAdjustment (0.00s) === RUN TestSSEKMSDecryptChunkView_RequiresOffsetAdjustment s3api_sse_decrypt_test.go:161: ✓ SSE-KMS decryption with offset adjustment successful s3api_sse_decrypt_test.go:182: ✓ Verified: double-adjusted IV produces corrupted data (range path must not pre-adjust) s3api_sse_decrypt_test.go:199: ✓ Verified: Base IV without offset produces corrupted data (as expected for SSE-KMS) --- PASS: TestSSEKMSDecryptChunkView_RequiresOffsetAdjustment (0.00s) === RUN TestSSEDecryptionDifferences s3api_sse_decrypt_test.go:205: SSE-C: Random IV per part → Use stored IV DIRECTLY (no offset) s3api_sse_decrypt_test.go:206: SSE-KMS: Base IV + offset → MUST call calculateIVWithOffset(baseIV, offset) s3api_sse_decrypt_test.go:207: SSE-S3: Base IV + offset → Stores ADJUSTED IV, use directly --- PASS: TestSSEDecryptionDifferences (0.00s) === RUN TestSSES3MultipartUploadStoresDerivedIV === RUN TestSSES3MultipartUploadStoresDerivedIV/Part_1_at_offset_0 s3api_sse_s3_upload_test.go:103: ✓ Derived IV decryption successful for offset 0 === RUN TestSSES3MultipartUploadStoresDerivedIV/Part_2_at_offset_1MB s3api_sse_s3_upload_test.go:103: ✓ Derived IV decryption successful for offset 1048576 s3api_sse_s3_upload_test.go:127: ✓ Verified: Base IV produces corrupted data at offset 1048576 (bug would cause this) === RUN TestSSES3MultipartUploadStoresDerivedIV/Part_3_at_offset_5MB s3api_sse_s3_upload_test.go:103: ✓ Derived IV decryption successful for offset 5242880 s3api_sse_s3_upload_test.go:127: ✓ Verified: Base IV produces corrupted data at offset 5242880 (bug would cause this) --- PASS: TestSSES3MultipartUploadStoresDerivedIV (0.00s) --- PASS: TestSSES3MultipartUploadStoresDerivedIV/Part_1_at_offset_0 (0.00s) --- PASS: TestSSES3MultipartUploadStoresDerivedIV/Part_2_at_offset_1MB (0.00s) --- PASS: TestSSES3MultipartUploadStoresDerivedIV/Part_3_at_offset_5MB (0.00s) === RUN TestHandleSSES3MultipartEncryptionFlow s3api_sse_s3_upload_test.go:203: ✓ Key.IV correctly updated with derived IV s3api_sse_s3_upload_test.go:225: ✓ Full encrypt-update_key-decrypt cycle successful --- PASS: TestHandleSSES3MultipartEncryptionFlow (0.00s) === RUN TestSSES3HeaderEncoding --- PASS: TestSSES3HeaderEncoding (0.00s) === RUN TestShouldWriteStreamingErrorResponse === RUN TestShouldWriteStreamingErrorResponse/nil_error === RUN TestShouldWriteStreamingErrorResponse/context_canceled === RUN TestShouldWriteStreamingErrorResponse/wrapped_context_canceled === RUN TestShouldWriteStreamingErrorResponse/grpc_canceled === RUN TestShouldWriteStreamingErrorResponse/wrapped_grpc_canceled === RUN TestShouldWriteStreamingErrorResponse/deadline_exceeded === RUN TestShouldWriteStreamingErrorResponse/wrapped_deadline_exceeded --- PASS: TestShouldWriteStreamingErrorResponse (0.01s) --- PASS: TestShouldWriteStreamingErrorResponse/nil_error (0.00s) --- PASS: TestShouldWriteStreamingErrorResponse/context_canceled (0.00s) --- PASS: TestShouldWriteStreamingErrorResponse/wrapped_context_canceled (0.00s) --- PASS: TestShouldWriteStreamingErrorResponse/grpc_canceled (0.00s) --- PASS: TestShouldWriteStreamingErrorResponse/wrapped_grpc_canceled (0.01s) --- PASS: TestShouldWriteStreamingErrorResponse/deadline_exceeded (0.00s) --- PASS: TestShouldWriteStreamingErrorResponse/wrapped_deadline_exceeded (0.00s) === RUN TestAssumeRole_AdminCallerScopedToRolePolicies --- PASS: TestAssumeRole_AdminCallerScopedToRolePolicies (0.02s) === RUN TestAssumeRole_SelfAssumptionWithoutPrincipalArn --- PASS: TestAssumeRole_SelfAssumptionWithoutPrincipalArn (0.01s) === RUN TestAssumeRole_RejectsNonRoleArn === RUN TestAssumeRole_RejectsNonRoleArn/arn:aws:iam:::user/test-user === RUN TestAssumeRole_RejectsNonRoleArn/arn:aws:iam::123456789012:user/test-user === RUN TestAssumeRole_RejectsNonRoleArn/arn:aws:iam::123456789012:group/test-team === RUN TestAssumeRole_RejectsNonRoleArn/test-user --- PASS: TestAssumeRole_RejectsNonRoleArn (0.01s) --- PASS: TestAssumeRole_RejectsNonRoleArn/arn:aws:iam:::user/test-user (0.00s) --- PASS: TestAssumeRole_RejectsNonRoleArn/arn:aws:iam::123456789012:user/test-user (0.00s) --- PASS: TestAssumeRole_RejectsNonRoleArn/arn:aws:iam::123456789012:group/test-team (0.00s) --- PASS: TestAssumeRole_RejectsNonRoleArn/test-user (0.00s) === RUN TestAssumeRole_NonAdminCallerAuthorizedByTrustPolicy === RUN TestAssumeRole_NonAdminCallerAuthorizedByTrustPolicy/trust_policy_admits_the_caller === RUN TestAssumeRole_NonAdminCallerAuthorizedByTrustPolicy/trust_policy_admits_a_specific_principal === RUN TestAssumeRole_NonAdminCallerAuthorizedByTrustPolicy/trust_policy_rejects_the_caller === RUN TestAssumeRole_NonAdminCallerAuthorizedByTrustPolicy/identity_policy_explicit_deny_wins_over_trust_policy === RUN TestAssumeRole_NonAdminCallerAuthorizedByTrustPolicy/session_policy_explicit_deny_blocks_role_chaining === RUN TestAssumeRole_NonAdminCallerAuthorizedByTrustPolicy/revoked_chained_session_cannot_assume W0824 22:18:56.770531 auth_credentials.go:2550 AssumeRole explicit-deny check failed for e2aefa9cdfea88c440aa3206d0092d27, denying: session has been revoked --- PASS: TestAssumeRole_NonAdminCallerAuthorizedByTrustPolicy (0.06s) --- PASS: TestAssumeRole_NonAdminCallerAuthorizedByTrustPolicy/trust_policy_admits_the_caller (0.01s) --- PASS: TestAssumeRole_NonAdminCallerAuthorizedByTrustPolicy/trust_policy_admits_a_specific_principal (0.01s) --- PASS: TestAssumeRole_NonAdminCallerAuthorizedByTrustPolicy/trust_policy_rejects_the_caller (0.00s) --- PASS: TestAssumeRole_NonAdminCallerAuthorizedByTrustPolicy/identity_policy_explicit_deny_wins_over_trust_policy (0.00s) --- PASS: TestAssumeRole_NonAdminCallerAuthorizedByTrustPolicy/session_policy_explicit_deny_blocks_role_chaining (0.02s) --- PASS: TestAssumeRole_NonAdminCallerAuthorizedByTrustPolicy/revoked_chained_session_cannot_assume (0.01s) === RUN TestCallerPrincipalArn --- PASS: TestCallerPrincipalArn (0.00s) === RUN TestAssumeRoleWithWebIdentity_DispatchesThroughIAMManager === RUN TestAssumeRoleWithWebIdentity_DispatchesThroughIAMManager/with_IAMManager_wired === RUN TestAssumeRoleWithWebIdentity_DispatchesThroughIAMManager/without_IAM_integration --- PASS: TestAssumeRoleWithWebIdentity_DispatchesThroughIAMManager (0.00s) --- PASS: TestAssumeRoleWithWebIdentity_DispatchesThroughIAMManager/with_IAMManager_wired (0.00s) --- PASS: TestAssumeRoleWithWebIdentity_DispatchesThroughIAMManager/without_IAM_integration (0.00s) === RUN TestAssumeRoleWithWebIdentity_AllowsEmptyRoleArn --- PASS: TestAssumeRoleWithWebIdentity_AllowsEmptyRoleArn (0.00s) === RUN TestAssumeRole_CallerIdentityFallback === RUN TestAssumeRole_CallerIdentityFallback/Caller_is_IAM_User,_No_RoleArn === RUN TestAssumeRole_CallerIdentityFallback/Caller_is_STS_Assumed_Role,_No_RoleArn === RUN TestAssumeRole_CallerIdentityFallback/Explicit_RoleArn_Provided === RUN TestAssumeRole_CallerIdentityFallback/Malformed_ARN --- PASS: TestAssumeRole_CallerIdentityFallback (0.01s) --- PASS: TestAssumeRole_CallerIdentityFallback/Caller_is_IAM_User,_No_RoleArn (0.00s) --- PASS: TestAssumeRole_CallerIdentityFallback/Caller_is_STS_Assumed_Role,_No_RoleArn (0.00s) --- PASS: TestAssumeRole_CallerIdentityFallback/Explicit_RoleArn_Provided (0.00s) --- PASS: TestAssumeRole_CallerIdentityFallback/Malformed_ARN (0.00s) === RUN TestAssumeRole_EmbedsRolePolicies === RUN TestAssumeRole_EmbedsRolePolicies/RoleWithAttachedPolicies === RUN TestAssumeRole_EmbedsRolePolicies/RoleWithoutAttachedPolicies --- PASS: TestAssumeRole_EmbedsRolePolicies (0.01s) --- PASS: TestAssumeRole_EmbedsRolePolicies/RoleWithAttachedPolicies (0.00s) --- PASS: TestAssumeRole_EmbedsRolePolicies/RoleWithoutAttachedPolicies (0.00s) === RUN TestSTSRoutesEmitAuditEntries I0824 22:18:56.802847 config_loader.go:73 Using explicit credential store: memory === RUN TestSTSRoutesEmitAuditEntries/AssumeRole === RUN TestSTSRoutesEmitAuditEntries/GetCallerIdentity === RUN TestSTSRoutesEmitAuditEntries/GetFederationToken === RUN TestSTSRoutesEmitAuditEntries/AssumeRoleWithWebIdentity === RUN TestSTSRoutesEmitAuditEntries/AssumeRoleWithLDAPIdentity === RUN TestSTSRoutesEmitAuditEntries/AssumeRole_via_POST_body --- PASS: TestSTSRoutesEmitAuditEntries (0.14s) --- PASS: TestSTSRoutesEmitAuditEntries/AssumeRole (0.03s) --- PASS: TestSTSRoutesEmitAuditEntries/GetCallerIdentity (0.00s) --- PASS: TestSTSRoutesEmitAuditEntries/GetFederationToken (0.01s) --- PASS: TestSTSRoutesEmitAuditEntries/AssumeRoleWithWebIdentity (0.00s) --- PASS: TestSTSRoutesEmitAuditEntries/AssumeRoleWithLDAPIdentity (0.00s) --- PASS: TestSTSRoutesEmitAuditEntries/AssumeRole_via_POST_body (0.01s) === RUN TestGetCallerIdentityHandler === RUN TestGetCallerIdentityHandler/static_credentials_report_the_user_ARN === RUN TestGetCallerIdentityHandler/session_credentials_report_the_assumed_role,_not_the_minting_user === RUN TestGetCallerIdentityHandler/bad_signature_is_denied === RUN TestGetCallerIdentityHandler/unsigned_request_is_denied --- PASS: TestGetCallerIdentityHandler (0.03s) --- PASS: TestGetCallerIdentityHandler/static_credentials_report_the_user_ARN (0.01s) --- PASS: TestGetCallerIdentityHandler/session_credentials_report_the_assumed_role,_not_the_minting_user (0.02s) --- PASS: TestGetCallerIdentityHandler/bad_signature_is_denied (0.00s) --- PASS: TestGetCallerIdentityHandler/unsigned_request_is_denied (0.00s) === RUN TestGetCallerIdentityResponse_XMLMarshal --- PASS: TestGetCallerIdentityResponse_XMLMarshal (0.00s) === RUN TestGetFederationToken_BasicFlow --- PASS: TestGetFederationToken_BasicFlow (0.00s) === RUN TestGetFederationToken_WithSessionPolicy --- PASS: TestGetFederationToken_WithSessionPolicy (0.00s) === RUN TestGetFederationToken_RejectTemporaryCredentials === RUN TestGetFederationToken_RejectTemporaryCredentials/SessionTokenInHeader === RUN TestGetFederationToken_RejectTemporaryCredentials/SessionTokenInQuery --- PASS: TestGetFederationToken_RejectTemporaryCredentials (0.00s) --- PASS: TestGetFederationToken_RejectTemporaryCredentials/SessionTokenInHeader (0.00s) --- PASS: TestGetFederationToken_RejectTemporaryCredentials/SessionTokenInQuery (0.00s) === RUN TestGetFederationToken_MissingName --- PASS: TestGetFederationToken_MissingName (0.00s) === RUN TestGetFederationToken_NameValidation === RUN TestGetFederationToken_NameValidation/TooShort === RUN TestGetFederationToken_NameValidation/TooLong === RUN TestGetFederationToken_NameValidation/MinLength === RUN TestGetFederationToken_NameValidation/MaxLength === RUN TestGetFederationToken_NameValidation/ValidSpecialChars === RUN TestGetFederationToken_NameValidation/InvalidChars_Space === RUN TestGetFederationToken_NameValidation/InvalidChars_Slash --- PASS: TestGetFederationToken_NameValidation (0.00s) --- PASS: TestGetFederationToken_NameValidation/TooShort (0.00s) --- PASS: TestGetFederationToken_NameValidation/TooLong (0.00s) --- PASS: TestGetFederationToken_NameValidation/MinLength (0.00s) --- PASS: TestGetFederationToken_NameValidation/MaxLength (0.00s) --- PASS: TestGetFederationToken_NameValidation/ValidSpecialChars (0.00s) --- PASS: TestGetFederationToken_NameValidation/InvalidChars_Space (0.00s) --- PASS: TestGetFederationToken_NameValidation/InvalidChars_Slash (0.00s) === RUN TestGetFederationToken_DurationValidation === RUN TestGetFederationToken_DurationValidation/BelowMinimum === RUN TestGetFederationToken_DurationValidation/AboveMaximum === RUN TestGetFederationToken_DurationValidation/InvalidFormat === RUN TestGetFederationToken_DurationValidation/MinimumValid === RUN TestGetFederationToken_DurationValidation/MaximumValid_36Hours === RUN TestGetFederationToken_DurationValidation/Default12Hours --- PASS: TestGetFederationToken_DurationValidation (0.00s) --- PASS: TestGetFederationToken_DurationValidation/BelowMinimum (0.00s) --- PASS: TestGetFederationToken_DurationValidation/AboveMaximum (0.00s) --- PASS: TestGetFederationToken_DurationValidation/InvalidFormat (0.00s) --- PASS: TestGetFederationToken_DurationValidation/MinimumValid (0.00s) --- PASS: TestGetFederationToken_DurationValidation/MaximumValid_36Hours (0.00s) --- PASS: TestGetFederationToken_DurationValidation/Default12Hours (0.00s) === RUN TestGetFederationToken_ResponseFormat --- PASS: TestGetFederationToken_ResponseFormat (0.00s) === RUN TestGetFederationToken_PolicyEmbedding --- PASS: TestGetFederationToken_PolicyEmbedding (0.00s) === RUN TestGetFederationToken_PolicyIntersection --- PASS: TestGetFederationToken_PolicyIntersection (0.00s) === RUN TestGetFederationToken_MalformedPolicy === RUN TestGetFederationToken_MalformedPolicy/InvalidJSON === RUN TestGetFederationToken_MalformedPolicy/EmptyObject === RUN TestGetFederationToken_MalformedPolicy/TooLarge === RUN TestGetFederationToken_MalformedPolicy/ValidPolicy === RUN TestGetFederationToken_MalformedPolicy/EmptyString --- PASS: TestGetFederationToken_MalformedPolicy (0.00s) --- PASS: TestGetFederationToken_MalformedPolicy/InvalidJSON (0.00s) --- PASS: TestGetFederationToken_MalformedPolicy/EmptyObject (0.00s) --- PASS: TestGetFederationToken_MalformedPolicy/TooLarge (0.00s) --- PASS: TestGetFederationToken_MalformedPolicy/ValidPolicy (0.00s) --- PASS: TestGetFederationToken_MalformedPolicy/EmptyString (0.00s) === RUN TestGetFederationToken_STSNotReady --- PASS: TestGetFederationToken_STSNotReady (0.00s) === RUN TestGetFederationToken_DefaultDuration --- PASS: TestGetFederationToken_DefaultDuration (0.00s) === RUN TestGetFederationToken_GetPoliciesForUser === RUN TestGetFederationToken_GetPoliciesForUser/NoUserStore === RUN TestGetFederationToken_GetPoliciesForUser/UserNotFound === RUN TestGetFederationToken_GetPoliciesForUser/UserWithPolicies === RUN TestGetFederationToken_GetPoliciesForUser/UserWithNoPolicies --- PASS: TestGetFederationToken_GetPoliciesForUser (0.00s) --- PASS: TestGetFederationToken_GetPoliciesForUser/NoUserStore (0.00s) --- PASS: TestGetFederationToken_GetPoliciesForUser/UserNotFound (0.00s) --- PASS: TestGetFederationToken_GetPoliciesForUser/UserWithPolicies (0.00s) --- PASS: TestGetFederationToken_GetPoliciesForUser/UserWithNoPolicies (0.00s) === RUN TestGetFederationToken_PolicyMergeAndDedup --- PASS: TestGetFederationToken_PolicyMergeAndDedup (0.00s) === RUN TestGetFederationToken_PolicyMergeNoManager --- PASS: TestGetFederationToken_PolicyMergeNoManager (0.00s) === RUN TestAssumeRoleWithWebIdentityOverHTTP === RUN TestAssumeRoleWithWebIdentityOverHTTP/succeeds_with_parameters_in_the_query === RUN TestAssumeRoleWithWebIdentityOverHTTP/succeeds_with_parameters_in_the_body === RUN TestAssumeRoleWithWebIdentityOverHTTP/inline_session_policy_is_reported_in_PackedPolicySize === RUN TestAssumeRoleWithWebIdentityOverHTTP/role_whose_trust_policy_rejects_the_provider_is_denied === RUN TestAssumeRoleWithWebIdentityOverHTTP/token_signed_by_a_key_we_do_not_publish_is_rejected === RUN TestAssumeRoleWithWebIdentityOverHTTP/token_naming_an_unknown_key_id_is_rejected --- PASS: TestAssumeRoleWithWebIdentityOverHTTP (2.92s) --- PASS: TestAssumeRoleWithWebIdentityOverHTTP/succeeds_with_parameters_in_the_query (0.03s) --- PASS: TestAssumeRoleWithWebIdentityOverHTTP/succeeds_with_parameters_in_the_body (0.02s) --- PASS: TestAssumeRoleWithWebIdentityOverHTTP/inline_session_policy_is_reported_in_PackedPolicySize (0.02s) --- PASS: TestAssumeRoleWithWebIdentityOverHTTP/role_whose_trust_policy_rejects_the_provider_is_denied (0.01s) --- PASS: TestAssumeRoleWithWebIdentityOverHTTP/token_signed_by_a_key_we_do_not_publish_is_rejected (0.81s) --- PASS: TestAssumeRoleWithWebIdentityOverHTTP/token_naming_an_unknown_key_id_is_rejected (1.56s) === RUN TestMaintenanceRestPathsMatchAWSBindings === RUN TestMaintenanceRestPathsMatchAWSBindings/PutTableBucketMaintenanceConfiguration === RUN TestMaintenanceRestPathsMatchAWSBindings/GetTableBucketMaintenanceConfiguration === RUN TestMaintenanceRestPathsMatchAWSBindings/PutTableMaintenanceConfiguration === RUN TestMaintenanceRestPathsMatchAWSBindings/GetTableMaintenanceConfiguration === RUN TestMaintenanceRestPathsMatchAWSBindings/GetTableMaintenanceJobStatus --- PASS: TestMaintenanceRestPathsMatchAWSBindings (0.01s) --- PASS: TestMaintenanceRestPathsMatchAWSBindings/PutTableBucketMaintenanceConfiguration (0.00s) --- PASS: TestMaintenanceRestPathsMatchAWSBindings/GetTableBucketMaintenanceConfiguration (0.00s) --- PASS: TestMaintenanceRestPathsMatchAWSBindings/PutTableMaintenanceConfiguration (0.00s) --- PASS: TestMaintenanceRestPathsMatchAWSBindings/GetTableMaintenanceConfiguration (0.00s) --- PASS: TestMaintenanceRestPathsMatchAWSBindings/GetTableMaintenanceJobStatus (0.00s) === RUN TestBuildPutTableMaintenanceConfigurationRequest --- PASS: TestBuildPutTableMaintenanceConfigurationRequest (0.01s) === RUN TestBuildPutTableBucketMaintenanceConfigurationRequest --- PASS: TestBuildPutTableBucketMaintenanceConfigurationRequest (0.00s) === RUN TestBuildMaintenanceRequestsRejectBadNamespace --- PASS: TestBuildMaintenanceRequestsRejectBadNamespace (0.00s) === RUN TestBuildListTablesRequestRejectsInvalidNamespaceQuery === RUN TestBuildListTablesRequestRejectsInvalidNamespaceQuery/uppercase === RUN TestBuildListTablesRequestRejectsInvalidNamespaceQuery/leading_hyphen === RUN TestBuildListTablesRequestRejectsInvalidNamespaceQuery/slash --- PASS: TestBuildListTablesRequestRejectsInvalidNamespaceQuery (0.00s) --- PASS: TestBuildListTablesRequestRejectsInvalidNamespaceQuery/uppercase (0.00s) --- PASS: TestBuildListTablesRequestRejectsInvalidNamespaceQuery/leading_hyphen (0.00s) --- PASS: TestBuildListTablesRequestRejectsInvalidNamespaceQuery/slash (0.00s) === RUN TestBuildGetTableRequestRejectsInvalidNamespaceQuery === RUN TestBuildGetTableRequestRejectsInvalidNamespaceQuery/uppercase === RUN TestBuildGetTableRequestRejectsInvalidNamespaceQuery/leading_hyphen === RUN TestBuildGetTableRequestRejectsInvalidNamespaceQuery/slash --- PASS: TestBuildGetTableRequestRejectsInvalidNamespaceQuery (0.00s) --- PASS: TestBuildGetTableRequestRejectsInvalidNamespaceQuery/uppercase (0.00s) --- PASS: TestBuildGetTableRequestRejectsInvalidNamespaceQuery/leading_hyphen (0.00s) --- PASS: TestBuildGetTableRequestRejectsInvalidNamespaceQuery/slash (0.00s) === RUN TestHandleRestOperationReturnsBadRequestForInvalidNamespaceQuery === RUN TestHandleRestOperationReturnsBadRequestForInvalidNamespaceQuery/uppercase === RUN TestHandleRestOperationReturnsBadRequestForInvalidNamespaceQuery/leading_hyphen === RUN TestHandleRestOperationReturnsBadRequestForInvalidNamespaceQuery/slash --- PASS: TestHandleRestOperationReturnsBadRequestForInvalidNamespaceQuery (0.01s) --- PASS: TestHandleRestOperationReturnsBadRequestForInvalidNamespaceQuery/uppercase (0.00s) --- PASS: TestHandleRestOperationReturnsBadRequestForInvalidNamespaceQuery/leading_hyphen (0.00s) --- PASS: TestHandleRestOperationReturnsBadRequestForInvalidNamespaceQuery/slash (0.00s) === RUN TestIsS3TablesSignedRequest === RUN TestIsS3TablesSignedRequest/regular_S3_auth_header === RUN TestIsS3TablesSignedRequest/S3_Tables_auth_header === RUN TestIsS3TablesSignedRequest/S3_Tables_presigned_query === RUN TestIsS3TablesSignedRequest/regular_S3_presigned_query === RUN TestIsS3TablesSignedRequest/unsigned_request === RUN TestIsS3TablesSignedRequest/malformed_credential === RUN TestIsS3TablesSignedRequest/service_substring_in_access_key_must_not_match --- PASS: TestIsS3TablesSignedRequest (0.01s) --- PASS: TestIsS3TablesSignedRequest/regular_S3_auth_header (0.00s) --- PASS: TestIsS3TablesSignedRequest/S3_Tables_auth_header (0.00s) --- PASS: TestIsS3TablesSignedRequest/S3_Tables_presigned_query (0.00s) --- PASS: TestIsS3TablesSignedRequest/regular_S3_presigned_query (0.00s) --- PASS: TestIsS3TablesSignedRequest/unsigned_request (0.00s) --- PASS: TestIsS3TablesSignedRequest/malformed_credential (0.00s) --- PASS: TestIsS3TablesSignedRequest/service_substring_in_access_key_must_not_match (0.00s) === RUN TestRouting_ListObjectsV2OnBucketNamedBuckets I0824 22:18:59.974347 config_loader.go:73 Using explicit credential store: memory --- PASS: TestRouting_ListObjectsV2OnBucketNamedBuckets (0.04s) === RUN TestRouting_S3TablesListTableBucketsStillReachable I0824 22:19:00.015426 config_loader.go:73 Using explicit credential store: memory --- PASS: TestRouting_S3TablesListTableBucketsStillReachable (0.04s) === RUN TestRouting_CreateBucketNamedBuckets I0824 22:19:00.051880 config_loader.go:73 Using explicit credential store: memory --- PASS: TestRouting_CreateBucketNamedBuckets (0.04s) === RUN TestRouting_GetObjectOnBucketNamedGetTable I0824 22:19:00.093836 config_loader.go:73 Using explicit credential store: memory --- PASS: TestRouting_GetObjectOnBucketNamedGetTable (0.07s) === RUN TestCopyObjectResponse 2026-08-24T22:19:00.16044423Z12345678 --- PASS: TestCopyObjectResponse (0.02s) === RUN TestCopyPartResponse 2026-08-24T22:19:00.177314524Z12345678 --- PASS: TestCopyPartResponse (0.00s) === RUN TestVersionIdFormatDetection === RUN TestVersionIdFormatDetection/new_format_-_inverted_timestamp === RUN TestVersionIdFormatDetection/new_format_-_high_value === RUN TestVersionIdFormatDetection/old_format_-_raw_timestamp === RUN TestVersionIdFormatDetection/old_format_-_low_value === RUN TestVersionIdFormatDetection/null_version === RUN TestVersionIdFormatDetection/short_version_ID === RUN TestVersionIdFormatDetection/empty_version_ID --- PASS: TestVersionIdFormatDetection (0.00s) --- PASS: TestVersionIdFormatDetection/new_format_-_inverted_timestamp (0.00s) --- PASS: TestVersionIdFormatDetection/new_format_-_high_value (0.00s) --- PASS: TestVersionIdFormatDetection/old_format_-_raw_timestamp (0.00s) --- PASS: TestVersionIdFormatDetection/old_format_-_low_value (0.00s) --- PASS: TestVersionIdFormatDetection/null_version (0.00s) --- PASS: TestVersionIdFormatDetection/short_version_ID (0.00s) --- PASS: TestVersionIdFormatDetection/empty_version_ID (0.00s) === RUN TestIsValidVersionID === RUN TestIsValidVersionID/#00 === RUN TestIsValidVersionID/null === RUN TestIsValidVersionID/opaque-version_123 === RUN TestIsValidVersionID/v1/../../../victim === RUN TestIsValidVersionID/v1\..\victim === RUN TestIsValidVersionID/bad\x00version --- PASS: TestIsValidVersionID (0.00s) --- PASS: TestIsValidVersionID/#00 (0.00s) --- PASS: TestIsValidVersionID/null (0.00s) --- PASS: TestIsValidVersionID/opaque-version_123 (0.00s) --- PASS: TestIsValidVersionID/v1/../../../victim (0.00s) --- PASS: TestIsValidVersionID/v1\..\victim (0.00s) --- PASS: TestIsValidVersionID/bad\x00version (0.00s) === RUN TestGenerateVersionIdFormats --- PASS: TestGenerateVersionIdFormats (0.00s) === RUN TestGetVersionTimestamp --- PASS: TestGetVersionTimestamp (0.00s) === RUN TestCompareVersionIdsSameFormatOld --- PASS: TestCompareVersionIdsSameFormatOld (0.00s) === RUN TestCompareVersionIdsSameFormatNew --- PASS: TestCompareVersionIdsSameFormatNew (0.00s) === RUN TestCompareVersionIdsMixedFormats --- PASS: TestCompareVersionIdsMixedFormats (0.00s) === RUN TestCompareVersionIdsNullHandling --- PASS: TestCompareVersionIdsNullHandling (0.00s) === RUN TestOldFormatBackwardCompatibility --- PASS: TestOldFormatBackwardCompatibility (0.00s) === RUN TestNewFormatSorting --- PASS: TestNewFormatSorting (0.00s) === RUN TestMixedFormatTransition --- PASS: TestMixedFormatTransition (0.00s) === RUN TestVersioningHealLogPrefix --- PASS: TestVersioningHealLogPrefix (0.00s) === RUN TestVersioningHealInfof_FormatStringSafe I0824 22:19:00.192253 s3api_versioning_reconciler.go:46 [versioning-heal] event=smoke bucket=a-bucket key=obj-with-%s-percent err=100% --- PASS: TestVersioningHealInfof_FormatStringSafe (0.00s) === RUN TestSanitizeHealArg === RUN TestSanitizeHealArg/plain_string === RUN TestSanitizeHealArg/underscored === RUN TestSanitizeHealArg/slashed_key === RUN TestSanitizeHealArg/space === RUN TestSanitizeHealArg/newline === RUN TestSanitizeHealArg/carriage_return === RUN TestSanitizeHealArg/tab === RUN TestSanitizeHealArg/quote === RUN TestSanitizeHealArg/backslash === RUN TestSanitizeHealArg/control_char_(DEL) === RUN TestSanitizeHealArg/event=_injection_attempt === RUN TestSanitizeHealArg/error_with_newline === RUN TestSanitizeHealArg/error_plain === RUN TestSanitizeHealArg/nil_arg === RUN TestSanitizeHealArg/int_arg === RUN TestSanitizeHealArg/bool_arg --- PASS: TestSanitizeHealArg (0.01s) --- PASS: TestSanitizeHealArg/plain_string (0.00s) --- PASS: TestSanitizeHealArg/underscored (0.00s) --- PASS: TestSanitizeHealArg/slashed_key (0.00s) --- PASS: TestSanitizeHealArg/space (0.00s) --- PASS: TestSanitizeHealArg/newline (0.00s) --- PASS: TestSanitizeHealArg/carriage_return (0.00s) --- PASS: TestSanitizeHealArg/tab (0.00s) --- PASS: TestSanitizeHealArg/quote (0.00s) --- PASS: TestSanitizeHealArg/backslash (0.00s) --- PASS: TestSanitizeHealArg/control_char_(DEL) (0.00s) --- PASS: TestSanitizeHealArg/event=_injection_attempt (0.00s) --- PASS: TestSanitizeHealArg/error_with_newline (0.00s) --- PASS: TestSanitizeHealArg/error_plain (0.00s) --- PASS: TestSanitizeHealArg/nil_arg (0.00s) --- PASS: TestSanitizeHealArg/int_arg (0.00s) --- PASS: TestSanitizeHealArg/bool_arg (0.00s) === RUN TestVersioningHealInfof_KeyWithWhitespaceStaysOneField --- PASS: TestVersioningHealInfof_KeyWithWhitespaceStaysOneField (0.00s) === RUN TestVersioningHealEventVocabulary --- PASS: TestVersioningHealEventVocabulary (0.00s) === RUN TestVersionsHealQueue_DedupOnEnqueue I0824 22:19:00.200619 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=obj queue_depth=1 --- PASS: TestVersionsHealQueue_DedupOnEnqueue (0.00s) === RUN TestVersionsHealQueue_CapacityCap I0824 22:19:00.201598 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key="\x00" queue_depth=1 I0824 22:19:00.201839 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key="\x01" queue_depth=2 I0824 22:19:00.201913 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key="\x02" queue_depth=3 I0824 22:19:00.202021 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key="\x03" queue_depth=4 I0824 22:19:00.202065 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key="\x04" queue_depth=5 I0824 22:19:00.202123 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key="\x05" queue_depth=6 I0824 22:19:00.202170 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key="\x06" queue_depth=7 I0824 22:19:00.202234 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key="\a" queue_depth=8 I0824 22:19:00.202307 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key="\b" queue_depth=9 I0824 22:19:00.202358 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key="\t" queue_depth=10 I0824 22:19:00.202416 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key="\n" queue_depth=11 I0824 22:19:00.202460 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key="\v" queue_depth=12 I0824 22:19:00.202750 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key="\f" queue_depth=13 I0824 22:19:00.202983 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key="\r" queue_depth=14 I0824 22:19:00.203418 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key="\x0e" queue_depth=15 I0824 22:19:00.203640 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key="\x0f" queue_depth=16 I0824 22:19:00.203723 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key="\x10" queue_depth=17 I0824 22:19:00.203791 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key="\x11" queue_depth=18 I0824 22:19:00.203848 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key="\x12" queue_depth=19 I0824 22:19:00.203908 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key="\x13" queue_depth=20 I0824 22:19:00.204014 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key="\x14" queue_depth=21 I0824 22:19:00.204057 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key="\x15" queue_depth=22 I0824 22:19:00.204103 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key="\x16" queue_depth=23 I0824 22:19:00.204160 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key="\x17" queue_depth=24 I0824 22:19:00.204201 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key="\x18" queue_depth=25 I0824 22:19:00.204273 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key="\x19" queue_depth=26 I0824 22:19:00.204364 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key="\x1a" queue_depth=27 I0824 22:19:00.204419 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key="\x1b" queue_depth=28 I0824 22:19:00.204683 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key="\x1c" queue_depth=29 I0824 22:19:00.204938 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key="\x1d" queue_depth=30 I0824 22:19:00.205023 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key="\x1e" queue_depth=31 I0824 22:19:00.205094 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key="\x1f" queue_depth=32 I0824 22:19:00.205145 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=" " queue_depth=33 I0824 22:19:00.205187 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=! queue_depth=34 I0824 22:19:00.205336 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key="\"" queue_depth=35 I0824 22:19:00.205496 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=# queue_depth=36 I0824 22:19:00.205702 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=$ queue_depth=37 I0824 22:19:00.205775 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=% queue_depth=38 I0824 22:19:00.205843 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=& queue_depth=39 I0824 22:19:00.205930 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=' queue_depth=40 I0824 22:19:00.206047 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=( queue_depth=41 I0824 22:19:00.206124 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=) queue_depth=42 I0824 22:19:00.206200 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=* queue_depth=43 I0824 22:19:00.206255 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=+ queue_depth=44 I0824 22:19:00.206308 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=, queue_depth=45 I0824 22:19:00.206635 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=- queue_depth=46 I0824 22:19:00.206861 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=. queue_depth=47 I0824 22:19:00.206942 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=/ queue_depth=48 I0824 22:19:00.206996 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=0 queue_depth=49 I0824 22:19:00.207050 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=1 queue_depth=50 I0824 22:19:00.207245 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=2 queue_depth=51 I0824 22:19:00.207424 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=3 queue_depth=52 I0824 22:19:00.207498 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=4 queue_depth=53 I0824 22:19:00.207542 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=5 queue_depth=54 I0824 22:19:00.207574 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=6 queue_depth=55 I0824 22:19:00.207642 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=7 queue_depth=56 I0824 22:19:00.207737 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=8 queue_depth=57 I0824 22:19:00.207860 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=9 queue_depth=58 I0824 22:19:00.207916 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=: queue_depth=59 I0824 22:19:00.207958 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=; queue_depth=60 I0824 22:19:00.208010 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=< queue_depth=61 I0824 22:19:00.208207 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key== queue_depth=62 I0824 22:19:00.208275 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=> queue_depth=63 I0824 22:19:00.208325 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=? queue_depth=64 I0824 22:19:00.208373 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=@ queue_depth=65 I0824 22:19:00.208411 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=A queue_depth=66 I0824 22:19:00.208582 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=B queue_depth=67 I0824 22:19:00.208760 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=C queue_depth=68 I0824 22:19:00.208952 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=D queue_depth=69 I0824 22:19:00.209135 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=E queue_depth=70 I0824 22:19:00.209293 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=F queue_depth=71 I0824 22:19:00.209368 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=G queue_depth=72 I0824 22:19:00.209432 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=H queue_depth=73 I0824 22:19:00.209582 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=I queue_depth=74 I0824 22:19:00.209650 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=J queue_depth=75 I0824 22:19:00.209744 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=K queue_depth=76 I0824 22:19:00.209894 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=L queue_depth=77 I0824 22:19:00.209946 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=M queue_depth=78 I0824 22:19:00.210211 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=N queue_depth=79 I0824 22:19:00.210398 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=O queue_depth=80 I0824 22:19:00.210456 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=P queue_depth=81 I0824 22:19:00.210505 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=Q queue_depth=82 I0824 22:19:00.210552 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=R queue_depth=83 I0824 22:19:00.210596 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=S queue_depth=84 I0824 22:19:00.210638 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=T queue_depth=85 I0824 22:19:00.210671 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=U queue_depth=86 I0824 22:19:00.210716 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=V queue_depth=87 I0824 22:19:00.210758 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=W queue_depth=88 I0824 22:19:00.210804 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=X queue_depth=89 I0824 22:19:00.210852 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=Y queue_depth=90 I0824 22:19:00.210938 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=Z queue_depth=91 I0824 22:19:00.210994 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=[ queue_depth=92 I0824 22:19:00.211046 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key="\\" queue_depth=93 I0824 22:19:00.211095 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=] queue_depth=94 I0824 22:19:00.211134 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=^ queue_depth=95 I0824 22:19:00.211167 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=_ queue_depth=96 I0824 22:19:00.211319 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=` queue_depth=97 I0824 22:19:00.211446 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=a queue_depth=98 I0824 22:19:00.211485 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=b queue_depth=99 I0824 22:19:00.211521 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=c queue_depth=100 I0824 22:19:00.211758 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=d queue_depth=101 I0824 22:19:00.211917 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=e queue_depth=102 I0824 22:19:00.211960 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=f queue_depth=103 I0824 22:19:00.212027 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=g queue_depth=104 I0824 22:19:00.212069 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=h queue_depth=105 I0824 22:19:00.212101 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=i queue_depth=106 I0824 22:19:00.212135 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=j queue_depth=107 I0824 22:19:00.212162 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=k queue_depth=108 I0824 22:19:00.212196 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=l queue_depth=109 I0824 22:19:00.212220 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=m queue_depth=110 I0824 22:19:00.212254 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=n queue_depth=111 I0824 22:19:00.212329 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=o queue_depth=112 I0824 22:19:00.212559 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=p queue_depth=113 I0824 22:19:00.212678 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=q queue_depth=114 I0824 22:19:00.212722 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=r queue_depth=115 I0824 22:19:00.212759 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=s queue_depth=116 I0824 22:19:00.212788 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=t queue_depth=117 I0824 22:19:00.212823 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=u queue_depth=118 I0824 22:19:00.212849 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=v queue_depth=119 I0824 22:19:00.212892 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=w queue_depth=120 I0824 22:19:00.212924 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=x queue_depth=121 I0824 22:19:00.212960 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=y queue_depth=122 I0824 22:19:00.212994 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=z queue_depth=123 I0824 22:19:00.213021 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key={ queue_depth=124 I0824 22:19:00.213069 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=| queue_depth=125 I0824 22:19:00.213102 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=} queue_depth=126 I0824 22:19:00.213132 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=~ queue_depth=127 I0824 22:19:00.213175 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key="\x7f" queue_depth=128 I0824 22:19:00.213214 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=€ queue_depth=129 I0824 22:19:00.213253 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key= queue_depth=130 I0824 22:19:00.213343 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=‚ queue_depth=131 I0824 22:19:00.213485 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ƒ queue_depth=132 I0824 22:19:00.213533 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=„ queue_depth=133 I0824 22:19:00.213569 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=… queue_depth=134 I0824 22:19:00.213595 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=† queue_depth=135 I0824 22:19:00.213630 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=‡ queue_depth=136 I0824 22:19:00.213654 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ˆ queue_depth=137 I0824 22:19:00.213703 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=‰ queue_depth=138 I0824 22:19:00.213739 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=Š queue_depth=139 I0824 22:19:00.213776 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=‹ queue_depth=140 I0824 22:19:00.213803 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=Œ queue_depth=141 I0824 22:19:00.213851 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key= queue_depth=142 I0824 22:19:00.213884 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=Ž queue_depth=143 I0824 22:19:00.213914 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key= queue_depth=144 I0824 22:19:00.213995 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key= queue_depth=145 I0824 22:19:00.214059 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=‘ queue_depth=146 I0824 22:19:00.214122 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=’ queue_depth=147 I0824 22:19:00.214461 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=“ queue_depth=148 I0824 22:19:00.214660 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=” queue_depth=149 I0824 22:19:00.214742 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=• queue_depth=150 I0824 22:19:00.214829 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=– queue_depth=151 I0824 22:19:00.214891 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=— queue_depth=152 I0824 22:19:00.214953 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=˜ queue_depth=153 I0824 22:19:00.215016 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=™ queue_depth=154 I0824 22:19:00.215111 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=š queue_depth=155 I0824 22:19:00.215176 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=› queue_depth=156 I0824 22:19:00.215230 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=œ queue_depth=157 I0824 22:19:00.215305 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key= queue_depth=158 I0824 22:19:00.215358 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ž queue_depth=159 I0824 22:19:00.215398 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=Ÿ queue_depth=160 I0824 22:19:00.215478 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=  queue_depth=161 I0824 22:19:00.215745 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=¡ queue_depth=162 I0824 22:19:00.215941 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=¢ queue_depth=163 I0824 22:19:00.216011 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=£ queue_depth=164 I0824 22:19:00.216098 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=¤ queue_depth=165 I0824 22:19:00.216154 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=¥ queue_depth=166 I0824 22:19:00.216230 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=¦ queue_depth=167 I0824 22:19:00.216279 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=§ queue_depth=168 I0824 22:19:00.216316 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=¨ queue_depth=169 I0824 22:19:00.216376 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=© queue_depth=170 I0824 22:19:00.216429 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ª queue_depth=171 I0824 22:19:00.216593 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=« queue_depth=172 I0824 22:19:00.216798 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=¬ queue_depth=173 I0824 22:19:00.216866 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=­ queue_depth=174 I0824 22:19:00.216922 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=® queue_depth=175 I0824 22:19:00.216972 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=¯ queue_depth=176 I0824 22:19:00.217018 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=° queue_depth=177 I0824 22:19:00.217186 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=± queue_depth=178 I0824 22:19:00.217466 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=² queue_depth=179 I0824 22:19:00.217648 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=³ queue_depth=180 I0824 22:19:00.217716 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=´ queue_depth=181 I0824 22:19:00.217770 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=µ queue_depth=182 I0824 22:19:00.217848 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=¶ queue_depth=183 I0824 22:19:00.217898 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=· queue_depth=184 I0824 22:19:00.217954 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=¸ queue_depth=185 I0824 22:19:00.218006 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=¹ queue_depth=186 I0824 22:19:00.218051 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=º queue_depth=187 I0824 22:19:00.218137 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=» queue_depth=188 I0824 22:19:00.218199 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=¼ queue_depth=189 I0824 22:19:00.218259 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=½ queue_depth=190 I0824 22:19:00.218314 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=¾ queue_depth=191 I0824 22:19:00.218365 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=¿ queue_depth=192 I0824 22:19:00.218410 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=À queue_depth=193 I0824 22:19:00.218459 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=Á queue_depth=194 I0824 22:19:00.218500 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key= queue_depth=195 I0824 22:19:00.218548 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=à queue_depth=196 I0824 22:19:00.218630 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=Ä queue_depth=197 I0824 22:19:00.218685 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=Å queue_depth=198 I0824 22:19:00.218730 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=Æ queue_depth=199 I0824 22:19:00.218783 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=Ç queue_depth=200 I0824 22:19:00.218834 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=È queue_depth=201 I0824 22:19:00.218901 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=É queue_depth=202 I0824 22:19:00.218929 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=Ê queue_depth=203 I0824 22:19:00.218968 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=Ë queue_depth=204 I0824 22:19:00.218997 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=Ì queue_depth=205 I0824 22:19:00.219125 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=Í queue_depth=206 I0824 22:19:00.219294 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=Î queue_depth=207 I0824 22:19:00.219330 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=Ï queue_depth=208 I0824 22:19:00.219373 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=Ð queue_depth=209 I0824 22:19:00.219404 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=Ñ queue_depth=210 I0824 22:19:00.219551 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=Ò queue_depth=211 I0824 22:19:00.219718 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=Ó queue_depth=212 I0824 22:19:00.219792 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=Ô queue_depth=213 I0824 22:19:00.219849 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=Õ queue_depth=214 I0824 22:19:00.219903 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=Ö queue_depth=215 I0824 22:19:00.219958 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=× queue_depth=216 I0824 22:19:00.220026 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=Ø queue_depth=217 I0824 22:19:00.220128 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=Ù queue_depth=218 I0824 22:19:00.220185 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=Ú queue_depth=219 I0824 22:19:00.220247 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=Û queue_depth=220 I0824 22:19:00.220298 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=Ü queue_depth=221 I0824 22:19:00.220339 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=Ý queue_depth=222 I0824 22:19:00.220402 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=Þ queue_depth=223 I0824 22:19:00.220453 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ß queue_depth=224 I0824 22:19:00.220723 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=à queue_depth=225 I0824 22:19:00.220866 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=á queue_depth=226 I0824 22:19:00.220913 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=â queue_depth=227 I0824 22:19:00.220978 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ã queue_depth=228 I0824 22:19:00.221025 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ä queue_depth=229 I0824 22:19:00.221074 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=å queue_depth=230 I0824 22:19:00.221121 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=æ queue_depth=231 I0824 22:19:00.221173 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ç queue_depth=232 I0824 22:19:00.221218 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=è queue_depth=233 I0824 22:19:00.221259 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=é queue_depth=234 I0824 22:19:00.221298 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ê queue_depth=235 I0824 22:19:00.221336 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ë queue_depth=236 I0824 22:19:00.221392 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ì queue_depth=237 I0824 22:19:00.221438 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=í queue_depth=238 I0824 22:19:00.221476 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=î queue_depth=239 I0824 22:19:00.221516 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ï queue_depth=240 I0824 22:19:00.221613 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ð queue_depth=241 I0824 22:19:00.221744 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ñ queue_depth=242 I0824 22:19:00.221797 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ò queue_depth=243 I0824 22:19:00.221975 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ó queue_depth=244 I0824 22:19:00.222223 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ô queue_depth=245 I0824 22:19:00.222413 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=õ queue_depth=246 I0824 22:19:00.222460 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ö queue_depth=247 I0824 22:19:00.222492 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=÷ queue_depth=248 I0824 22:19:00.222530 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ø queue_depth=249 I0824 22:19:00.222559 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ù queue_depth=250 I0824 22:19:00.222600 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ú queue_depth=251 I0824 22:19:00.222638 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=û queue_depth=252 I0824 22:19:00.222671 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ü queue_depth=253 I0824 22:19:00.222716 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ý queue_depth=254 I0824 22:19:00.222774 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=þ queue_depth=255 I0824 22:19:00.222816 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ÿ queue_depth=256 I0824 22:19:00.222858 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=Ā queue_depth=257 I0824 22:19:00.222895 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ā queue_depth=258 I0824 22:19:00.222932 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=Ă queue_depth=259 I0824 22:19:00.222996 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ă queue_depth=260 I0824 22:19:00.223121 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=Ą queue_depth=261 I0824 22:19:00.223265 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ą queue_depth=262 I0824 22:19:00.223305 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=Ć queue_depth=263 I0824 22:19:00.223343 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ć queue_depth=264 I0824 22:19:00.223379 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=Ĉ queue_depth=265 I0824 22:19:00.223419 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ĉ queue_depth=266 I0824 22:19:00.223455 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=Ċ queue_depth=267 I0824 22:19:00.223505 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ċ queue_depth=268 I0824 22:19:00.223563 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=Č queue_depth=269 I0824 22:19:00.223600 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=č queue_depth=270 I0824 22:19:00.223669 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=Ď queue_depth=271 I0824 22:19:00.223710 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ď queue_depth=272 I0824 22:19:00.223738 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=Đ queue_depth=273 I0824 22:19:00.223778 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=đ queue_depth=274 I0824 22:19:00.223844 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=Ē queue_depth=275 I0824 22:19:00.223958 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ē queue_depth=276 I0824 22:19:00.224037 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=Ĕ queue_depth=277 I0824 22:19:00.224818 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ĕ queue_depth=278 I0824 22:19:00.225073 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=Ė queue_depth=279 I0824 22:19:00.225261 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ė queue_depth=280 I0824 22:19:00.225320 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=Ę queue_depth=281 I0824 22:19:00.225370 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ę queue_depth=282 I0824 22:19:00.225463 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=Ě queue_depth=283 I0824 22:19:00.225602 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ě queue_depth=284 I0824 22:19:00.225677 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=Ĝ queue_depth=285 I0824 22:19:00.225765 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ĝ queue_depth=286 I0824 22:19:00.225857 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=Ğ queue_depth=287 I0824 22:19:00.225917 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ğ queue_depth=288 I0824 22:19:00.225974 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=Ġ queue_depth=289 I0824 22:19:00.226050 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ġ queue_depth=290 I0824 22:19:00.226106 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=Ģ queue_depth=291 I0824 22:19:00.226343 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ģ queue_depth=292 I0824 22:19:00.226570 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=Ĥ queue_depth=293 I0824 22:19:00.226737 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ĥ queue_depth=294 I0824 22:19:00.226796 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=Ħ queue_depth=295 I0824 22:19:00.227003 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ħ queue_depth=296 I0824 22:19:00.227188 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=Ĩ queue_depth=297 I0824 22:19:00.227268 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ĩ queue_depth=298 I0824 22:19:00.227443 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=Ī queue_depth=299 I0824 22:19:00.227592 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ī queue_depth=300 I0824 22:19:00.227682 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=Ĭ queue_depth=301 I0824 22:19:00.227806 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ĭ queue_depth=302 I0824 22:19:00.227912 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=Į queue_depth=303 I0824 22:19:00.228079 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=į queue_depth=304 I0824 22:19:00.228259 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=İ queue_depth=305 I0824 22:19:00.228317 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ı queue_depth=306 I0824 22:19:00.228360 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=IJ queue_depth=307 I0824 22:19:00.228399 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ij queue_depth=308 I0824 22:19:00.228479 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=Ĵ queue_depth=309 I0824 22:19:00.228719 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ĵ queue_depth=310 I0824 22:19:00.228871 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=Ķ queue_depth=311 I0824 22:19:00.228923 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ķ queue_depth=312 I0824 22:19:00.228956 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ĸ queue_depth=313 I0824 22:19:00.229018 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=Ĺ queue_depth=314 I0824 22:19:00.229071 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ĺ queue_depth=315 I0824 22:19:00.229099 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=Ļ queue_depth=316 I0824 22:19:00.229129 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ļ queue_depth=317 I0824 22:19:00.229164 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=Ľ queue_depth=318 I0824 22:19:00.229188 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ľ queue_depth=319 I0824 22:19:00.229225 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=Ŀ queue_depth=320 I0824 22:19:00.229254 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ŀ queue_depth=321 I0824 22:19:00.229297 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=Ł queue_depth=322 I0824 22:19:00.229339 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ł queue_depth=323 I0824 22:19:00.229391 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=Ń queue_depth=324 I0824 22:19:00.229430 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ń queue_depth=325 I0824 22:19:00.229459 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=Ņ queue_depth=326 I0824 22:19:00.229582 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ņ queue_depth=327 I0824 22:19:00.229734 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=Ň queue_depth=328 I0824 22:19:00.229779 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ň queue_depth=329 I0824 22:19:00.229833 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ʼn queue_depth=330 I0824 22:19:00.229869 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=Ŋ queue_depth=331 I0824 22:19:00.229898 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ŋ queue_depth=332 I0824 22:19:00.229933 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=Ō queue_depth=333 I0824 22:19:00.229974 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ō queue_depth=334 I0824 22:19:00.230005 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=Ŏ queue_depth=335 I0824 22:19:00.230058 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ŏ queue_depth=336 I0824 22:19:00.230129 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=Ő queue_depth=337 I0824 22:19:00.230175 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ő queue_depth=338 I0824 22:19:00.230215 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=Œ queue_depth=339 I0824 22:19:00.230245 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=œ queue_depth=340 I0824 22:19:00.230285 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=Ŕ queue_depth=341 I0824 22:19:00.230312 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ŕ queue_depth=342 I0824 22:19:00.230472 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=Ŗ queue_depth=343 I0824 22:19:00.230628 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ŗ queue_depth=344 I0824 22:19:00.230684 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=Ř queue_depth=345 I0824 22:19:00.230713 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ř queue_depth=346 I0824 22:19:00.230750 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=Ś queue_depth=347 I0824 22:19:00.230778 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ś queue_depth=348 I0824 22:19:00.230815 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=Ŝ queue_depth=349 I0824 22:19:00.230842 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ŝ queue_depth=350 I0824 22:19:00.230895 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=Ş queue_depth=351 I0824 22:19:00.230939 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ş queue_depth=352 I0824 22:19:00.230987 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=Š queue_depth=353 I0824 22:19:00.231030 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=š queue_depth=354 I0824 22:19:00.231072 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=Ţ queue_depth=355 I0824 22:19:00.231103 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ţ queue_depth=356 I0824 22:19:00.231141 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=Ť queue_depth=357 I0824 22:19:00.231170 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ť queue_depth=358 I0824 22:19:00.231212 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=Ŧ queue_depth=359 I0824 22:19:00.231274 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ŧ queue_depth=360 I0824 22:19:00.231315 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=Ũ queue_depth=361 I0824 22:19:00.231361 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ũ queue_depth=362 I0824 22:19:00.231400 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=Ū queue_depth=363 I0824 22:19:00.231436 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ū queue_depth=364 I0824 22:19:00.231484 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=Ŭ queue_depth=365 I0824 22:19:00.231521 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ŭ queue_depth=366 I0824 22:19:00.231557 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=Ů queue_depth=367 I0824 22:19:00.231594 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ů queue_depth=368 I0824 22:19:00.231629 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=Ű queue_depth=369 I0824 22:19:00.231661 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ű queue_depth=370 I0824 22:19:00.231689 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=Ų queue_depth=371 I0824 22:19:00.231744 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ų queue_depth=372 I0824 22:19:00.231775 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=Ŵ queue_depth=373 I0824 22:19:00.231814 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ŵ queue_depth=374 I0824 22:19:00.231846 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=Ŷ queue_depth=375 I0824 22:19:00.231997 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ŷ queue_depth=376 I0824 22:19:00.232191 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=Ÿ queue_depth=377 I0824 22:19:00.232244 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=Ź queue_depth=378 I0824 22:19:00.232298 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ź queue_depth=379 I0824 22:19:00.232361 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=Ż queue_depth=380 I0824 22:19:00.232413 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ż queue_depth=381 I0824 22:19:00.232453 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=Ž queue_depth=382 I0824 22:19:00.232484 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ž queue_depth=383 I0824 22:19:00.232566 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ſ queue_depth=384 I0824 22:19:00.232605 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ƀ queue_depth=385 I0824 22:19:00.232630 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=Ɓ queue_depth=386 I0824 22:19:00.232654 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=Ƃ queue_depth=387 I0824 22:19:00.232689 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ƃ queue_depth=388 I0824 22:19:00.232717 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=Ƅ queue_depth=389 I0824 22:19:00.232750 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ƅ queue_depth=390 I0824 22:19:00.232776 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=Ɔ queue_depth=391 I0824 22:19:00.232818 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=Ƈ queue_depth=392 I0824 22:19:00.232866 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ƈ queue_depth=393 I0824 22:19:00.232922 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=Ɖ queue_depth=394 I0824 22:19:00.233007 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=Ɗ queue_depth=395 I0824 22:19:00.233076 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=Ƌ queue_depth=396 I0824 22:19:00.233113 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ƌ queue_depth=397 I0824 22:19:00.233172 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ƍ queue_depth=398 I0824 22:19:00.233223 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=Ǝ queue_depth=399 I0824 22:19:00.233260 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=Ə queue_depth=400 I0824 22:19:00.233285 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=Ɛ queue_depth=401 I0824 22:19:00.233324 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=Ƒ queue_depth=402 I0824 22:19:00.233350 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ƒ queue_depth=403 I0824 22:19:00.233396 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=Ɠ queue_depth=404 I0824 22:19:00.233430 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=Ɣ queue_depth=405 I0824 22:19:00.233485 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ƕ queue_depth=406 I0824 22:19:00.233523 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=Ɩ queue_depth=407 I0824 22:19:00.233553 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=Ɨ queue_depth=408 I0824 22:19:00.233679 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=Ƙ queue_depth=409 I0824 22:19:00.233877 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ƙ queue_depth=410 I0824 22:19:00.234356 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ƚ queue_depth=411 I0824 22:19:00.234581 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ƛ queue_depth=412 I0824 22:19:00.234647 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=Ɯ queue_depth=413 I0824 22:19:00.235348 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=Ɲ queue_depth=414 I0824 22:19:00.235700 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ƞ queue_depth=415 I0824 22:19:00.236127 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=Ɵ queue_depth=416 I0824 22:19:00.236304 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=Ơ queue_depth=417 I0824 22:19:00.236367 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ơ queue_depth=418 I0824 22:19:00.236406 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=Ƣ queue_depth=419 I0824 22:19:00.236441 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ƣ queue_depth=420 I0824 22:19:00.236567 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=Ƥ queue_depth=421 I0824 22:19:00.236689 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ƥ queue_depth=422 I0824 22:19:00.236730 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=Ʀ queue_depth=423 I0824 22:19:00.236761 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=Ƨ queue_depth=424 I0824 22:19:00.236843 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ƨ queue_depth=425 I0824 22:19:00.236885 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=Ʃ queue_depth=426 I0824 22:19:00.236946 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ƪ queue_depth=427 I0824 22:19:00.236975 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ƫ queue_depth=428 I0824 22:19:00.237016 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=Ƭ queue_depth=429 I0824 22:19:00.237043 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ƭ queue_depth=430 I0824 22:19:00.237100 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=Ʈ queue_depth=431 I0824 22:19:00.237128 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=Ư queue_depth=432 I0824 22:19:00.237153 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ư queue_depth=433 I0824 22:19:00.237204 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=Ʊ queue_depth=434 I0824 22:19:00.237231 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=Ʋ queue_depth=435 I0824 22:19:00.237260 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=Ƴ queue_depth=436 I0824 22:19:00.237286 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ƴ queue_depth=437 I0824 22:19:00.237312 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=Ƶ queue_depth=438 I0824 22:19:00.237341 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ƶ queue_depth=439 I0824 22:19:00.237368 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=Ʒ queue_depth=440 I0824 22:19:00.237394 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=Ƹ queue_depth=441 I0824 22:19:00.237420 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ƹ queue_depth=442 I0824 22:19:00.237445 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ƺ queue_depth=443 I0824 22:19:00.237472 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ƻ queue_depth=444 I0824 22:19:00.237534 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=Ƽ queue_depth=445 I0824 22:19:00.237563 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ƽ queue_depth=446 I0824 22:19:00.237594 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ƾ queue_depth=447 I0824 22:19:00.237622 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ƿ queue_depth=448 I0824 22:19:00.237829 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ǀ queue_depth=449 I0824 22:19:00.237941 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ǁ queue_depth=450 I0824 22:19:00.237988 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ǂ queue_depth=451 I0824 22:19:00.238024 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ǃ queue_depth=452 I0824 22:19:00.238051 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=DŽ queue_depth=453 I0824 22:19:00.238225 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=Dž queue_depth=454 I0824 22:19:00.238330 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=dž queue_depth=455 I0824 22:19:00.238371 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=LJ queue_depth=456 I0824 22:19:00.238402 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=Lj queue_depth=457 I0824 22:19:00.238431 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=lj queue_depth=458 I0824 22:19:00.238489 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=NJ queue_depth=459 I0824 22:19:00.238518 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=Nj queue_depth=460 I0824 22:19:00.238544 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=nj queue_depth=461 I0824 22:19:00.238571 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=Ǎ queue_depth=462 I0824 22:19:00.238595 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ǎ queue_depth=463 I0824 22:19:00.238620 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=Ǐ queue_depth=464 I0824 22:19:00.238646 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ǐ queue_depth=465 I0824 22:19:00.238670 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=Ǒ queue_depth=466 I0824 22:19:00.238715 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ǒ queue_depth=467 I0824 22:19:00.238742 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=Ǔ queue_depth=468 I0824 22:19:00.238769 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ǔ queue_depth=469 I0824 22:19:00.238795 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=Ǖ queue_depth=470 I0824 22:19:00.238821 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ǖ queue_depth=471 I0824 22:19:00.238847 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=Ǘ queue_depth=472 I0824 22:19:00.238871 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ǘ queue_depth=473 I0824 22:19:00.238898 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=Ǚ queue_depth=474 I0824 22:19:00.238924 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ǚ queue_depth=475 I0824 22:19:00.238951 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=Ǜ queue_depth=476 I0824 22:19:00.238980 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ǜ queue_depth=477 I0824 22:19:00.239006 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ǝ queue_depth=478 I0824 22:19:00.239034 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=Ǟ queue_depth=479 I0824 22:19:00.239116 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ǟ queue_depth=480 I0824 22:19:00.239156 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=Ǡ queue_depth=481 I0824 22:19:00.239235 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ǡ queue_depth=482 I0824 22:19:00.239270 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=Ǣ queue_depth=483 I0824 22:19:00.239296 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ǣ queue_depth=484 I0824 22:19:00.239322 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=Ǥ queue_depth=485 I0824 22:19:00.239389 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ǥ queue_depth=486 I0824 22:19:00.239441 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=Ǧ queue_depth=487 I0824 22:19:00.239472 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ǧ queue_depth=488 I0824 22:19:00.239499 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=Ǩ queue_depth=489 I0824 22:19:00.239526 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ǩ queue_depth=490 I0824 22:19:00.239552 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=Ǫ queue_depth=491 I0824 22:19:00.239597 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ǫ queue_depth=492 I0824 22:19:00.239668 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=Ǭ queue_depth=493 I0824 22:19:00.239696 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ǭ queue_depth=494 I0824 22:19:00.239721 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=Ǯ queue_depth=495 I0824 22:19:00.239751 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ǯ queue_depth=496 I0824 22:19:00.239775 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ǰ queue_depth=497 I0824 22:19:00.239848 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=DZ queue_depth=498 I0824 22:19:00.239885 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=Dz queue_depth=499 I0824 22:19:00.239912 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=dz queue_depth=500 I0824 22:19:00.239938 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=Ǵ queue_depth=501 I0824 22:19:00.239969 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ǵ queue_depth=502 I0824 22:19:00.239998 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=Ƕ queue_depth=503 I0824 22:19:00.240023 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=Ƿ queue_depth=504 I0824 22:19:00.240072 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=Ǹ queue_depth=505 I0824 22:19:00.240100 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ǹ queue_depth=506 I0824 22:19:00.240131 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=Ǻ queue_depth=507 I0824 22:19:00.240194 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ǻ queue_depth=508 I0824 22:19:00.240227 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=Ǽ queue_depth=509 I0824 22:19:00.240254 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ǽ queue_depth=510 I0824 22:19:00.240285 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=Ǿ queue_depth=511 I0824 22:19:00.240312 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ǿ queue_depth=512 I0824 22:19:00.240337 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=Ȁ queue_depth=513 I0824 22:19:00.240432 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ȁ queue_depth=514 I0824 22:19:00.240467 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=Ȃ queue_depth=515 I0824 22:19:00.240495 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ȃ queue_depth=516 I0824 22:19:00.240561 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=Ȅ queue_depth=517 I0824 22:19:00.240587 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ȅ queue_depth=518 I0824 22:19:00.240615 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=Ȇ queue_depth=519 I0824 22:19:00.240669 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ȇ queue_depth=520 I0824 22:19:00.240703 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=Ȉ queue_depth=521 I0824 22:19:00.240730 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ȉ queue_depth=522 I0824 22:19:00.240756 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=Ȋ queue_depth=523 I0824 22:19:00.240782 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ȋ queue_depth=524 I0824 22:19:00.240806 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=Ȍ queue_depth=525 I0824 22:19:00.240832 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ȍ queue_depth=526 I0824 22:19:00.240857 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=Ȏ queue_depth=527 I0824 22:19:00.240882 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ȏ queue_depth=528 I0824 22:19:00.240909 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=Ȑ queue_depth=529 I0824 22:19:00.240955 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ȑ queue_depth=530 I0824 22:19:00.240981 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=Ȓ queue_depth=531 I0824 22:19:00.241009 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ȓ queue_depth=532 I0824 22:19:00.241033 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=Ȕ queue_depth=533 I0824 22:19:00.241059 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ȕ queue_depth=534 I0824 22:19:00.241083 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=Ȗ queue_depth=535 I0824 22:19:00.241109 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ȗ queue_depth=536 I0824 22:19:00.241133 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=Ș queue_depth=537 I0824 22:19:00.241157 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ș queue_depth=538 I0824 22:19:00.241182 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=Ț queue_depth=539 I0824 22:19:00.241209 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ț queue_depth=540 I0824 22:19:00.241233 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=Ȝ queue_depth=541 I0824 22:19:00.241276 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ȝ queue_depth=542 I0824 22:19:00.241302 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=Ȟ queue_depth=543 I0824 22:19:00.241327 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ȟ queue_depth=544 I0824 22:19:00.241353 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=Ƞ queue_depth=545 I0824 22:19:00.241379 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ȡ queue_depth=546 I0824 22:19:00.241410 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=Ȣ queue_depth=547 I0824 22:19:00.241474 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ȣ queue_depth=548 I0824 22:19:00.241501 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=Ȥ queue_depth=549 I0824 22:19:00.241528 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ȥ queue_depth=550 I0824 22:19:00.241570 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=Ȧ queue_depth=551 I0824 22:19:00.241596 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ȧ queue_depth=552 I0824 22:19:00.241646 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=Ȩ queue_depth=553 I0824 22:19:00.241677 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ȩ queue_depth=554 I0824 22:19:00.241702 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=Ȫ queue_depth=555 I0824 22:19:00.241727 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ȫ queue_depth=556 I0824 22:19:00.241754 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=Ȭ queue_depth=557 I0824 22:19:00.241779 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ȭ queue_depth=558 I0824 22:19:00.241804 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=Ȯ queue_depth=559 I0824 22:19:00.241849 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ȯ queue_depth=560 I0824 22:19:00.241876 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=Ȱ queue_depth=561 I0824 22:19:00.241902 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ȱ queue_depth=562 I0824 22:19:00.241928 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=Ȳ queue_depth=563 I0824 22:19:00.241952 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ȳ queue_depth=564 I0824 22:19:00.241977 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ȴ queue_depth=565 I0824 22:19:00.242002 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ȵ queue_depth=566 I0824 22:19:00.242027 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ȶ queue_depth=567 I0824 22:19:00.242074 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ȷ queue_depth=568 I0824 22:19:00.242102 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ȸ queue_depth=569 I0824 22:19:00.242128 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ȹ queue_depth=570 I0824 22:19:00.242217 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=Ⱥ queue_depth=571 I0824 22:19:00.242251 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=Ȼ queue_depth=572 I0824 22:19:00.242278 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ȼ queue_depth=573 I0824 22:19:00.242304 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=Ƚ queue_depth=574 I0824 22:19:00.242330 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=Ⱦ queue_depth=575 I0824 22:19:00.242356 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ȿ queue_depth=576 I0824 22:19:00.242382 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ɀ queue_depth=577 I0824 22:19:00.242406 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=Ɂ queue_depth=578 I0824 22:19:00.242431 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ɂ queue_depth=579 I0824 22:19:00.242524 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=Ƀ queue_depth=580 I0824 22:19:00.242558 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=Ʉ queue_depth=581 I0824 22:19:00.242616 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=Ʌ queue_depth=582 I0824 22:19:00.242643 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=Ɇ queue_depth=583 I0824 22:19:00.242673 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ɇ queue_depth=584 I0824 22:19:00.242700 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=Ɉ queue_depth=585 I0824 22:19:00.242751 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ɉ queue_depth=586 I0824 22:19:00.242782 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=Ɋ queue_depth=587 I0824 22:19:00.242830 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ɋ queue_depth=588 I0824 22:19:00.242859 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=Ɍ queue_depth=589 I0824 22:19:00.242888 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ɍ queue_depth=590 I0824 22:19:00.242940 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=Ɏ queue_depth=591 I0824 22:19:00.242971 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ɏ queue_depth=592 I0824 22:19:00.242995 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ɐ queue_depth=593 I0824 22:19:00.243040 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ɑ queue_depth=594 I0824 22:19:00.243067 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ɒ queue_depth=595 I0824 22:19:00.243092 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ɓ queue_depth=596 I0824 22:19:00.243122 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ɔ queue_depth=597 I0824 22:19:00.243147 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ɕ queue_depth=598 I0824 22:19:00.243172 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ɖ queue_depth=599 I0824 22:19:00.243218 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ɗ queue_depth=600 I0824 22:19:00.243249 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ɘ queue_depth=601 I0824 22:19:00.243276 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ə queue_depth=602 I0824 22:19:00.243302 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ɚ queue_depth=603 I0824 22:19:00.243328 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ɛ queue_depth=604 I0824 22:19:00.243504 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ɜ queue_depth=605 I0824 22:19:00.243666 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ɝ queue_depth=606 I0824 22:19:00.243724 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ɞ queue_depth=607 I0824 22:19:00.243759 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ɟ queue_depth=608 I0824 22:19:00.243811 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ɠ queue_depth=609 I0824 22:19:00.243843 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ɡ queue_depth=610 I0824 22:19:00.243872 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ɢ queue_depth=611 I0824 22:19:00.243898 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ɣ queue_depth=612 I0824 22:19:00.243929 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ɤ queue_depth=613 I0824 22:19:00.243954 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ɥ queue_depth=614 I0824 22:19:00.243978 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ɦ queue_depth=615 I0824 22:19:00.244078 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ɧ queue_depth=616 I0824 22:19:00.244200 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ɨ queue_depth=617 I0824 22:19:00.244375 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ɩ queue_depth=618 I0824 22:19:00.244557 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ɪ queue_depth=619 I0824 22:19:00.244705 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ɫ queue_depth=620 I0824 22:19:00.244789 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ɬ queue_depth=621 I0824 22:19:00.244826 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ɭ queue_depth=622 I0824 22:19:00.244860 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ɮ queue_depth=623 I0824 22:19:00.244896 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ɯ queue_depth=624 I0824 22:19:00.244925 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ɰ queue_depth=625 I0824 22:19:00.244954 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ɱ queue_depth=626 I0824 22:19:00.244979 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ɲ queue_depth=627 I0824 22:19:00.245003 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ɳ queue_depth=628 I0824 22:19:00.245030 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ɴ queue_depth=629 I0824 22:19:00.245056 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ɵ queue_depth=630 I0824 22:19:00.245082 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ɶ queue_depth=631 I0824 22:19:00.245111 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ɷ queue_depth=632 I0824 22:19:00.245135 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ɸ queue_depth=633 I0824 22:19:00.245160 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ɹ queue_depth=634 I0824 22:19:00.245185 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ɺ queue_depth=635 I0824 22:19:00.245209 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ɻ queue_depth=636 I0824 22:19:00.245233 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ɼ queue_depth=637 I0824 22:19:00.245305 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ɽ queue_depth=638 I0824 22:19:00.245329 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ɾ queue_depth=639 I0824 22:19:00.245356 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ɿ queue_depth=640 I0824 22:19:00.245404 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ʀ queue_depth=641 I0824 22:19:00.245430 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ʁ queue_depth=642 I0824 22:19:00.245457 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ʂ queue_depth=643 I0824 22:19:00.245483 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ʃ queue_depth=644 I0824 22:19:00.245513 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ʄ queue_depth=645 I0824 22:19:00.245538 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ʅ queue_depth=646 I0824 22:19:00.245561 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ʆ queue_depth=647 I0824 22:19:00.245586 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ʇ queue_depth=648 I0824 22:19:00.245613 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ʈ queue_depth=649 I0824 22:19:00.245712 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ʉ queue_depth=650 I0824 22:19:00.245745 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ʊ queue_depth=651 I0824 22:19:00.245819 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ʋ queue_depth=652 I0824 22:19:00.245876 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ʌ queue_depth=653 I0824 22:19:00.245907 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ʍ queue_depth=654 I0824 22:19:00.245937 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ʎ queue_depth=655 I0824 22:19:00.245964 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ʏ queue_depth=656 I0824 22:19:00.245989 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ʐ queue_depth=657 I0824 22:19:00.246014 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ʑ queue_depth=658 I0824 22:19:00.246041 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ʒ queue_depth=659 I0824 22:19:00.246067 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ʓ queue_depth=660 I0824 22:19:00.246119 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ʔ queue_depth=661 I0824 22:19:00.246194 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ʕ queue_depth=662 I0824 22:19:00.246232 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ʖ queue_depth=663 I0824 22:19:00.246260 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ʗ queue_depth=664 I0824 22:19:00.246287 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ʘ queue_depth=665 I0824 22:19:00.246312 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ʙ queue_depth=666 I0824 22:19:00.246340 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ʚ queue_depth=667 I0824 22:19:00.246368 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ʛ queue_depth=668 I0824 22:19:00.246393 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ʜ queue_depth=669 I0824 22:19:00.246421 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ʝ queue_depth=670 I0824 22:19:00.246445 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ʞ queue_depth=671 I0824 22:19:00.246491 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ʟ queue_depth=672 I0824 22:19:00.246559 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ʠ queue_depth=673 I0824 22:19:00.246604 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ʡ queue_depth=674 I0824 22:19:00.246629 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ʢ queue_depth=675 I0824 22:19:00.246654 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ʣ queue_depth=676 I0824 22:19:00.246680 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ʤ queue_depth=677 I0824 22:19:00.246705 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ʥ queue_depth=678 I0824 22:19:00.246731 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ʦ queue_depth=679 I0824 22:19:00.246757 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ʧ queue_depth=680 I0824 22:19:00.246780 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ʨ queue_depth=681 I0824 22:19:00.246808 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ʩ queue_depth=682 I0824 22:19:00.246832 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ʪ queue_depth=683 I0824 22:19:00.246923 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ʫ queue_depth=684 I0824 22:19:00.246996 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ʬ queue_depth=685 I0824 22:19:00.247033 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ʭ queue_depth=686 I0824 22:19:00.247063 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ʮ queue_depth=687 I0824 22:19:00.247089 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ʯ queue_depth=688 I0824 22:19:00.247116 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ʰ queue_depth=689 I0824 22:19:00.247142 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ʱ queue_depth=690 I0824 22:19:00.247168 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ʲ queue_depth=691 I0824 22:19:00.247195 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ʳ queue_depth=692 I0824 22:19:00.247221 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ʴ queue_depth=693 I0824 22:19:00.247248 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ʵ queue_depth=694 I0824 22:19:00.247274 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ʶ queue_depth=695 I0824 22:19:00.247300 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ʷ queue_depth=696 I0824 22:19:00.247329 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ʸ queue_depth=697 I0824 22:19:00.247355 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ʹ queue_depth=698 I0824 22:19:00.247380 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ʺ queue_depth=699 I0824 22:19:00.247406 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ʻ queue_depth=700 I0824 22:19:00.247433 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ʼ queue_depth=701 I0824 22:19:00.247459 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ʽ queue_depth=702 I0824 22:19:00.247506 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ʾ queue_depth=703 I0824 22:19:00.247553 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ʿ queue_depth=704 I0824 22:19:00.247582 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ˀ queue_depth=705 I0824 22:19:00.247605 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ˁ queue_depth=706 I0824 22:19:00.247631 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=˂ queue_depth=707 I0824 22:19:00.247657 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=˃ queue_depth=708 I0824 22:19:00.247705 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=˄ queue_depth=709 I0824 22:19:00.247734 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=˅ queue_depth=710 I0824 22:19:00.247758 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ˆ queue_depth=711 I0824 22:19:00.247784 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ˇ queue_depth=712 I0824 22:19:00.247809 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ˈ queue_depth=713 I0824 22:19:00.247833 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ˉ queue_depth=714 I0824 22:19:00.247859 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ˊ queue_depth=715 I0824 22:19:00.247884 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ˋ queue_depth=716 I0824 22:19:00.247909 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ˌ queue_depth=717 I0824 22:19:00.248006 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ˍ queue_depth=718 I0824 22:19:00.248127 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ˎ queue_depth=719 I0824 22:19:00.248172 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ˏ queue_depth=720 I0824 22:19:00.248218 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ː queue_depth=721 I0824 22:19:00.248246 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ˑ queue_depth=722 I0824 22:19:00.248275 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=˒ queue_depth=723 I0824 22:19:00.248307 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=˓ queue_depth=724 I0824 22:19:00.248334 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=˔ queue_depth=725 I0824 22:19:00.248381 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=˕ queue_depth=726 I0824 22:19:00.248410 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=˖ queue_depth=727 I0824 22:19:00.248437 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=˗ queue_depth=728 I0824 22:19:00.248463 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=˘ queue_depth=729 I0824 22:19:00.248488 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=˙ queue_depth=730 I0824 22:19:00.248515 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=˚ queue_depth=731 I0824 22:19:00.248542 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=˛ queue_depth=732 I0824 22:19:00.248567 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=˜ queue_depth=733 I0824 22:19:00.248652 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=˝ queue_depth=734 I0824 22:19:00.248714 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=˞ queue_depth=735 I0824 22:19:00.248782 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=˟ queue_depth=736 I0824 22:19:00.248816 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ˠ queue_depth=737 I0824 22:19:00.248842 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ˡ queue_depth=738 I0824 22:19:00.248895 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ˢ queue_depth=739 I0824 22:19:00.248922 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ˣ queue_depth=740 I0824 22:19:00.248947 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ˤ queue_depth=741 I0824 22:19:00.248973 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=˥ queue_depth=742 I0824 22:19:00.248997 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=˦ queue_depth=743 I0824 22:19:00.249060 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=˧ queue_depth=744 I0824 22:19:00.249094 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=˨ queue_depth=745 I0824 22:19:00.249121 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=˩ queue_depth=746 I0824 22:19:00.249147 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=˪ queue_depth=747 I0824 22:19:00.249174 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=˫ queue_depth=748 I0824 22:19:00.249199 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ˬ queue_depth=749 I0824 22:19:00.249226 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=˭ queue_depth=750 I0824 22:19:00.249281 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ˮ queue_depth=751 I0824 22:19:00.249352 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=˯ queue_depth=752 I0824 22:19:00.249399 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=˰ queue_depth=753 I0824 22:19:00.249427 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=˱ queue_depth=754 I0824 22:19:00.249452 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=˲ queue_depth=755 I0824 22:19:00.249532 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=˳ queue_depth=756 I0824 22:19:00.249640 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=˴ queue_depth=757 I0824 22:19:00.249707 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=˵ queue_depth=758 I0824 22:19:00.249755 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=˶ queue_depth=759 I0824 22:19:00.249788 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=˷ queue_depth=760 I0824 22:19:00.249819 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=˸ queue_depth=761 I0824 22:19:00.249848 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=˹ queue_depth=762 I0824 22:19:00.249876 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=˺ queue_depth=763 I0824 22:19:00.249901 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=˻ queue_depth=764 I0824 22:19:00.249925 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=˼ queue_depth=765 I0824 22:19:00.249969 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=˽ queue_depth=766 I0824 22:19:00.249997 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=˾ queue_depth=767 I0824 22:19:00.250025 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=˿ queue_depth=768 I0824 22:19:00.250052 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=̀ queue_depth=769 I0824 22:19:00.250078 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=́ queue_depth=770 I0824 22:19:00.250105 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=̂ queue_depth=771 I0824 22:19:00.250132 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=̃ queue_depth=772 I0824 22:19:00.250157 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=̄ queue_depth=773 I0824 22:19:00.250182 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=̅ queue_depth=774 I0824 22:19:00.250208 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=̆ queue_depth=775 I0824 22:19:00.250254 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=̇ queue_depth=776 I0824 22:19:00.250282 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=̈ queue_depth=777 I0824 22:19:00.250308 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=̉ queue_depth=778 I0824 22:19:00.250333 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=̊ queue_depth=779 I0824 22:19:00.250356 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=̋ queue_depth=780 I0824 22:19:00.250381 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=̌ queue_depth=781 I0824 22:19:00.250406 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=̍ queue_depth=782 I0824 22:19:00.250430 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=̎ queue_depth=783 I0824 22:19:00.250506 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=̏ queue_depth=784 I0824 22:19:00.250624 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=̐ queue_depth=785 I0824 22:19:00.250683 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=̑ queue_depth=786 I0824 22:19:00.250766 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=̒ queue_depth=787 I0824 22:19:00.250796 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=̓ queue_depth=788 I0824 22:19:00.250822 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=̔ queue_depth=789 I0824 22:19:00.250849 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=̕ queue_depth=790 I0824 22:19:00.250877 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=̖ queue_depth=791 I0824 22:19:00.250901 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=̗ queue_depth=792 I0824 22:19:00.250926 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=̘ queue_depth=793 I0824 22:19:00.250951 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=̙ queue_depth=794 I0824 22:19:00.250976 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=̚ queue_depth=795 I0824 22:19:00.251002 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=̛ queue_depth=796 I0824 22:19:00.251029 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=̜ queue_depth=797 I0824 22:19:00.251082 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=̝ queue_depth=798 I0824 22:19:00.251109 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=̞ queue_depth=799 I0824 22:19:00.251134 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=̟ queue_depth=800 I0824 22:19:00.251160 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=̠ queue_depth=801 I0824 22:19:00.251185 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=̡ queue_depth=802 I0824 22:19:00.251210 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=̢ queue_depth=803 I0824 22:19:00.251237 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=̣ queue_depth=804 I0824 22:19:00.251261 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=̤ queue_depth=805 I0824 22:19:00.251284 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=̥ queue_depth=806 I0824 22:19:00.251309 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=̦ queue_depth=807 I0824 22:19:00.251335 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=̧ queue_depth=808 I0824 22:19:00.251363 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=̨ queue_depth=809 I0824 22:19:00.251391 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=̩ queue_depth=810 I0824 22:19:00.251416 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=̪ queue_depth=811 I0824 22:19:00.251443 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=̫ queue_depth=812 I0824 22:19:00.251490 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=̬ queue_depth=813 I0824 22:19:00.251547 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=̭ queue_depth=814 I0824 22:19:00.251573 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=̮ queue_depth=815 I0824 22:19:00.251598 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=̯ queue_depth=816 I0824 22:19:00.251667 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=̰ queue_depth=817 I0824 22:19:00.251715 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=̱ queue_depth=818 I0824 22:19:00.251743 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=̲ queue_depth=819 I0824 22:19:00.251768 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=̳ queue_depth=820 I0824 22:19:00.251814 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=̴ queue_depth=821 I0824 22:19:00.251845 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=̵ queue_depth=822 I0824 22:19:00.251870 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=̶ queue_depth=823 I0824 22:19:00.251896 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=̷ queue_depth=824 I0824 22:19:00.251922 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key≠ queue_depth=825 I0824 22:19:00.251976 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=̹ queue_depth=826 I0824 22:19:00.252006 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=̺ queue_depth=827 I0824 22:19:00.252059 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=̻ queue_depth=828 I0824 22:19:00.252091 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=̼ queue_depth=829 I0824 22:19:00.252116 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=̽ queue_depth=830 I0824 22:19:00.252141 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=̾ queue_depth=831 I0824 22:19:00.252186 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=̿ queue_depth=832 I0824 22:19:00.252228 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=̀ queue_depth=833 I0824 22:19:00.252254 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=́ queue_depth=834 I0824 22:19:00.252280 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=͂ queue_depth=835 I0824 22:19:00.252304 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=̓ queue_depth=836 I0824 22:19:00.252328 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=̈́ queue_depth=837 I0824 22:19:00.252353 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ͅ queue_depth=838 I0824 22:19:00.252493 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=͆ queue_depth=839 I0824 22:19:00.252617 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=͇ queue_depth=840 I0824 22:19:00.252664 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=͈ queue_depth=841 I0824 22:19:00.252707 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=͉ queue_depth=842 I0824 22:19:00.252749 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=͊ queue_depth=843 I0824 22:19:00.252785 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=͋ queue_depth=844 I0824 22:19:00.252822 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=͌ queue_depth=845 I0824 22:19:00.252859 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=͍ queue_depth=846 I0824 22:19:00.252891 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=͎ queue_depth=847 I0824 22:19:00.252928 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=͏ queue_depth=848 I0824 22:19:00.252964 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=͐ queue_depth=849 I0824 22:19:00.253193 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=͑ queue_depth=850 I0824 22:19:00.253291 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=͒ queue_depth=851 I0824 22:19:00.253337 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=͓ queue_depth=852 I0824 22:19:00.253368 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=͔ queue_depth=853 I0824 22:19:00.253423 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=͕ queue_depth=854 I0824 22:19:00.253506 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=͖ queue_depth=855 I0824 22:19:00.253574 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=͗ queue_depth=856 I0824 22:19:00.253607 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=͘ queue_depth=857 I0824 22:19:00.253649 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=͙ queue_depth=858 I0824 22:19:00.253683 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=͚ queue_depth=859 I0824 22:19:00.253714 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=͛ queue_depth=860 I0824 22:19:00.253743 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=͜ queue_depth=861 I0824 22:19:00.253768 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=͝ queue_depth=862 I0824 22:19:00.253802 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=͞ queue_depth=863 I0824 22:19:00.253831 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=͟ queue_depth=864 I0824 22:19:00.253862 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=͠ queue_depth=865 I0824 22:19:00.253887 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=͡ queue_depth=866 I0824 22:19:00.253918 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=͢ queue_depth=867 I0824 22:19:00.253992 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ͣ queue_depth=868 I0824 22:19:00.254026 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ͤ queue_depth=869 I0824 22:19:00.254053 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ͥ queue_depth=870 I0824 22:19:00.254090 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ͦ queue_depth=871 I0824 22:19:00.254125 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ͧ queue_depth=872 I0824 22:19:00.254166 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ͨ queue_depth=873 I0824 22:19:00.254198 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ͩ queue_depth=874 I0824 22:19:00.254230 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ͪ queue_depth=875 I0824 22:19:00.254259 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ͫ queue_depth=876 I0824 22:19:00.254283 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ͬ queue_depth=877 I0824 22:19:00.254322 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ͭ queue_depth=878 I0824 22:19:00.254353 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ͮ queue_depth=879 I0824 22:19:00.254391 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ͯ queue_depth=880 I0824 22:19:00.254417 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=Ͱ queue_depth=881 I0824 22:19:00.275231 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ͱ queue_depth=882 I0824 22:19:00.275765 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=Ͳ queue_depth=883 I0824 22:19:00.275901 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ͳ queue_depth=884 I0824 22:19:00.275963 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ʹ queue_depth=885 I0824 22:19:00.276014 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=͵ queue_depth=886 I0824 22:19:00.276150 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=Ͷ queue_depth=887 I0824 22:19:00.276225 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ͷ queue_depth=888 I0824 22:19:00.276429 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=͸ queue_depth=889 I0824 22:19:00.276770 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=͹ queue_depth=890 I0824 22:19:00.276966 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ͺ queue_depth=891 I0824 22:19:00.277084 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ͻ queue_depth=892 I0824 22:19:00.277158 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ͼ queue_depth=893 I0824 22:19:00.277255 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ͽ queue_depth=894 I0824 22:19:00.277349 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=; queue_depth=895 I0824 22:19:00.277493 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=Ϳ queue_depth=896 I0824 22:19:00.278870 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=΀ queue_depth=897 I0824 22:19:00.279146 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=΁ queue_depth=898 I0824 22:19:00.279362 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=΂ queue_depth=899 I0824 22:19:00.279439 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=΃ queue_depth=900 I0824 22:19:00.279752 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=΄ queue_depth=901 I0824 22:19:00.279943 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=΅ queue_depth=902 I0824 22:19:00.280062 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=Ά queue_depth=903 I0824 22:19:00.280147 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=· queue_depth=904 I0824 22:19:00.280211 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=Έ queue_depth=905 I0824 22:19:00.280275 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=Ή queue_depth=906 I0824 22:19:00.280333 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=Ί queue_depth=907 I0824 22:19:00.280551 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=΋ queue_depth=908 I0824 22:19:00.280754 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=Ό queue_depth=909 I0824 22:19:00.280811 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=΍ queue_depth=910 I0824 22:19:00.280858 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=Ύ queue_depth=911 I0824 22:19:00.280935 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=Ώ queue_depth=912 I0824 22:19:00.281013 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ΐ queue_depth=913 I0824 22:19:00.281130 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=Α queue_depth=914 I0824 22:19:00.281308 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=Β queue_depth=915 I0824 22:19:00.281560 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=Γ queue_depth=916 I0824 22:19:00.281781 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=Δ queue_depth=917 I0824 22:19:00.281843 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=Ε queue_depth=918 I0824 22:19:00.281890 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=Ζ queue_depth=919 I0824 22:19:00.281930 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=Η queue_depth=920 I0824 22:19:00.282180 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=Θ queue_depth=921 I0824 22:19:00.282394 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=Ι queue_depth=922 I0824 22:19:00.282517 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=Κ queue_depth=923 I0824 22:19:00.282709 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=Λ queue_depth=924 I0824 22:19:00.282814 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=Μ queue_depth=925 I0824 22:19:00.282882 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=Ν queue_depth=926 I0824 22:19:00.282933 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=Ξ queue_depth=927 I0824 22:19:00.283005 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=Ο queue_depth=928 I0824 22:19:00.283046 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=Π queue_depth=929 I0824 22:19:00.283144 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=Ρ queue_depth=930 I0824 22:19:00.283233 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=΢ queue_depth=931 I0824 22:19:00.283304 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=Σ queue_depth=932 I0824 22:19:00.283426 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=Τ queue_depth=933 I0824 22:19:00.283492 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=Υ queue_depth=934 I0824 22:19:00.283553 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=Φ queue_depth=935 I0824 22:19:00.283737 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=Χ queue_depth=936 I0824 22:19:00.283825 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=Ψ queue_depth=937 I0824 22:19:00.283898 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=Ω queue_depth=938 I0824 22:19:00.283961 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=Ϊ queue_depth=939 I0824 22:19:00.284000 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=Ϋ queue_depth=940 I0824 22:19:00.284043 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ά queue_depth=941 I0824 22:19:00.284107 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=έ queue_depth=942 I0824 22:19:00.284169 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ή queue_depth=943 I0824 22:19:00.284354 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ί queue_depth=944 I0824 22:19:00.284796 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ΰ queue_depth=945 I0824 22:19:00.284924 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=α queue_depth=946 I0824 22:19:00.285067 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=β queue_depth=947 I0824 22:19:00.285227 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=γ queue_depth=948 I0824 22:19:00.285554 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=δ queue_depth=949 I0824 22:19:00.285804 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ε queue_depth=950 I0824 22:19:00.285872 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ζ queue_depth=951 I0824 22:19:00.285913 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=η queue_depth=952 I0824 22:19:00.285973 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=θ queue_depth=953 I0824 22:19:00.286039 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ι queue_depth=954 I0824 22:19:00.286159 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=κ queue_depth=955 I0824 22:19:00.286224 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=λ queue_depth=956 I0824 22:19:00.286405 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=μ queue_depth=957 I0824 22:19:00.286514 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ν queue_depth=958 I0824 22:19:00.286575 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ξ queue_depth=959 I0824 22:19:00.286634 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ο queue_depth=960 I0824 22:19:00.286676 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=π queue_depth=961 I0824 22:19:00.286759 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ρ queue_depth=962 I0824 22:19:00.286814 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ς queue_depth=963 I0824 22:19:00.286860 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=σ queue_depth=964 I0824 22:19:00.286891 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=τ queue_depth=965 I0824 22:19:00.287006 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=υ queue_depth=966 I0824 22:19:00.287081 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=φ queue_depth=967 I0824 22:19:00.287142 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=χ queue_depth=968 I0824 22:19:00.287249 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ψ queue_depth=969 I0824 22:19:00.287318 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ω queue_depth=970 I0824 22:19:00.287356 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ϊ queue_depth=971 I0824 22:19:00.287394 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ϋ queue_depth=972 I0824 22:19:00.287443 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ό queue_depth=973 I0824 22:19:00.287490 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ύ queue_depth=974 I0824 22:19:00.287528 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ώ queue_depth=975 I0824 22:19:00.287605 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=Ϗ queue_depth=976 I0824 22:19:00.287664 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ϐ queue_depth=977 I0824 22:19:00.287721 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ϑ queue_depth=978 I0824 22:19:00.287772 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ϒ queue_depth=979 I0824 22:19:00.287818 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ϓ queue_depth=980 I0824 22:19:00.287871 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ϔ queue_depth=981 I0824 22:19:00.288095 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ϕ queue_depth=982 I0824 22:19:00.288176 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ϖ queue_depth=983 I0824 22:19:00.288366 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ϗ queue_depth=984 I0824 22:19:00.288485 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=Ϙ queue_depth=985 I0824 22:19:00.288711 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ϙ queue_depth=986 I0824 22:19:00.288763 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=Ϛ queue_depth=987 I0824 22:19:00.288800 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ϛ queue_depth=988 I0824 22:19:00.288838 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=Ϝ queue_depth=989 I0824 22:19:00.289138 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ϝ queue_depth=990 I0824 22:19:00.289246 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=Ϟ queue_depth=991 I0824 22:19:00.290337 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ϟ queue_depth=992 I0824 22:19:00.290482 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=Ϡ queue_depth=993 I0824 22:19:00.290545 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ϡ queue_depth=994 I0824 22:19:00.290637 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=Ϣ queue_depth=995 I0824 22:19:00.290685 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ϣ queue_depth=996 I0824 22:19:00.290734 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=Ϥ queue_depth=997 I0824 22:19:00.290781 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ϥ queue_depth=998 I0824 22:19:00.290829 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=Ϧ queue_depth=999 I0824 22:19:00.290865 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ϧ queue_depth=1000 I0824 22:19:00.290965 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=Ϩ queue_depth=1001 I0824 22:19:00.291030 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ϩ queue_depth=1002 I0824 22:19:00.291088 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=Ϫ queue_depth=1003 I0824 22:19:00.291488 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ϫ queue_depth=1004 I0824 22:19:00.291566 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=Ϭ queue_depth=1005 I0824 22:19:00.291616 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ϭ queue_depth=1006 I0824 22:19:00.291647 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=Ϯ queue_depth=1007 I0824 22:19:00.291703 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ϯ queue_depth=1008 I0824 22:19:00.291752 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ϰ queue_depth=1009 I0824 22:19:00.291819 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ϱ queue_depth=1010 I0824 22:19:00.291880 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ϲ queue_depth=1011 I0824 22:19:00.291945 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ϳ queue_depth=1012 I0824 22:19:00.292008 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ϴ queue_depth=1013 I0824 22:19:00.292100 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ϵ queue_depth=1014 I0824 22:19:00.292312 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=϶ queue_depth=1015 I0824 22:19:00.292391 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=Ϸ queue_depth=1016 I0824 22:19:00.292534 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ϸ queue_depth=1017 I0824 22:19:00.292604 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=Ϲ queue_depth=1018 I0824 22:19:00.292666 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=Ϻ queue_depth=1019 I0824 22:19:00.292761 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ϻ queue_depth=1020 I0824 22:19:00.292810 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ϼ queue_depth=1021 I0824 22:19:00.292949 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=Ͻ queue_depth=1022 I0824 22:19:00.293062 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=Ͼ queue_depth=1023 I0824 22:19:00.293119 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=Ͽ queue_depth=1024 I0824 22:19:00.293323 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=Ѐ queue_depth=1025 I0824 22:19:00.293392 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=Ё queue_depth=1026 I0824 22:19:00.293426 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=Ђ queue_depth=1027 I0824 22:19:00.293505 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=Ѓ queue_depth=1028 I0824 22:19:00.293542 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=Є queue_depth=1029 I0824 22:19:00.293594 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=Ѕ queue_depth=1030 I0824 22:19:00.293689 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=І queue_depth=1031 I0824 22:19:00.293744 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=Ї queue_depth=1032 I0824 22:19:00.293818 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=Ј queue_depth=1033 I0824 22:19:00.293896 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=Љ queue_depth=1034 I0824 22:19:00.293960 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=Њ queue_depth=1035 I0824 22:19:00.294031 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=Ћ queue_depth=1036 I0824 22:19:00.294093 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=Ќ queue_depth=1037 I0824 22:19:00.294131 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=Ѝ queue_depth=1038 I0824 22:19:00.294229 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=Ў queue_depth=1039 I0824 22:19:00.294374 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=Џ queue_depth=1040 I0824 22:19:00.294451 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=А queue_depth=1041 I0824 22:19:00.294611 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=Б queue_depth=1042 I0824 22:19:00.294912 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=В queue_depth=1043 I0824 22:19:00.295184 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=Г queue_depth=1044 I0824 22:19:00.295360 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=Д queue_depth=1045 I0824 22:19:00.295474 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=Е queue_depth=1046 I0824 22:19:00.295561 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=Ж queue_depth=1047 I0824 22:19:00.295825 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=З queue_depth=1048 I0824 22:19:00.296154 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=И queue_depth=1049 I0824 22:19:00.296395 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=Й queue_depth=1050 I0824 22:19:00.296632 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=К queue_depth=1051 I0824 22:19:00.296696 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=Л queue_depth=1052 I0824 22:19:00.296777 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=М queue_depth=1053 I0824 22:19:00.296896 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=Н queue_depth=1054 I0824 22:19:00.296964 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=О queue_depth=1055 I0824 22:19:00.297198 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=П queue_depth=1056 I0824 22:19:00.297268 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=Р queue_depth=1057 I0824 22:19:00.297366 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=С queue_depth=1058 I0824 22:19:00.297639 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=Т queue_depth=1059 I0824 22:19:00.297854 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=У queue_depth=1060 I0824 22:19:00.298048 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=Ф queue_depth=1061 I0824 22:19:00.298110 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=Х queue_depth=1062 I0824 22:19:00.298200 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=Ц queue_depth=1063 I0824 22:19:00.298390 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=Ч queue_depth=1064 I0824 22:19:00.298477 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=Ш queue_depth=1065 I0824 22:19:00.298747 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=Щ queue_depth=1066 I0824 22:19:00.299042 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=Ъ queue_depth=1067 I0824 22:19:00.299238 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=Ы queue_depth=1068 I0824 22:19:00.299345 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=Ь queue_depth=1069 I0824 22:19:00.299538 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=Э queue_depth=1070 I0824 22:19:00.299654 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=Ю queue_depth=1071 I0824 22:19:00.300119 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=Я queue_depth=1072 I0824 22:19:00.300333 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=а queue_depth=1073 I0824 22:19:00.300547 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=б queue_depth=1074 I0824 22:19:00.300719 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=в queue_depth=1075 I0824 22:19:00.300804 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=г queue_depth=1076 I0824 22:19:00.300906 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=д queue_depth=1077 I0824 22:19:00.300971 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=е queue_depth=1078 I0824 22:19:00.301067 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ж queue_depth=1079 I0824 22:19:00.301319 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=з queue_depth=1080 I0824 22:19:00.301524 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=и queue_depth=1081 I0824 22:19:00.301717 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=й queue_depth=1082 I0824 22:19:00.301938 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=к queue_depth=1083 I0824 22:19:00.302016 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=л queue_depth=1084 I0824 22:19:00.302237 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=м queue_depth=1085 I0824 22:19:00.302410 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=н queue_depth=1086 I0824 22:19:00.302622 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=о queue_depth=1087 I0824 22:19:00.302736 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=п queue_depth=1088 I0824 22:19:00.302955 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=р queue_depth=1089 I0824 22:19:00.303024 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=с queue_depth=1090 I0824 22:19:00.303108 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=т queue_depth=1091 I0824 22:19:00.303201 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=у queue_depth=1092 I0824 22:19:00.303469 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ф queue_depth=1093 I0824 22:19:00.303614 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=х queue_depth=1094 I0824 22:19:00.303804 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ц queue_depth=1095 I0824 22:19:00.303865 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ч queue_depth=1096 I0824 22:19:00.303907 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ш queue_depth=1097 I0824 22:19:00.303979 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=щ queue_depth=1098 I0824 22:19:00.304057 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ъ queue_depth=1099 I0824 22:19:00.304155 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ы queue_depth=1100 I0824 22:19:00.304240 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ь queue_depth=1101 I0824 22:19:00.304327 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=э queue_depth=1102 I0824 22:19:00.304783 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ю queue_depth=1103 I0824 22:19:00.304885 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=я queue_depth=1104 I0824 22:19:00.304958 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ѐ queue_depth=1105 I0824 22:19:00.305035 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ё queue_depth=1106 I0824 22:19:00.305114 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ђ queue_depth=1107 I0824 22:19:00.305254 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ѓ queue_depth=1108 I0824 22:19:00.305328 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=є queue_depth=1109 I0824 22:19:00.305398 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ѕ queue_depth=1110 I0824 22:19:00.305461 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=і queue_depth=1111 I0824 22:19:00.305550 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ї queue_depth=1112 I0824 22:19:00.305708 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ј queue_depth=1113 I0824 22:19:00.306134 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=љ queue_depth=1114 I0824 22:19:00.306347 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=њ queue_depth=1115 I0824 22:19:00.306448 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ћ queue_depth=1116 I0824 22:19:00.306492 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ќ queue_depth=1117 I0824 22:19:00.306543 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ѝ queue_depth=1118 I0824 22:19:00.306602 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ў queue_depth=1119 I0824 22:19:00.306634 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=џ queue_depth=1120 I0824 22:19:00.306666 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=Ѡ queue_depth=1121 I0824 22:19:00.306711 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ѡ queue_depth=1122 I0824 22:19:00.306743 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=Ѣ queue_depth=1123 I0824 22:19:00.306768 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ѣ queue_depth=1124 I0824 22:19:00.306797 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=Ѥ queue_depth=1125 I0824 22:19:00.306840 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ѥ queue_depth=1126 I0824 22:19:00.307054 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=Ѧ queue_depth=1127 I0824 22:19:00.307141 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ѧ queue_depth=1128 I0824 22:19:00.307198 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=Ѩ queue_depth=1129 I0824 22:19:00.307261 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ѩ queue_depth=1130 I0824 22:19:00.307325 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=Ѫ queue_depth=1131 I0824 22:19:00.307397 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ѫ queue_depth=1132 I0824 22:19:00.307507 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=Ѭ queue_depth=1133 I0824 22:19:00.307592 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ѭ queue_depth=1134 I0824 22:19:00.307717 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=Ѯ queue_depth=1135 I0824 22:19:00.307796 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ѯ queue_depth=1136 I0824 22:19:00.307838 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=Ѱ queue_depth=1137 I0824 22:19:00.307880 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ѱ queue_depth=1138 I0824 22:19:00.307960 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=Ѳ queue_depth=1139 I0824 22:19:00.308042 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ѳ queue_depth=1140 I0824 22:19:00.308131 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=Ѵ queue_depth=1141 I0824 22:19:00.308207 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ѵ queue_depth=1142 I0824 22:19:00.308298 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=Ѷ queue_depth=1143 I0824 22:19:00.308361 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ѷ queue_depth=1144 I0824 22:19:00.308426 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=Ѹ queue_depth=1145 I0824 22:19:00.308539 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ѹ queue_depth=1146 I0824 22:19:00.308971 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=Ѻ queue_depth=1147 I0824 22:19:00.309233 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ѻ queue_depth=1148 I0824 22:19:00.309368 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=Ѽ queue_depth=1149 I0824 22:19:00.309603 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ѽ queue_depth=1150 I0824 22:19:00.309670 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=Ѿ queue_depth=1151 I0824 22:19:00.309780 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ѿ queue_depth=1152 I0824 22:19:00.309853 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=Ҁ queue_depth=1153 I0824 22:19:00.310196 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ҁ queue_depth=1154 I0824 22:19:00.310461 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=҂ queue_depth=1155 I0824 22:19:00.310585 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=҃ queue_depth=1156 I0824 22:19:00.310760 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=҄ queue_depth=1157 I0824 22:19:00.310891 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=҅ queue_depth=1158 I0824 22:19:00.311057 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=҆ queue_depth=1159 I0824 22:19:00.311192 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=҇ queue_depth=1160 I0824 22:19:00.311432 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=҈ queue_depth=1161 I0824 22:19:00.311611 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=҉ queue_depth=1162 I0824 22:19:00.311672 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=Ҋ queue_depth=1163 I0824 22:19:00.311862 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ҋ queue_depth=1164 I0824 22:19:00.311911 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=Ҍ queue_depth=1165 I0824 22:19:00.311992 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ҍ queue_depth=1166 I0824 22:19:00.312225 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=Ҏ queue_depth=1167 I0824 22:19:00.312323 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ҏ queue_depth=1168 I0824 22:19:00.312685 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=Ґ queue_depth=1169 I0824 22:19:00.312869 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ґ queue_depth=1170 I0824 22:19:00.312951 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=Ғ queue_depth=1171 I0824 22:19:00.313111 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ғ queue_depth=1172 I0824 22:19:00.313167 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=Ҕ queue_depth=1173 I0824 22:19:00.313288 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ҕ queue_depth=1174 I0824 22:19:00.313470 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=Җ queue_depth=1175 I0824 22:19:00.313534 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=җ queue_depth=1176 I0824 22:19:00.313608 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=Ҙ queue_depth=1177 I0824 22:19:00.313680 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ҙ queue_depth=1178 I0824 22:19:00.313798 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=Қ queue_depth=1179 I0824 22:19:00.313996 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=қ queue_depth=1180 I0824 22:19:00.314255 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=Ҝ queue_depth=1181 I0824 22:19:00.314633 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ҝ queue_depth=1182 I0824 22:19:00.314718 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=Ҟ queue_depth=1183 I0824 22:19:00.314843 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ҟ queue_depth=1184 I0824 22:19:00.314914 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=Ҡ queue_depth=1185 I0824 22:19:00.315264 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ҡ queue_depth=1186 I0824 22:19:00.315458 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=Ң queue_depth=1187 I0824 22:19:00.315504 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ң queue_depth=1188 I0824 22:19:00.315534 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=Ҥ queue_depth=1189 I0824 22:19:00.315566 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ҥ queue_depth=1190 I0824 22:19:00.315593 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=Ҧ queue_depth=1191 I0824 22:19:00.315619 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ҧ queue_depth=1192 I0824 22:19:00.315643 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=Ҩ queue_depth=1193 I0824 22:19:00.315671 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ҩ queue_depth=1194 I0824 22:19:00.315825 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=Ҫ queue_depth=1195 I0824 22:19:00.315927 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ҫ queue_depth=1196 I0824 22:19:00.315978 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=Ҭ queue_depth=1197 I0824 22:19:00.316030 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ҭ queue_depth=1198 I0824 22:19:00.316145 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=Ү queue_depth=1199 I0824 22:19:00.316205 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ү queue_depth=1200 I0824 22:19:00.316281 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=Ұ queue_depth=1201 I0824 22:19:00.316371 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ұ queue_depth=1202 I0824 22:19:00.316510 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=Ҳ queue_depth=1203 I0824 22:19:00.316572 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ҳ queue_depth=1204 I0824 22:19:00.316618 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=Ҵ queue_depth=1205 I0824 22:19:00.316739 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ҵ queue_depth=1206 I0824 22:19:00.316977 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=Ҷ queue_depth=1207 I0824 22:19:00.317043 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ҷ queue_depth=1208 I0824 22:19:00.317139 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=Ҹ queue_depth=1209 I0824 22:19:00.317207 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ҹ queue_depth=1210 I0824 22:19:00.317334 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=Һ queue_depth=1211 I0824 22:19:00.317409 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=һ queue_depth=1212 I0824 22:19:00.317665 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=Ҽ queue_depth=1213 I0824 22:19:00.317935 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ҽ queue_depth=1214 I0824 22:19:00.318021 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=Ҿ queue_depth=1215 I0824 22:19:00.318086 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ҿ queue_depth=1216 I0824 22:19:00.318140 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=Ӏ queue_depth=1217 I0824 22:19:00.318190 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=Ӂ queue_depth=1218 I0824 22:19:00.318235 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ӂ queue_depth=1219 I0824 22:19:00.318281 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=Ӄ queue_depth=1220 I0824 22:19:00.318331 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ӄ queue_depth=1221 I0824 22:19:00.318401 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=Ӆ queue_depth=1222 I0824 22:19:00.318437 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ӆ queue_depth=1223 I0824 22:19:00.318481 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=Ӈ queue_depth=1224 I0824 22:19:00.318536 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ӈ queue_depth=1225 I0824 22:19:00.318679 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=Ӊ queue_depth=1226 I0824 22:19:00.318823 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ӊ queue_depth=1227 I0824 22:19:00.318895 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=Ӌ queue_depth=1228 I0824 22:19:00.318997 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ӌ queue_depth=1229 I0824 22:19:00.319085 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=Ӎ queue_depth=1230 I0824 22:19:00.319471 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ӎ queue_depth=1231 I0824 22:19:00.319673 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ӏ queue_depth=1232 I0824 22:19:00.319895 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=Ӑ queue_depth=1233 I0824 22:19:00.319973 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ӑ queue_depth=1234 I0824 22:19:00.320203 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=Ӓ queue_depth=1235 I0824 22:19:00.320563 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ӓ queue_depth=1236 I0824 22:19:00.320819 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=Ӕ queue_depth=1237 I0824 22:19:00.321014 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ӕ queue_depth=1238 I0824 22:19:00.321155 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=Ӗ queue_depth=1239 I0824 22:19:00.321409 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ӗ queue_depth=1240 I0824 22:19:00.321473 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=Ә queue_depth=1241 I0824 22:19:00.321554 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ә queue_depth=1242 I0824 22:19:00.321594 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=Ӛ queue_depth=1243 I0824 22:19:00.321620 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ӛ queue_depth=1244 I0824 22:19:00.321654 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=Ӝ queue_depth=1245 I0824 22:19:00.321782 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ӝ queue_depth=1246 I0824 22:19:00.322004 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=Ӟ queue_depth=1247 I0824 22:19:00.322082 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ӟ queue_depth=1248 I0824 22:19:00.322119 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=Ӡ queue_depth=1249 I0824 22:19:00.322147 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ӡ queue_depth=1250 I0824 22:19:00.322204 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=Ӣ queue_depth=1251 I0824 22:19:00.322378 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ӣ queue_depth=1252 I0824 22:19:00.322427 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=Ӥ queue_depth=1253 I0824 22:19:00.322521 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ӥ queue_depth=1254 I0824 22:19:00.322592 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=Ӧ queue_depth=1255 I0824 22:19:00.322667 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ӧ queue_depth=1256 I0824 22:19:00.322738 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=Ө queue_depth=1257 I0824 22:19:00.322822 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ө queue_depth=1258 I0824 22:19:00.322881 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=Ӫ queue_depth=1259 I0824 22:19:00.322930 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ӫ queue_depth=1260 I0824 22:19:00.322989 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=Ӭ queue_depth=1261 I0824 22:19:00.323146 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ӭ queue_depth=1262 I0824 22:19:00.323394 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=Ӯ queue_depth=1263 I0824 22:19:00.323508 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ӯ queue_depth=1264 I0824 22:19:00.323709 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=Ӱ queue_depth=1265 I0824 22:19:00.323821 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ӱ queue_depth=1266 I0824 22:19:00.323904 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=Ӳ queue_depth=1267 I0824 22:19:00.323946 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ӳ queue_depth=1268 I0824 22:19:00.323990 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=Ӵ queue_depth=1269 I0824 22:19:00.324046 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ӵ queue_depth=1270 I0824 22:19:00.324094 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=Ӷ queue_depth=1271 I0824 22:19:00.324139 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ӷ queue_depth=1272 I0824 22:19:00.324236 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=Ӹ queue_depth=1273 I0824 22:19:00.324370 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ӹ queue_depth=1274 I0824 22:19:00.324440 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=Ӻ queue_depth=1275 I0824 22:19:00.325599 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ӻ queue_depth=1276 I0824 22:19:00.325966 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=Ӽ queue_depth=1277 I0824 22:19:00.326114 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ӽ queue_depth=1278 I0824 22:19:00.326459 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=Ӿ queue_depth=1279 I0824 22:19:00.326630 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ӿ queue_depth=1280 I0824 22:19:00.326717 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=Ԁ queue_depth=1281 I0824 22:19:00.326778 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ԁ queue_depth=1282 I0824 22:19:00.326811 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=Ԃ queue_depth=1283 I0824 22:19:00.326837 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ԃ queue_depth=1284 I0824 22:19:00.326865 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=Ԅ queue_depth=1285 I0824 22:19:00.326962 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ԅ queue_depth=1286 I0824 22:19:00.326995 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=Ԇ queue_depth=1287 I0824 22:19:00.327032 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ԇ queue_depth=1288 I0824 22:19:00.327058 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=Ԉ queue_depth=1289 I0824 22:19:00.327110 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ԉ queue_depth=1290 I0824 22:19:00.327144 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=Ԋ queue_depth=1291 I0824 22:19:00.327224 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ԋ queue_depth=1292 I0824 22:19:00.327350 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=Ԍ queue_depth=1293 I0824 22:19:00.327395 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ԍ queue_depth=1294 I0824 22:19:00.327430 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=Ԏ queue_depth=1295 I0824 22:19:00.327523 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ԏ queue_depth=1296 I0824 22:19:00.327629 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=Ԑ queue_depth=1297 I0824 22:19:00.327833 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ԑ queue_depth=1298 I0824 22:19:00.327883 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=Ԓ queue_depth=1299 I0824 22:19:00.327940 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ԓ queue_depth=1300 I0824 22:19:00.327988 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=Ԕ queue_depth=1301 I0824 22:19:00.328198 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ԕ queue_depth=1302 I0824 22:19:00.328250 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=Ԗ queue_depth=1303 I0824 22:19:00.328325 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ԗ queue_depth=1304 I0824 22:19:00.328382 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=Ԙ queue_depth=1305 I0824 22:19:00.328561 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ԙ queue_depth=1306 I0824 22:19:00.328623 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=Ԛ queue_depth=1307 I0824 22:19:00.328678 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ԛ queue_depth=1308 I0824 22:19:00.328915 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=Ԝ queue_depth=1309 I0824 22:19:00.329094 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ԝ queue_depth=1310 I0824 22:19:00.329278 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=Ԟ queue_depth=1311 I0824 22:19:00.329558 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ԟ queue_depth=1312 I0824 22:19:00.329830 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=Ԡ queue_depth=1313 I0824 22:19:00.330015 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ԡ queue_depth=1314 I0824 22:19:00.330117 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=Ԣ queue_depth=1315 I0824 22:19:00.330178 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ԣ queue_depth=1316 I0824 22:19:00.330233 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=Ԥ queue_depth=1317 I0824 22:19:00.330754 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ԥ queue_depth=1318 I0824 22:19:00.330914 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=Ԧ queue_depth=1319 I0824 22:19:00.330999 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ԧ queue_depth=1320 I0824 22:19:00.331036 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=Ԩ queue_depth=1321 I0824 22:19:00.331080 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ԩ queue_depth=1322 I0824 22:19:00.331115 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=Ԫ queue_depth=1323 I0824 22:19:00.331146 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ԫ queue_depth=1324 I0824 22:19:00.331189 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=Ԭ queue_depth=1325 I0824 22:19:00.331250 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ԭ queue_depth=1326 I0824 22:19:00.331280 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=Ԯ queue_depth=1327 I0824 22:19:00.331313 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ԯ queue_depth=1328 I0824 22:19:00.331338 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=԰ queue_depth=1329 I0824 22:19:00.331484 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=Ա queue_depth=1330 I0824 22:19:00.331655 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=Բ queue_depth=1331 I0824 22:19:00.331692 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=Գ queue_depth=1332 I0824 22:19:00.331722 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=Դ queue_depth=1333 I0824 22:19:00.331814 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=Ե queue_depth=1334 I0824 22:19:00.331858 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=Զ queue_depth=1335 I0824 22:19:00.331890 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=Է queue_depth=1336 I0824 22:19:00.331921 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=Ը queue_depth=1337 I0824 22:19:00.331948 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=Թ queue_depth=1338 I0824 22:19:00.331973 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=Ժ queue_depth=1339 I0824 22:19:00.332026 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=Ի queue_depth=1340 I0824 22:19:00.332060 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=Լ queue_depth=1341 I0824 22:19:00.332156 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=Խ queue_depth=1342 I0824 22:19:00.332313 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=Ծ queue_depth=1343 I0824 22:19:00.332355 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=Կ queue_depth=1344 I0824 22:19:00.332425 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=Հ queue_depth=1345 I0824 22:19:00.332465 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=Ձ queue_depth=1346 I0824 22:19:00.332496 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=Ղ queue_depth=1347 I0824 22:19:00.332521 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=Ճ queue_depth=1348 I0824 22:19:00.332556 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=Մ queue_depth=1349 I0824 22:19:00.332587 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=Յ queue_depth=1350 I0824 22:19:00.332618 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=Ն queue_depth=1351 I0824 22:19:00.332666 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=Շ queue_depth=1352 I0824 22:19:00.332709 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=Ո queue_depth=1353 I0824 22:19:00.332873 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=Չ queue_depth=1354 I0824 22:19:00.332944 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=Պ queue_depth=1355 I0824 22:19:00.332980 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=Ջ queue_depth=1356 I0824 22:19:00.333008 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=Ռ queue_depth=1357 I0824 22:19:00.333039 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=Ս queue_depth=1358 I0824 22:19:00.333063 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=Վ queue_depth=1359 I0824 22:19:00.333094 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=Տ queue_depth=1360 I0824 22:19:00.333120 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=Ր queue_depth=1361 I0824 22:19:00.333150 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=Ց queue_depth=1362 I0824 22:19:00.333174 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=Ւ queue_depth=1363 I0824 22:19:00.333237 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=Փ queue_depth=1364 I0824 22:19:00.333272 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=Ք queue_depth=1365 I0824 22:19:00.333298 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=Օ queue_depth=1366 I0824 22:19:00.333332 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=Ֆ queue_depth=1367 I0824 22:19:00.333363 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=՗ queue_depth=1368 I0824 22:19:00.333391 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=՘ queue_depth=1369 I0824 22:19:00.333417 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ՙ queue_depth=1370 I0824 22:19:00.333514 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=՚ queue_depth=1371 I0824 22:19:00.333623 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=՛ queue_depth=1372 I0824 22:19:00.333662 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=՜ queue_depth=1373 I0824 22:19:00.333698 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=՝ queue_depth=1374 I0824 22:19:00.333728 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=՞ queue_depth=1375 I0824 22:19:00.333802 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=՟ queue_depth=1376 I0824 22:19:00.333869 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ՠ queue_depth=1377 I0824 22:19:00.333965 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ա queue_depth=1378 I0824 22:19:00.334088 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=բ queue_depth=1379 I0824 22:19:00.334119 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=գ queue_depth=1380 I0824 22:19:00.334201 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=դ queue_depth=1381 I0824 22:19:00.334234 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ե queue_depth=1382 I0824 22:19:00.334263 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=զ queue_depth=1383 I0824 22:19:00.334290 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=է queue_depth=1384 I0824 22:19:00.334322 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ը queue_depth=1385 I0824 22:19:00.334371 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=թ queue_depth=1386 I0824 22:19:00.334402 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ժ queue_depth=1387 I0824 22:19:00.334439 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ի queue_depth=1388 I0824 22:19:00.334465 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=լ queue_depth=1389 I0824 22:19:00.334526 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=խ queue_depth=1390 I0824 22:19:00.334557 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ծ queue_depth=1391 I0824 22:19:00.334583 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=կ queue_depth=1392 I0824 22:19:00.334611 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=հ queue_depth=1393 I0824 22:19:00.334642 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ձ queue_depth=1394 I0824 22:19:00.334672 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ղ queue_depth=1395 I0824 22:19:00.334832 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ճ queue_depth=1396 I0824 22:19:00.334969 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=մ queue_depth=1397 I0824 22:19:00.335321 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=յ queue_depth=1398 I0824 22:19:00.335630 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ն queue_depth=1399 I0824 22:19:00.335965 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=շ queue_depth=1400 I0824 22:19:00.336140 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ո queue_depth=1401 I0824 22:19:00.336201 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=չ queue_depth=1402 I0824 22:19:00.336310 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=պ queue_depth=1403 I0824 22:19:00.336449 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ջ queue_depth=1404 I0824 22:19:00.336508 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ռ queue_depth=1405 I0824 22:19:00.336578 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ս queue_depth=1406 I0824 22:19:00.336679 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=վ queue_depth=1407 I0824 22:19:00.336710 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=տ queue_depth=1408 I0824 22:19:00.336766 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ր queue_depth=1409 I0824 22:19:00.336799 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ց queue_depth=1410 I0824 22:19:00.336950 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ւ queue_depth=1411 I0824 22:19:00.337072 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=փ queue_depth=1412 I0824 22:19:00.337104 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ք queue_depth=1413 I0824 22:19:00.337130 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=օ queue_depth=1414 I0824 22:19:00.337194 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ֆ queue_depth=1415 I0824 22:19:00.337239 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=և queue_depth=1416 I0824 22:19:00.337315 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ֈ queue_depth=1417 I0824 22:19:00.337353 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=։ queue_depth=1418 I0824 22:19:00.337380 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=֊ queue_depth=1419 I0824 22:19:00.337441 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=֋ queue_depth=1420 I0824 22:19:00.337632 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=֌ queue_depth=1421 I0824 22:19:00.337672 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=֍ queue_depth=1422 I0824 22:19:00.337720 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=֎ queue_depth=1423 I0824 22:19:00.337748 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=֏ queue_depth=1424 I0824 22:19:00.337776 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=֐ queue_depth=1425 I0824 22:19:00.337807 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=֑ queue_depth=1426 I0824 22:19:00.337834 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=֒ queue_depth=1427 I0824 22:19:00.337865 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=֓ queue_depth=1428 I0824 22:19:00.337889 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=֔ queue_depth=1429 I0824 22:19:00.337919 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=֕ queue_depth=1430 I0824 22:19:00.337964 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=֖ queue_depth=1431 I0824 22:19:00.338038 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=֗ queue_depth=1432 I0824 22:19:00.338071 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=֘ queue_depth=1433 I0824 22:19:00.338097 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=֙ queue_depth=1434 I0824 22:19:00.338170 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=֚ queue_depth=1435 I0824 22:19:00.338205 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=֛ queue_depth=1436 I0824 22:19:00.338244 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=֜ queue_depth=1437 I0824 22:19:00.338269 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=֝ queue_depth=1438 I0824 22:19:00.338296 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=֞ queue_depth=1439 I0824 22:19:00.338329 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=֟ queue_depth=1440 I0824 22:19:00.338352 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=֠ queue_depth=1441 I0824 22:19:00.338405 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=֡ queue_depth=1442 I0824 22:19:00.338432 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=֢ queue_depth=1443 I0824 22:19:00.338590 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=֣ queue_depth=1444 I0824 22:19:00.338732 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=֤ queue_depth=1445 I0824 22:19:00.338773 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=֥ queue_depth=1446 I0824 22:19:00.338801 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=֦ queue_depth=1447 I0824 22:19:00.338885 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=֧ queue_depth=1448 I0824 22:19:00.338963 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=֨ queue_depth=1449 I0824 22:19:00.339092 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=֩ queue_depth=1450 I0824 22:19:00.339138 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=֪ queue_depth=1451 I0824 22:19:00.339202 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=֫ queue_depth=1452 I0824 22:19:00.339246 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=֬ queue_depth=1453 I0824 22:19:00.339288 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=֭ queue_depth=1454 I0824 22:19:00.339317 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=֮ queue_depth=1455 I0824 22:19:00.339345 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=֯ queue_depth=1456 I0824 22:19:00.339382 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ְ queue_depth=1457 I0824 22:19:00.339407 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ֱ queue_depth=1458 I0824 22:19:00.339432 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ֲ queue_depth=1459 I0824 22:19:00.339464 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ֳ queue_depth=1460 I0824 22:19:00.339488 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ִ queue_depth=1461 I0824 22:19:00.339587 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ֵ queue_depth=1462 I0824 22:19:00.339639 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ֶ queue_depth=1463 I0824 22:19:00.339671 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ַ queue_depth=1464 I0824 22:19:00.339750 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ָ queue_depth=1465 I0824 22:19:00.339950 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ֹ queue_depth=1466 I0824 22:19:00.340005 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ֺ queue_depth=1467 I0824 22:19:00.340054 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ֻ queue_depth=1468 I0824 22:19:00.340236 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ּ queue_depth=1469 I0824 22:19:00.340306 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ֽ queue_depth=1470 I0824 22:19:00.340417 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=־ queue_depth=1471 I0824 22:19:00.340658 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ֿ queue_depth=1472 I0824 22:19:00.340856 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=׀ queue_depth=1473 I0824 22:19:00.341020 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ׁ queue_depth=1474 I0824 22:19:00.341086 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ׂ queue_depth=1475 I0824 22:19:00.341139 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=׃ queue_depth=1476 I0824 22:19:00.341424 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ׄ queue_depth=1477 I0824 22:19:00.341714 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ׅ queue_depth=1478 I0824 22:19:00.341808 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=׆ queue_depth=1479 I0824 22:19:00.342038 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ׇ queue_depth=1480 I0824 22:19:00.342250 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=׈ queue_depth=1481 I0824 22:19:00.342439 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=׉ queue_depth=1482 I0824 22:19:00.342621 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=׊ queue_depth=1483 I0824 22:19:00.342687 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=׋ queue_depth=1484 I0824 22:19:00.342728 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=׌ queue_depth=1485 I0824 22:19:00.342761 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=׍ queue_depth=1486 I0824 22:19:00.342796 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=׎ queue_depth=1487 I0824 22:19:00.342841 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=׏ queue_depth=1488 I0824 22:19:00.342874 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=א queue_depth=1489 I0824 22:19:00.342962 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ב queue_depth=1490 I0824 22:19:00.343114 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ג queue_depth=1491 I0824 22:19:00.343154 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ד queue_depth=1492 I0824 22:19:00.343198 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ה queue_depth=1493 I0824 22:19:00.343259 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ו queue_depth=1494 I0824 22:19:00.343294 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ז queue_depth=1495 I0824 22:19:00.343322 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ח queue_depth=1496 I0824 22:19:00.343356 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ט queue_depth=1497 I0824 22:19:00.343388 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=י queue_depth=1498 I0824 22:19:00.343513 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ך queue_depth=1499 I0824 22:19:00.343802 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=כ queue_depth=1500 I0824 22:19:00.343921 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ל queue_depth=1501 I0824 22:19:00.343956 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ם queue_depth=1502 I0824 22:19:00.343987 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=מ queue_depth=1503 I0824 22:19:00.344017 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ן queue_depth=1504 I0824 22:19:00.344063 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=נ queue_depth=1505 I0824 22:19:00.344117 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ס queue_depth=1506 I0824 22:19:00.344148 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ע queue_depth=1507 I0824 22:19:00.344176 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ף queue_depth=1508 I0824 22:19:00.344368 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=פ queue_depth=1509 I0824 22:19:00.344781 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ץ queue_depth=1510 I0824 22:19:00.345038 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=צ queue_depth=1511 I0824 22:19:00.345257 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ק queue_depth=1512 I0824 22:19:00.345446 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ר queue_depth=1513 I0824 22:19:00.345699 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ש queue_depth=1514 I0824 22:19:00.345856 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ת queue_depth=1515 I0824 22:19:00.345926 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=׫ queue_depth=1516 I0824 22:19:00.345984 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=׬ queue_depth=1517 I0824 22:19:00.346067 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=׭ queue_depth=1518 I0824 22:19:00.346120 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=׮ queue_depth=1519 I0824 22:19:00.346171 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ׯ queue_depth=1520 I0824 22:19:00.346333 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=װ queue_depth=1521 I0824 22:19:00.346707 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ױ queue_depth=1522 I0824 22:19:00.346973 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ײ queue_depth=1523 I0824 22:19:00.347168 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=׳ queue_depth=1524 I0824 22:19:00.347230 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=״ queue_depth=1525 I0824 22:19:00.347329 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=׵ queue_depth=1526 I0824 22:19:00.347555 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=׶ queue_depth=1527 I0824 22:19:00.347617 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=׷ queue_depth=1528 I0824 22:19:00.347805 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=׸ queue_depth=1529 I0824 22:19:00.347873 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=׹ queue_depth=1530 I0824 22:19:00.347924 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=׺ queue_depth=1531 I0824 22:19:00.347978 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=׻ queue_depth=1532 I0824 22:19:00.348202 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=׼ queue_depth=1533 I0824 22:19:00.348385 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=׽ queue_depth=1534 I0824 22:19:00.348565 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=׾ queue_depth=1535 I0824 22:19:00.348629 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=׿ queue_depth=1536 I0824 22:19:00.348685 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=؀ queue_depth=1537 I0824 22:19:00.348781 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=؁ queue_depth=1538 I0824 22:19:00.348949 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=؂ queue_depth=1539 I0824 22:19:00.349009 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=؃ queue_depth=1540 I0824 22:19:00.349079 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=؄ queue_depth=1541 I0824 22:19:00.349149 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=؅ queue_depth=1542 I0824 22:19:00.349335 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=؆ queue_depth=1543 I0824 22:19:00.349545 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=؇ queue_depth=1544 I0824 22:19:00.349683 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=؈ queue_depth=1545 I0824 22:19:00.349743 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=؉ queue_depth=1546 I0824 22:19:00.349780 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=؊ queue_depth=1547 I0824 22:19:00.349822 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=؋ queue_depth=1548 I0824 22:19:00.349853 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=، queue_depth=1549 I0824 22:19:00.349887 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=؍ queue_depth=1550 I0824 22:19:00.349921 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=؎ queue_depth=1551 I0824 22:19:00.349954 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=؏ queue_depth=1552 I0824 22:19:00.349986 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ؐ queue_depth=1553 I0824 22:19:00.350012 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ؑ queue_depth=1554 I0824 22:19:00.350043 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ؒ queue_depth=1555 I0824 22:19:00.350090 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ؓ queue_depth=1556 I0824 22:19:00.350125 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ؔ queue_depth=1557 I0824 22:19:00.350151 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ؕ queue_depth=1558 I0824 22:19:00.350193 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ؖ queue_depth=1559 I0824 22:19:00.350222 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ؗ queue_depth=1560 I0824 22:19:00.350370 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ؘ queue_depth=1561 I0824 22:19:00.350510 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ؙ queue_depth=1562 I0824 22:19:00.350548 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ؚ queue_depth=1563 I0824 22:19:00.350635 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=؛ queue_depth=1564 I0824 22:19:00.350758 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=؜ queue_depth=1565 I0824 22:19:00.350802 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=؝ queue_depth=1566 I0824 22:19:00.350852 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=؞ queue_depth=1567 I0824 22:19:00.350881 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=؟ queue_depth=1568 I0824 22:19:00.350906 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ؠ queue_depth=1569 I0824 22:19:00.350933 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ء queue_depth=1570 I0824 22:19:00.350983 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=آ queue_depth=1571 I0824 22:19:00.351010 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=أ queue_depth=1572 I0824 22:19:00.351035 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ؤ queue_depth=1573 I0824 22:19:00.351060 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=إ queue_depth=1574 I0824 22:19:00.351085 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ئ queue_depth=1575 I0824 22:19:00.351206 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ا queue_depth=1576 I0824 22:19:00.351255 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ب queue_depth=1577 I0824 22:19:00.351291 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ة queue_depth=1578 I0824 22:19:00.351322 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ت queue_depth=1579 I0824 22:19:00.351350 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ث queue_depth=1580 I0824 22:19:00.351376 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ج queue_depth=1581 I0824 22:19:00.351469 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ح queue_depth=1582 I0824 22:19:00.351502 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=خ queue_depth=1583 I0824 22:19:00.351527 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=د queue_depth=1584 I0824 22:19:00.351553 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ذ queue_depth=1585 I0824 22:19:00.351598 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ر queue_depth=1586 I0824 22:19:00.351631 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ز queue_depth=1587 I0824 22:19:00.351677 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=س queue_depth=1588 I0824 22:19:00.351705 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ش queue_depth=1589 I0824 22:19:00.351733 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ص queue_depth=1590 I0824 22:19:00.351759 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ض queue_depth=1591 I0824 22:19:00.351785 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ط queue_depth=1592 I0824 22:19:00.351811 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ظ queue_depth=1593 I0824 22:19:00.351836 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ع queue_depth=1594 I0824 22:19:00.351860 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=غ queue_depth=1595 I0824 22:19:00.351884 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ػ queue_depth=1596 I0824 22:19:00.351910 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ؼ queue_depth=1597 I0824 22:19:00.351935 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ؽ queue_depth=1598 I0824 22:19:00.351989 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ؾ queue_depth=1599 I0824 22:19:00.352020 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ؿ queue_depth=1600 I0824 22:19:00.352087 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ـ queue_depth=1601 I0824 22:19:00.352119 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ف queue_depth=1602 I0824 22:19:00.352146 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ق queue_depth=1603 I0824 22:19:00.352171 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ك queue_depth=1604 I0824 22:19:00.352196 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ل queue_depth=1605 I0824 22:19:00.352221 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=م queue_depth=1606 I0824 22:19:00.352246 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ن queue_depth=1607 I0824 22:19:00.352292 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ه queue_depth=1608 I0824 22:19:00.352374 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=و queue_depth=1609 I0824 22:19:00.352409 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ى queue_depth=1610 I0824 22:19:00.352438 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ي queue_depth=1611 I0824 22:19:00.352464 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ً queue_depth=1612 I0824 22:19:00.352488 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ٌ queue_depth=1613 I0824 22:19:00.352513 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ٍ queue_depth=1614 I0824 22:19:00.352537 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=َ queue_depth=1615 I0824 22:19:00.352565 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ُ queue_depth=1616 I0824 22:19:00.352590 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ِ queue_depth=1617 I0824 22:19:00.352614 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ّ queue_depth=1618 I0824 22:19:00.352639 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ْ queue_depth=1619 I0824 22:19:00.352684 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ٓ queue_depth=1620 I0824 22:19:00.352709 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ٔ queue_depth=1621 I0824 22:19:00.352733 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ٕ queue_depth=1622 I0824 22:19:00.352759 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ٖ queue_depth=1623 I0824 22:19:00.352783 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ٗ queue_depth=1624 I0824 22:19:00.352806 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=٘ queue_depth=1625 I0824 22:19:00.352831 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ٙ queue_depth=1626 I0824 22:19:00.352856 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ٚ queue_depth=1627 I0824 22:19:00.352901 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ٛ queue_depth=1628 I0824 22:19:00.352926 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ٜ queue_depth=1629 I0824 22:19:00.352952 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ٝ queue_depth=1630 I0824 22:19:00.352995 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ٞ queue_depth=1631 I0824 22:19:00.353039 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ٟ queue_depth=1632 I0824 22:19:00.353073 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=٠ queue_depth=1633 I0824 22:19:00.353105 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=١ queue_depth=1634 I0824 22:19:00.353156 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=٢ queue_depth=1635 I0824 22:19:00.353181 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=٣ queue_depth=1636 I0824 22:19:00.353212 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=٤ queue_depth=1637 I0824 22:19:00.353245 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=٥ queue_depth=1638 I0824 22:19:00.353278 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=٦ queue_depth=1639 I0824 22:19:00.353686 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=٧ queue_depth=1640 I0824 22:19:00.353816 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=٨ queue_depth=1641 I0824 22:19:00.353982 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=٩ queue_depth=1642 I0824 22:19:00.354314 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=٪ queue_depth=1643 I0824 22:19:00.354568 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=٫ queue_depth=1644 I0824 22:19:00.354759 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=٬ queue_depth=1645 I0824 22:19:00.355002 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=٭ queue_depth=1646 I0824 22:19:00.355188 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ٮ queue_depth=1647 I0824 22:19:00.355430 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ٯ queue_depth=1648 I0824 22:19:00.355615 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ٰ queue_depth=1649 I0824 22:19:00.355682 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ٱ queue_depth=1650 I0824 22:19:00.355763 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ٲ queue_depth=1651 I0824 22:19:00.355917 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ٳ queue_depth=1652 I0824 22:19:00.355982 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ٴ queue_depth=1653 I0824 22:19:00.356078 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ٵ queue_depth=1654 I0824 22:19:00.356200 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ٶ queue_depth=1655 I0824 22:19:00.356374 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ٷ queue_depth=1656 I0824 22:19:00.356491 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ٸ queue_depth=1657 I0824 22:19:00.356693 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ٹ queue_depth=1658 I0824 22:19:00.356840 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ٺ queue_depth=1659 I0824 22:19:00.356931 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ٻ queue_depth=1660 I0824 22:19:00.356987 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ټ queue_depth=1661 I0824 22:19:00.357040 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ٽ queue_depth=1662 I0824 22:19:00.357249 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=پ queue_depth=1663 I0824 22:19:00.357305 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ٿ queue_depth=1664 I0824 22:19:00.357607 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ڀ queue_depth=1665 I0824 22:19:00.357793 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ځ queue_depth=1666 I0824 22:19:00.357841 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ڂ queue_depth=1667 I0824 22:19:00.357921 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ڃ queue_depth=1668 I0824 22:19:00.358008 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ڄ queue_depth=1669 I0824 22:19:00.358128 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=څ queue_depth=1670 I0824 22:19:00.358165 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=چ queue_depth=1671 I0824 22:19:00.358194 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ڇ queue_depth=1672 I0824 22:19:00.358226 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ڈ queue_depth=1673 I0824 22:19:00.358254 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ډ queue_depth=1674 I0824 22:19:00.358433 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ڊ queue_depth=1675 I0824 22:19:00.358632 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ڋ queue_depth=1676 I0824 22:19:00.358815 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ڌ queue_depth=1677 I0824 22:19:00.358965 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ڍ queue_depth=1678 I0824 22:19:00.359025 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ڎ queue_depth=1679 I0824 22:19:00.359107 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ڏ queue_depth=1680 I0824 22:19:00.359279 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ڐ queue_depth=1681 I0824 22:19:00.359452 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ڑ queue_depth=1682 I0824 22:19:00.359537 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ڒ queue_depth=1683 I0824 22:19:00.359599 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ړ queue_depth=1684 I0824 22:19:00.359657 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ڔ queue_depth=1685 I0824 22:19:00.359714 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ڕ queue_depth=1686 I0824 22:19:00.359771 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ږ queue_depth=1687 I0824 22:19:00.359824 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ڗ queue_depth=1688 I0824 22:19:00.359911 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ژ queue_depth=1689 I0824 22:19:00.359957 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ڙ queue_depth=1690 I0824 22:19:00.359995 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ښ queue_depth=1691 I0824 22:19:00.360068 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ڛ queue_depth=1692 I0824 22:19:00.360256 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ڜ queue_depth=1693 I0824 22:19:00.360382 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ڝ queue_depth=1694 I0824 22:19:00.360432 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ڞ queue_depth=1695 I0824 22:19:00.360487 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ڟ queue_depth=1696 I0824 22:19:00.360522 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ڠ queue_depth=1697 I0824 22:19:00.360553 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ڡ queue_depth=1698 I0824 22:19:00.360580 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ڢ queue_depth=1699 I0824 22:19:00.360624 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ڣ queue_depth=1700 I0824 22:19:00.361506 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ڤ queue_depth=1701 I0824 22:19:00.361826 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ڥ queue_depth=1702 I0824 22:19:00.361954 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ڦ queue_depth=1703 I0824 22:19:00.362024 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ڧ queue_depth=1704 I0824 22:19:00.362071 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ڨ queue_depth=1705 I0824 22:19:00.362107 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ک queue_depth=1706 I0824 22:19:00.362146 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ڪ queue_depth=1707 I0824 22:19:00.362297 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ګ queue_depth=1708 I0824 22:19:00.362431 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ڬ queue_depth=1709 I0824 22:19:00.362469 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ڭ queue_depth=1710 I0824 22:19:00.362503 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ڮ queue_depth=1711 I0824 22:19:00.362537 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=گ queue_depth=1712 I0824 22:19:00.362565 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ڰ queue_depth=1713 I0824 22:19:00.362597 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ڱ queue_depth=1714 I0824 22:19:00.362642 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ڲ queue_depth=1715 I0824 22:19:00.362675 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ڳ queue_depth=1716 I0824 22:19:00.362701 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ڴ queue_depth=1717 I0824 22:19:00.362732 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ڵ queue_depth=1718 I0824 22:19:00.362756 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ڶ queue_depth=1719 I0824 22:19:00.362787 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ڷ queue_depth=1720 I0824 22:19:00.362811 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ڸ queue_depth=1721 I0824 22:19:00.362875 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ڹ queue_depth=1722 I0824 22:19:00.362907 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ں queue_depth=1723 I0824 22:19:00.362958 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ڻ queue_depth=1724 I0824 22:19:00.362992 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ڼ queue_depth=1725 I0824 22:19:00.363071 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ڽ queue_depth=1726 I0824 22:19:00.363115 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ھ queue_depth=1727 I0824 22:19:00.363184 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ڿ queue_depth=1728 I0824 22:19:00.363221 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ۀ queue_depth=1729 I0824 22:19:00.363262 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ہ queue_depth=1730 I0824 22:19:00.363317 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ۂ queue_depth=1731 I0824 22:19:00.363349 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ۃ queue_depth=1732 I0824 22:19:00.363376 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ۄ queue_depth=1733 I0824 22:19:00.363408 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ۅ queue_depth=1734 I0824 22:19:00.363433 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ۆ queue_depth=1735 I0824 22:19:00.363512 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ۇ queue_depth=1736 I0824 22:19:00.363636 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ۈ queue_depth=1737 I0824 22:19:00.363669 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ۉ queue_depth=1738 I0824 22:19:00.363701 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ۊ queue_depth=1739 I0824 22:19:00.363726 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ۋ queue_depth=1740 I0824 22:19:00.363779 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ی queue_depth=1741 I0824 22:19:00.363873 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ۍ queue_depth=1742 I0824 22:19:00.363932 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ێ queue_depth=1743 I0824 22:19:00.363968 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ۏ queue_depth=1744 I0824 22:19:00.363995 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ې queue_depth=1745 I0824 22:19:00.364271 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ۑ queue_depth=1746 I0824 22:19:00.364520 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ے queue_depth=1747 I0824 22:19:00.364812 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ۓ queue_depth=1748 I0824 22:19:00.365057 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=۔ queue_depth=1749 I0824 22:19:00.365289 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ە queue_depth=1750 I0824 22:19:00.365479 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ۖ queue_depth=1751 I0824 22:19:00.365584 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ۗ queue_depth=1752 I0824 22:19:00.365766 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ۘ queue_depth=1753 I0824 22:19:00.365856 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ۙ queue_depth=1754 I0824 22:19:00.366019 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ۚ queue_depth=1755 I0824 22:19:00.366076 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ۛ queue_depth=1756 I0824 22:19:00.366125 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ۜ queue_depth=1757 I0824 22:19:00.366171 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=۝ queue_depth=1758 I0824 22:19:00.366204 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=۞ queue_depth=1759 I0824 22:19:00.366242 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=۟ queue_depth=1760 I0824 22:19:00.366269 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=۠ queue_depth=1761 I0824 22:19:00.366337 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ۡ queue_depth=1762 I0824 22:19:00.366373 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ۢ queue_depth=1763 I0824 22:19:00.366411 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ۣ queue_depth=1764 I0824 22:19:00.366442 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ۤ queue_depth=1765 I0824 22:19:00.366497 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ۥ queue_depth=1766 I0824 22:19:00.366533 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ۦ queue_depth=1767 I0824 22:19:00.366562 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ۧ queue_depth=1768 I0824 22:19:00.366595 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ۨ queue_depth=1769 I0824 22:19:00.366740 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=۩ queue_depth=1770 I0824 22:19:00.366864 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=۪ queue_depth=1771 I0824 22:19:00.366899 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=۫ queue_depth=1772 I0824 22:19:00.366926 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=۬ queue_depth=1773 I0824 22:19:00.367044 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ۭ queue_depth=1774 I0824 22:19:00.367197 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ۮ queue_depth=1775 I0824 22:19:00.367244 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ۯ queue_depth=1776 I0824 22:19:00.367293 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=۰ queue_depth=1777 I0824 22:19:00.367350 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=۱ queue_depth=1778 I0824 22:19:00.367403 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=۲ queue_depth=1779 I0824 22:19:00.367460 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=۳ queue_depth=1780 I0824 22:19:00.367519 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=۴ queue_depth=1781 I0824 22:19:00.367716 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=۵ queue_depth=1782 I0824 22:19:00.367770 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=۶ queue_depth=1783 I0824 22:19:00.367821 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=۷ queue_depth=1784 I0824 22:19:00.367872 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=۸ queue_depth=1785 I0824 22:19:00.367926 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=۹ queue_depth=1786 I0824 22:19:00.368013 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ۺ queue_depth=1787 I0824 22:19:00.368074 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ۻ queue_depth=1788 I0824 22:19:00.368166 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ۼ queue_depth=1789 I0824 22:19:00.368364 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=۽ queue_depth=1790 I0824 22:19:00.368512 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=۾ queue_depth=1791 I0824 22:19:00.368581 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ۿ queue_depth=1792 I0824 22:19:00.368644 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=܀ queue_depth=1793 I0824 22:19:00.368697 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=܁ queue_depth=1794 I0824 22:19:00.368755 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=܂ queue_depth=1795 I0824 22:19:00.368808 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=܃ queue_depth=1796 I0824 22:19:00.368900 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=܄ queue_depth=1797 I0824 22:19:00.369102 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=܅ queue_depth=1798 I0824 22:19:00.369286 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=܆ queue_depth=1799 I0824 22:19:00.369350 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=܇ queue_depth=1800 I0824 22:19:00.369432 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=܈ queue_depth=1801 I0824 22:19:00.369577 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=܉ queue_depth=1802 I0824 22:19:00.369622 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=܊ queue_depth=1803 I0824 22:19:00.369821 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=܋ queue_depth=1804 I0824 22:19:00.370039 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=܌ queue_depth=1805 I0824 22:19:00.370186 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=܍ queue_depth=1806 I0824 22:19:00.370472 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=܎ queue_depth=1807 I0824 22:19:00.370650 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=܏ queue_depth=1808 I0824 22:19:00.370696 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ܐ queue_depth=1809 I0824 22:19:00.370786 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ܑ queue_depth=1810 I0824 22:19:00.370944 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ܒ queue_depth=1811 I0824 22:19:00.371053 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ܓ queue_depth=1812 I0824 22:19:00.371090 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ܔ queue_depth=1813 I0824 22:19:00.371119 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ܕ queue_depth=1814 I0824 22:19:00.371150 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ܖ queue_depth=1815 I0824 22:19:00.371199 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ܗ queue_depth=1816 I0824 22:19:00.371235 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ܘ queue_depth=1817 I0824 22:19:00.371268 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ܙ queue_depth=1818 I0824 22:19:00.371292 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ܚ queue_depth=1819 I0824 22:19:00.371319 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ܛ queue_depth=1820 I0824 22:19:00.371351 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ܜ queue_depth=1821 I0824 22:19:00.371376 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ܝ queue_depth=1822 I0824 22:19:00.371654 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ܞ queue_depth=1823 I0824 22:19:00.371804 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ܟ queue_depth=1824 I0824 22:19:00.371955 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ܠ queue_depth=1825 I0824 22:19:00.371996 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ܡ queue_depth=1826 I0824 22:19:00.372028 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ܢ queue_depth=1827 I0824 22:19:00.372057 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ܣ queue_depth=1828 I0824 22:19:00.372090 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ܤ queue_depth=1829 I0824 22:19:00.372175 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ܥ queue_depth=1830 I0824 22:19:00.372291 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ܦ queue_depth=1831 I0824 22:19:00.372353 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ܧ queue_depth=1832 I0824 22:19:00.372448 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ܨ queue_depth=1833 I0824 22:19:00.372501 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ܩ queue_depth=1834 I0824 22:19:00.372530 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ܪ queue_depth=1835 I0824 22:19:00.372564 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ܫ queue_depth=1836 I0824 22:19:00.372595 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ܬ queue_depth=1837 I0824 22:19:00.372654 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ܭ queue_depth=1838 I0824 22:19:00.372689 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ܮ queue_depth=1839 I0824 22:19:00.372770 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ܯ queue_depth=1840 I0824 22:19:00.372883 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ܰ queue_depth=1841 I0824 22:19:00.372919 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ܱ queue_depth=1842 I0824 22:19:00.372946 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ܲ queue_depth=1843 I0824 22:19:00.372976 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ܳ queue_depth=1844 I0824 22:19:00.373007 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ܴ queue_depth=1845 I0824 22:19:00.373033 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ܵ queue_depth=1846 I0824 22:19:00.373063 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ܶ queue_depth=1847 I0824 22:19:00.373089 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ܷ queue_depth=1848 I0824 22:19:00.373148 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ܸ queue_depth=1849 I0824 22:19:00.373197 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ܹ queue_depth=1850 I0824 22:19:00.373228 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ܺ queue_depth=1851 I0824 22:19:00.373254 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ܻ queue_depth=1852 I0824 22:19:00.373287 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ܼ queue_depth=1853 I0824 22:19:00.373311 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ܽ queue_depth=1854 I0824 22:19:00.373343 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ܾ queue_depth=1855 I0824 22:19:00.373370 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ܿ queue_depth=1856 I0824 22:19:00.373399 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=݀ queue_depth=1857 I0824 22:19:00.373497 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=݁ queue_depth=1858 I0824 22:19:00.373610 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=݂ queue_depth=1859 I0824 22:19:00.373652 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=݃ queue_depth=1860 I0824 22:19:00.373699 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=݄ queue_depth=1861 I0824 22:19:00.373732 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=݅ queue_depth=1862 I0824 22:19:00.373758 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=݆ queue_depth=1863 I0824 22:19:00.373808 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=݇ queue_depth=1864 I0824 22:19:00.373834 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=݈ queue_depth=1865 I0824 22:19:00.373870 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=݉ queue_depth=1866 I0824 22:19:00.373902 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=݊ queue_depth=1867 I0824 22:19:00.373929 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=݋ queue_depth=1868 I0824 22:19:00.373960 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=݌ queue_depth=1869 I0824 22:19:00.373986 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ݍ queue_depth=1870 I0824 22:19:00.374017 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ݎ queue_depth=1871 I0824 22:19:00.374067 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ݏ queue_depth=1872 I0824 22:19:00.374479 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ݐ queue_depth=1873 I0824 22:19:00.374787 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ݑ queue_depth=1874 I0824 22:19:00.375020 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ݒ queue_depth=1875 I0824 22:19:00.375200 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ݓ queue_depth=1876 I0824 22:19:00.375266 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ݔ queue_depth=1877 I0824 22:19:00.375325 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ݕ queue_depth=1878 I0824 22:19:00.375412 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ݖ queue_depth=1879 I0824 22:19:00.375506 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ݗ queue_depth=1880 I0824 22:19:00.375701 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ݘ queue_depth=1881 I0824 22:19:00.375768 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ݙ queue_depth=1882 I0824 22:19:00.375832 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ݚ queue_depth=1883 I0824 22:19:00.375921 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ݛ queue_depth=1884 I0824 22:19:00.375980 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ݜ queue_depth=1885 I0824 22:19:00.376035 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ݝ queue_depth=1886 I0824 22:19:00.376088 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ݞ queue_depth=1887 I0824 22:19:00.376169 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ݟ queue_depth=1888 I0824 22:19:00.376241 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ݠ queue_depth=1889 I0824 22:19:00.376319 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ݡ queue_depth=1890 I0824 22:19:00.376586 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ݢ queue_depth=1891 I0824 22:19:00.376984 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ݣ queue_depth=1892 I0824 22:19:00.377205 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ݤ queue_depth=1893 I0824 22:19:00.377287 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ݥ queue_depth=1894 I0824 22:19:00.377491 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ݦ queue_depth=1895 I0824 22:19:00.377658 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ݧ queue_depth=1896 I0824 22:19:00.377723 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ݨ queue_depth=1897 I0824 22:19:00.377781 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ݩ queue_depth=1898 I0824 22:19:00.377867 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ݪ queue_depth=1899 I0824 22:19:00.377933 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ݫ queue_depth=1900 I0824 22:19:00.378130 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ݬ queue_depth=1901 I0824 22:19:00.378304 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ݭ queue_depth=1902 I0824 22:19:00.378388 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ݮ queue_depth=1903 I0824 22:19:00.378541 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ݯ queue_depth=1904 I0824 22:19:00.378611 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ݰ queue_depth=1905 I0824 22:19:00.378907 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ݱ queue_depth=1906 I0824 22:19:00.379125 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ݲ queue_depth=1907 I0824 22:19:00.379320 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ݳ queue_depth=1908 I0824 22:19:00.379502 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ݴ queue_depth=1909 I0824 22:19:00.379654 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ݵ queue_depth=1910 I0824 22:19:00.379801 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ݶ queue_depth=1911 I0824 22:19:00.379860 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ݷ queue_depth=1912 I0824 22:19:00.380013 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ݸ queue_depth=1913 I0824 22:19:00.380192 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ݹ queue_depth=1914 I0824 22:19:00.380234 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ݺ queue_depth=1915 I0824 22:19:00.380273 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ݻ queue_depth=1916 I0824 22:19:00.380313 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ݼ queue_depth=1917 I0824 22:19:00.380351 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ݽ queue_depth=1918 I0824 22:19:00.380388 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ݾ queue_depth=1919 I0824 22:19:00.380416 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ݿ queue_depth=1920 I0824 22:19:00.380450 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ހ queue_depth=1921 I0824 22:19:00.380480 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ށ queue_depth=1922 I0824 22:19:00.380516 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ނ queue_depth=1923 I0824 22:19:00.381165 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ރ queue_depth=1924 I0824 22:19:00.381319 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ބ queue_depth=1925 I0824 22:19:00.381388 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ޅ queue_depth=1926 I0824 22:19:00.381425 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ކ queue_depth=1927 I0824 22:19:00.381462 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=އ queue_depth=1928 I0824 22:19:00.381518 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ވ queue_depth=1929 I0824 22:19:00.381551 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=މ queue_depth=1930 I0824 22:19:00.381577 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ފ queue_depth=1931 I0824 22:19:00.381608 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ދ queue_depth=1932 I0824 22:19:00.381635 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ތ queue_depth=1933 I0824 22:19:00.381667 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ލ queue_depth=1934 I0824 22:19:00.381694 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ގ queue_depth=1935 I0824 22:19:00.381725 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ޏ queue_depth=1936 I0824 22:19:00.381749 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ސ queue_depth=1937 I0824 22:19:00.381781 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ޑ queue_depth=1938 I0824 22:19:00.383411 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ޒ queue_depth=1939 I0824 22:19:00.383669 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ޓ queue_depth=1940 I0824 22:19:00.383727 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ޔ queue_depth=1941 I0824 22:19:00.383766 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ޕ queue_depth=1942 I0824 22:19:00.383819 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ޖ queue_depth=1943 I0824 22:19:00.383854 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ޗ queue_depth=1944 I0824 22:19:00.383929 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ޘ queue_depth=1945 I0824 22:19:00.383972 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ޙ queue_depth=1946 I0824 22:19:00.384001 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ޚ queue_depth=1947 I0824 22:19:00.384387 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ޛ queue_depth=1948 I0824 22:19:00.384518 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ޜ queue_depth=1949 I0824 22:19:00.384577 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ޝ queue_depth=1950 I0824 22:19:00.384606 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ޞ queue_depth=1951 I0824 22:19:00.384637 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ޟ queue_depth=1952 I0824 22:19:00.384669 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ޠ queue_depth=1953 I0824 22:19:00.384708 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ޡ queue_depth=1954 I0824 22:19:00.384977 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ޢ queue_depth=1955 I0824 22:19:00.385297 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ޣ queue_depth=1956 I0824 22:19:00.385592 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ޤ queue_depth=1957 I0824 22:19:00.385863 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ޥ queue_depth=1958 I0824 22:19:00.386100 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ަ queue_depth=1959 I0824 22:19:00.386381 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ާ queue_depth=1960 I0824 22:19:00.386633 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ި queue_depth=1961 I0824 22:19:00.386847 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ީ queue_depth=1962 I0824 22:19:00.387102 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ު queue_depth=1963 I0824 22:19:00.387341 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ޫ queue_depth=1964 I0824 22:19:00.387613 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ެ queue_depth=1965 I0824 22:19:00.387840 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ޭ queue_depth=1966 I0824 22:19:00.388100 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ޮ queue_depth=1967 I0824 22:19:00.388338 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ޯ queue_depth=1968 I0824 22:19:00.388552 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ް queue_depth=1969 I0824 22:19:00.388798 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ޱ queue_depth=1970 I0824 22:19:00.389043 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=޲ queue_depth=1971 I0824 22:19:00.389600 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=޳ queue_depth=1972 I0824 22:19:00.390109 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=޴ queue_depth=1973 I0824 22:19:00.390826 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=޵ queue_depth=1974 I0824 22:19:00.391283 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=޶ queue_depth=1975 I0824 22:19:00.391680 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=޷ queue_depth=1976 I0824 22:19:00.392018 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=޸ queue_depth=1977 I0824 22:19:00.392445 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=޹ queue_depth=1978 I0824 22:19:00.392828 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=޺ queue_depth=1979 I0824 22:19:00.393189 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=޻ queue_depth=1980 I0824 22:19:00.393574 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=޼ queue_depth=1981 I0824 22:19:00.393966 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=޽ queue_depth=1982 I0824 22:19:00.394389 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=޾ queue_depth=1983 I0824 22:19:00.395042 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=޿ queue_depth=1984 I0824 22:19:00.395486 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=߀ queue_depth=1985 I0824 22:19:00.395843 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=߁ queue_depth=1986 I0824 22:19:00.396213 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=߂ queue_depth=1987 I0824 22:19:00.396546 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=߃ queue_depth=1988 I0824 22:19:00.396916 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=߄ queue_depth=1989 I0824 22:19:00.397243 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=߅ queue_depth=1990 I0824 22:19:00.397515 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=߆ queue_depth=1991 I0824 22:19:00.397765 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=߇ queue_depth=1992 I0824 22:19:00.398024 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=߈ queue_depth=1993 I0824 22:19:00.398536 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=߉ queue_depth=1994 I0824 22:19:00.398923 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ߊ queue_depth=1995 I0824 22:19:00.399287 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ߋ queue_depth=1996 I0824 22:19:00.399537 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ߌ queue_depth=1997 I0824 22:19:00.399768 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ߍ queue_depth=1998 I0824 22:19:00.400022 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ߎ queue_depth=1999 I0824 22:19:00.400267 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ߏ queue_depth=2000 I0824 22:19:00.400475 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ߐ queue_depth=2001 I0824 22:19:00.400729 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ߑ queue_depth=2002 I0824 22:19:00.401000 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ߒ queue_depth=2003 I0824 22:19:00.401216 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ߓ queue_depth=2004 I0824 22:19:00.401656 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ߔ queue_depth=2005 I0824 22:19:00.401964 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ߕ queue_depth=2006 I0824 22:19:00.402315 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ߖ queue_depth=2007 I0824 22:19:00.402656 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ߗ queue_depth=2008 I0824 22:19:00.403020 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ߘ queue_depth=2009 I0824 22:19:00.403419 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ߙ queue_depth=2010 I0824 22:19:00.403741 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ߚ queue_depth=2011 I0824 22:19:00.403877 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ߛ queue_depth=2012 I0824 22:19:00.404043 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ߜ queue_depth=2013 I0824 22:19:00.404601 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ߝ queue_depth=2014 I0824 22:19:00.405064 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ߞ queue_depth=2015 I0824 22:19:00.405364 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ߟ queue_depth=2016 I0824 22:19:00.405660 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ߠ queue_depth=2017 I0824 22:19:00.406041 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ߡ queue_depth=2018 I0824 22:19:00.406351 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ߢ queue_depth=2019 I0824 22:19:00.406728 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ߣ queue_depth=2020 I0824 22:19:00.407070 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ߤ queue_depth=2021 I0824 22:19:00.407448 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ߥ queue_depth=2022 I0824 22:19:00.407826 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ߦ queue_depth=2023 I0824 22:19:00.408137 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ߧ queue_depth=2024 I0824 22:19:00.408393 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ߨ queue_depth=2025 I0824 22:19:00.408635 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ߩ queue_depth=2026 I0824 22:19:00.408873 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ߪ queue_depth=2027 I0824 22:19:00.409139 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=߫ queue_depth=2028 I0824 22:19:00.409379 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=߬ queue_depth=2029 I0824 22:19:00.409601 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=߭ queue_depth=2030 I0824 22:19:00.409821 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=߮ queue_depth=2031 I0824 22:19:00.410046 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=߯ queue_depth=2032 I0824 22:19:00.410262 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=߰ queue_depth=2033 I0824 22:19:00.410570 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=߱ queue_depth=2034 I0824 22:19:00.410799 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=߲ queue_depth=2035 I0824 22:19:00.410992 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=߳ queue_depth=2036 I0824 22:19:00.411178 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ߴ queue_depth=2037 I0824 22:19:00.411550 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ߵ queue_depth=2038 I0824 22:19:00.411809 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=߶ queue_depth=2039 I0824 22:19:00.411979 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=߷ queue_depth=2040 I0824 22:19:00.412074 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=߸ queue_depth=2041 I0824 22:19:00.412349 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=߹ queue_depth=2042 I0824 22:19:00.412627 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ߺ queue_depth=2043 I0824 22:19:00.412865 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=߻ queue_depth=2044 I0824 22:19:00.413172 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=߼ queue_depth=2045 I0824 22:19:00.413434 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=߽ queue_depth=2046 I0824 22:19:00.413715 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=߾ queue_depth=2047 I0824 22:19:00.413991 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=߿ queue_depth=2048 I0824 22:19:00.414615 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ࠀ queue_depth=2049 I0824 22:19:00.415047 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ࠁ queue_depth=2050 I0824 22:19:00.415320 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ࠂ queue_depth=2051 I0824 22:19:00.415482 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ࠃ queue_depth=2052 I0824 22:19:00.415627 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ࠄ queue_depth=2053 I0824 22:19:00.415841 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ࠅ queue_depth=2054 I0824 22:19:00.416039 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ࠆ queue_depth=2055 I0824 22:19:00.416520 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ࠇ queue_depth=2056 I0824 22:19:00.416802 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ࠈ queue_depth=2057 I0824 22:19:00.416973 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ࠉ queue_depth=2058 I0824 22:19:00.417030 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ࠊ queue_depth=2059 I0824 22:19:00.417081 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ࠋ queue_depth=2060 I0824 22:19:00.417151 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ࠌ queue_depth=2061 I0824 22:19:00.417264 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ࠍ queue_depth=2062 I0824 22:19:00.417453 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ࠎ queue_depth=2063 I0824 22:19:00.417624 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ࠏ queue_depth=2064 I0824 22:19:00.417688 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ࠐ queue_depth=2065 I0824 22:19:00.417759 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ࠑ queue_depth=2066 I0824 22:19:00.417815 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ࠒ queue_depth=2067 I0824 22:19:00.417897 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ࠓ queue_depth=2068 I0824 22:19:00.418084 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ࠔ queue_depth=2069 I0824 22:19:00.418305 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ࠕ queue_depth=2070 I0824 22:19:00.418979 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ࠖ queue_depth=2071 I0824 22:19:00.419553 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ࠗ queue_depth=2072 I0824 22:19:00.419928 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=࠘ queue_depth=2073 I0824 22:19:00.420281 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=࠙ queue_depth=2074 I0824 22:19:00.420609 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ࠚ queue_depth=2075 I0824 22:19:00.420989 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ࠛ queue_depth=2076 I0824 22:19:00.421450 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ࠜ queue_depth=2077 I0824 22:19:00.422169 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ࠝ queue_depth=2078 I0824 22:19:00.422747 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ࠞ queue_depth=2079 I0824 22:19:00.423178 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ࠟ queue_depth=2080 I0824 22:19:00.423549 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ࠠ queue_depth=2081 I0824 22:19:00.424916 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ࠡ queue_depth=2082 I0824 22:19:00.425253 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ࠢ queue_depth=2083 I0824 22:19:00.425490 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ࠣ queue_depth=2084 I0824 22:19:00.425661 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ࠤ queue_depth=2085 I0824 22:19:00.425835 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ࠥ queue_depth=2086 I0824 22:19:00.425902 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ࠦ queue_depth=2087 I0824 22:19:00.425986 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ࠧ queue_depth=2088 I0824 22:19:00.426044 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ࠨ queue_depth=2089 I0824 22:19:00.426201 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ࠩ queue_depth=2090 I0824 22:19:00.426275 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ࠪ queue_depth=2091 I0824 22:19:00.426463 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ࠫ queue_depth=2092 I0824 22:19:00.426752 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ࠬ queue_depth=2093 I0824 22:19:00.426974 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=࠭ queue_depth=2094 I0824 22:19:00.427248 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=࠮ queue_depth=2095 I0824 22:19:00.427473 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=࠯ queue_depth=2096 I0824 22:19:00.427731 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=࠰ queue_depth=2097 I0824 22:19:00.427993 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=࠱ queue_depth=2098 I0824 22:19:00.428242 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=࠲ queue_depth=2099 I0824 22:19:00.428471 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=࠳ queue_depth=2100 I0824 22:19:00.428786 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=࠴ queue_depth=2101 I0824 22:19:00.429131 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=࠵ queue_depth=2102 I0824 22:19:00.429516 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=࠶ queue_depth=2103 I0824 22:19:00.430234 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=࠷ queue_depth=2104 I0824 22:19:00.430739 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=࠸ queue_depth=2105 I0824 22:19:00.431081 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=࠹ queue_depth=2106 I0824 22:19:00.431423 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=࠺ queue_depth=2107 I0824 22:19:00.431774 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=࠻ queue_depth=2108 I0824 22:19:00.432230 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=࠼ queue_depth=2109 I0824 22:19:00.432668 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=࠽ queue_depth=2110 I0824 22:19:00.433105 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=࠾ queue_depth=2111 I0824 22:19:00.433488 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=࠿ queue_depth=2112 I0824 22:19:00.433820 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ࡀ queue_depth=2113 I0824 22:19:00.434468 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ࡁ queue_depth=2114 I0824 22:19:00.434868 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ࡂ queue_depth=2115 I0824 22:19:00.435110 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ࡃ queue_depth=2116 I0824 22:19:00.435333 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ࡄ queue_depth=2117 I0824 22:19:00.435564 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ࡅ queue_depth=2118 I0824 22:19:00.435766 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ࡆ queue_depth=2119 I0824 22:19:00.435977 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ࡇ queue_depth=2120 I0824 22:19:00.436173 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ࡈ queue_depth=2121 I0824 22:19:00.436408 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ࡉ queue_depth=2122 I0824 22:19:00.436603 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ࡊ queue_depth=2123 I0824 22:19:00.436832 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ࡋ queue_depth=2124 I0824 22:19:00.437032 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ࡌ queue_depth=2125 I0824 22:19:00.437261 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ࡍ queue_depth=2126 I0824 22:19:00.437466 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ࡎ queue_depth=2127 I0824 22:19:00.437660 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ࡏ queue_depth=2128 I0824 22:19:00.437850 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ࡐ queue_depth=2129 I0824 22:19:00.438025 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ࡑ queue_depth=2130 I0824 22:19:00.438207 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ࡒ queue_depth=2131 I0824 22:19:00.438358 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ࡓ queue_depth=2132 I0824 22:19:00.438430 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ࡔ queue_depth=2133 I0824 22:19:00.438493 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ࡕ queue_depth=2134 I0824 22:19:00.438575 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ࡖ queue_depth=2135 I0824 22:19:00.439354 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ࡗ queue_depth=2136 I0824 22:19:00.440113 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ࡘ queue_depth=2137 I0824 22:19:00.440913 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=࡙ queue_depth=2138 I0824 22:19:00.441388 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=࡚ queue_depth=2139 I0824 22:19:00.441761 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=࡛ queue_depth=2140 I0824 22:19:00.442195 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=࡜ queue_depth=2141 I0824 22:19:00.442650 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=࡝ queue_depth=2142 I0824 22:19:00.443097 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=࡞ queue_depth=2143 I0824 22:19:00.443467 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=࡟ queue_depth=2144 I0824 22:19:00.444784 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ࡠ queue_depth=2145 I0824 22:19:00.445251 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ࡡ queue_depth=2146 I0824 22:19:00.445512 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ࡢ queue_depth=2147 I0824 22:19:00.445706 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ࡣ queue_depth=2148 I0824 22:19:00.445911 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ࡤ queue_depth=2149 I0824 22:19:00.446112 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ࡥ queue_depth=2150 I0824 22:19:00.446350 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ࡦ queue_depth=2151 I0824 22:19:00.446558 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ࡧ queue_depth=2152 I0824 22:19:00.446827 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ࡨ queue_depth=2153 I0824 22:19:00.447099 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ࡩ queue_depth=2154 I0824 22:19:00.447380 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ࡪ queue_depth=2155 I0824 22:19:00.447628 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=࡫ queue_depth=2156 I0824 22:19:00.447899 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=࡬ queue_depth=2157 I0824 22:19:00.448434 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=࡭ queue_depth=2158 I0824 22:19:00.448761 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=࡮ queue_depth=2159 I0824 22:19:00.449055 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=࡯ queue_depth=2160 I0824 22:19:00.449562 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ࡰ queue_depth=2161 I0824 22:19:00.450024 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ࡱ queue_depth=2162 I0824 22:19:00.450440 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ࡲ queue_depth=2163 I0824 22:19:00.450841 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ࡳ queue_depth=2164 I0824 22:19:00.451258 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ࡴ queue_depth=2165 I0824 22:19:00.451638 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ࡵ queue_depth=2166 I0824 22:19:00.452056 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ࡶ queue_depth=2167 I0824 22:19:00.452424 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ࡷ queue_depth=2168 I0824 22:19:00.452858 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ࡸ queue_depth=2169 I0824 22:19:00.453570 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ࡹ queue_depth=2170 I0824 22:19:00.454106 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ࡺ queue_depth=2171 I0824 22:19:00.454816 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ࡻ queue_depth=2172 I0824 22:19:00.455249 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ࡼ queue_depth=2173 I0824 22:19:00.455515 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ࡽ queue_depth=2174 I0824 22:19:00.455832 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ࡾ queue_depth=2175 I0824 22:19:00.456148 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ࡿ queue_depth=2176 I0824 22:19:00.456389 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ࢀ queue_depth=2177 I0824 22:19:00.456726 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ࢁ queue_depth=2178 I0824 22:19:00.456946 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ࢂ queue_depth=2179 I0824 22:19:00.457209 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ࢃ queue_depth=2180 I0824 22:19:00.457434 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ࢄ queue_depth=2181 I0824 22:19:00.457640 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ࢅ queue_depth=2182 I0824 22:19:00.457905 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ࢆ queue_depth=2183 I0824 22:19:00.458103 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ࢇ queue_depth=2184 I0824 22:19:00.458400 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=࢈ queue_depth=2185 I0824 22:19:00.458683 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ࢉ queue_depth=2186 I0824 22:19:00.458984 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ࢊ queue_depth=2187 I0824 22:19:00.459268 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ࢋ queue_depth=2188 I0824 22:19:00.459556 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ࢌ queue_depth=2189 I0824 22:19:00.459845 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ࢍ queue_depth=2190 I0824 22:19:00.460067 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ࢎ queue_depth=2191 I0824 22:19:00.460352 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=࢏ queue_depth=2192 I0824 22:19:00.460585 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=࢐ queue_depth=2193 I0824 22:19:00.460828 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=࢑ queue_depth=2194 I0824 22:19:00.461083 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=࢒ queue_depth=2195 I0824 22:19:00.461474 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=࢓ queue_depth=2196 I0824 22:19:00.461908 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=࢔ queue_depth=2197 I0824 22:19:00.462241 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=࢕ queue_depth=2198 I0824 22:19:00.462685 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=࢖ queue_depth=2199 I0824 22:19:00.463070 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ࢗ queue_depth=2200 I0824 22:19:00.463446 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=࢘ queue_depth=2201 I0824 22:19:00.463854 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=࢙ queue_depth=2202 I0824 22:19:00.465668 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=࢚ queue_depth=2203 I0824 22:19:00.466239 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=࢛ queue_depth=2204 I0824 22:19:00.466628 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=࢜ queue_depth=2205 I0824 22:19:00.467001 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=࢝ queue_depth=2206 I0824 22:19:00.467412 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=࢞ queue_depth=2207 I0824 22:19:00.467748 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=࢟ queue_depth=2208 I0824 22:19:00.468104 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ࢠ queue_depth=2209 I0824 22:19:00.468478 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ࢡ queue_depth=2210 I0824 22:19:00.468820 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ࢢ queue_depth=2211 I0824 22:19:00.469025 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ࢣ queue_depth=2212 I0824 22:19:00.469149 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ࢤ queue_depth=2213 I0824 22:19:00.469304 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ࢥ queue_depth=2214 I0824 22:19:00.469366 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ࢦ queue_depth=2215 I0824 22:19:00.469413 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ࢧ queue_depth=2216 I0824 22:19:00.469470 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ࢨ queue_depth=2217 I0824 22:19:00.469533 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ࢩ queue_depth=2218 I0824 22:19:00.469577 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ࢪ queue_depth=2219 I0824 22:19:00.469882 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ࢫ queue_depth=2220 I0824 22:19:00.470180 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ࢬ queue_depth=2221 I0824 22:19:00.470384 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ࢭ queue_depth=2222 I0824 22:19:00.470579 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ࢮ queue_depth=2223 I0824 22:19:00.470773 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ࢯ queue_depth=2224 I0824 22:19:00.470925 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ࢰ queue_depth=2225 I0824 22:19:00.470987 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ࢱ queue_depth=2226 I0824 22:19:00.471038 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ࢲ queue_depth=2227 I0824 22:19:00.471125 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ࢳ queue_depth=2228 I0824 22:19:00.471176 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ࢴ queue_depth=2229 I0824 22:19:00.471219 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ࢵ queue_depth=2230 I0824 22:19:00.471266 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ࢶ queue_depth=2231 I0824 22:19:00.471312 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ࢷ queue_depth=2232 I0824 22:19:00.471509 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ࢸ queue_depth=2233 I0824 22:19:00.471708 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ࢹ queue_depth=2234 I0824 22:19:00.471869 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ࢺ queue_depth=2235 I0824 22:19:00.472435 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ࢻ queue_depth=2236 I0824 22:19:00.472871 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ࢼ queue_depth=2237 I0824 22:19:00.473192 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ࢽ queue_depth=2238 I0824 22:19:00.473435 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ࢾ queue_depth=2239 I0824 22:19:00.473730 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ࢿ queue_depth=2240 I0824 22:19:00.474027 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ࣀ queue_depth=2241 I0824 22:19:00.474772 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ࣁ queue_depth=2242 I0824 22:19:00.475131 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ࣂ queue_depth=2243 I0824 22:19:00.475860 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ࣃ queue_depth=2244 I0824 22:19:00.476238 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ࣄ queue_depth=2245 I0824 22:19:00.476440 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ࣅ queue_depth=2246 I0824 22:19:00.476674 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ࣆ queue_depth=2247 I0824 22:19:00.476902 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ࣇ queue_depth=2248 I0824 22:19:00.477115 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ࣈ queue_depth=2249 I0824 22:19:00.477361 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ࣉ queue_depth=2250 I0824 22:19:00.477557 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=࣊ queue_depth=2251 I0824 22:19:00.477791 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=࣋ queue_depth=2252 I0824 22:19:00.478041 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=࣌ queue_depth=2253 I0824 22:19:00.478310 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=࣍ queue_depth=2254 I0824 22:19:00.478541 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=࣎ queue_depth=2255 I0824 22:19:00.478796 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=࣏ queue_depth=2256 I0824 22:19:00.479020 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=࣐ queue_depth=2257 I0824 22:19:00.479246 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=࣑ queue_depth=2258 I0824 22:19:00.479518 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=࣒ queue_depth=2259 I0824 22:19:00.479754 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=࣓ queue_depth=2260 I0824 22:19:00.479968 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ࣔ queue_depth=2261 I0824 22:19:00.480254 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ࣕ queue_depth=2262 I0824 22:19:00.480602 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ࣖ queue_depth=2263 I0824 22:19:00.480917 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ࣗ queue_depth=2264 I0824 22:19:00.481158 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ࣘ queue_depth=2265 I0824 22:19:00.481464 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ࣙ queue_depth=2266 I0824 22:19:00.481758 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ࣚ queue_depth=2267 I0824 22:19:00.482093 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ࣛ queue_depth=2268 I0824 22:19:00.482675 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ࣜ queue_depth=2269 I0824 22:19:00.483033 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ࣝ queue_depth=2270 I0824 22:19:00.483244 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ࣞ queue_depth=2271 I0824 22:19:00.483404 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ࣟ queue_depth=2272 I0824 22:19:00.483468 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=࣠ queue_depth=2273 I0824 22:19:00.483516 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=࣡ queue_depth=2274 I0824 22:19:00.483560 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=࣢ queue_depth=2275 I0824 22:19:00.483640 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ࣣ queue_depth=2276 I0824 22:19:00.483787 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ࣤ queue_depth=2277 I0824 22:19:00.483845 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ࣥ queue_depth=2278 I0824 22:19:00.483885 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ࣦ queue_depth=2279 I0824 22:19:00.483957 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ࣧ queue_depth=2280 I0824 22:19:00.484230 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ࣨ queue_depth=2281 I0824 22:19:00.484459 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ࣩ queue_depth=2282 I0824 22:19:00.484690 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=࣪ queue_depth=2283 I0824 22:19:00.484929 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=࣫ queue_depth=2284 I0824 22:19:00.485094 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=࣬ queue_depth=2285 I0824 22:19:00.485182 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=࣭ queue_depth=2286 I0824 22:19:00.485334 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=࣮ queue_depth=2287 I0824 22:19:00.485393 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=࣯ queue_depth=2288 I0824 22:19:00.485440 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ࣰ queue_depth=2289 I0824 22:19:00.485529 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ࣱ queue_depth=2290 I0824 22:19:00.485595 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ࣲ queue_depth=2291 I0824 22:19:00.485661 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ࣳ queue_depth=2292 I0824 22:19:00.485739 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ࣴ queue_depth=2293 I0824 22:19:00.485980 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ࣵ queue_depth=2294 I0824 22:19:00.486216 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ࣶ queue_depth=2295 I0824 22:19:00.486459 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ࣷ queue_depth=2296 I0824 22:19:00.486664 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ࣸ queue_depth=2297 I0824 22:19:00.486782 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ࣹ queue_depth=2298 I0824 22:19:00.486973 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ࣺ queue_depth=2299 I0824 22:19:00.487175 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ࣻ queue_depth=2300 I0824 22:19:00.487406 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ࣼ queue_depth=2301 I0824 22:19:00.487944 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ࣽ queue_depth=2302 I0824 22:19:00.488356 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ࣾ queue_depth=2303 I0824 22:19:00.488978 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ࣿ queue_depth=2304 I0824 22:19:00.489432 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ऀ queue_depth=2305 I0824 22:19:00.489765 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ँ queue_depth=2306 I0824 22:19:00.490080 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ं queue_depth=2307 I0824 22:19:00.490340 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ः queue_depth=2308 I0824 22:19:00.490651 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ऄ queue_depth=2309 I0824 22:19:00.490926 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=अ queue_depth=2310 I0824 22:19:00.491238 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=आ queue_depth=2311 I0824 22:19:00.491497 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=इ queue_depth=2312 I0824 22:19:00.491662 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ई queue_depth=2313 I0824 22:19:00.491933 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=उ queue_depth=2314 I0824 22:19:00.492211 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ऊ queue_depth=2315 I0824 22:19:00.492456 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ऋ queue_depth=2316 I0824 22:19:00.492720 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ऌ queue_depth=2317 I0824 22:19:00.492948 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ऍ queue_depth=2318 I0824 22:19:00.493144 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ऎ queue_depth=2319 I0824 22:19:00.493223 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ए queue_depth=2320 I0824 22:19:00.493443 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ऐ queue_depth=2321 I0824 22:19:00.493559 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ऑ queue_depth=2322 I0824 22:19:00.493741 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ऒ queue_depth=2323 I0824 22:19:00.494533 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ओ queue_depth=2324 I0824 22:19:00.494857 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=औ queue_depth=2325 I0824 22:19:00.495383 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=क queue_depth=2326 I0824 22:19:00.495725 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ख queue_depth=2327 I0824 22:19:00.495924 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ग queue_depth=2328 I0824 22:19:00.496023 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=घ queue_depth=2329 I0824 22:19:00.496200 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ङ queue_depth=2330 I0824 22:19:00.496448 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=च queue_depth=2331 I0824 22:19:00.496779 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=छ queue_depth=2332 I0824 22:19:00.497043 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ज queue_depth=2333 I0824 22:19:00.497316 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=झ queue_depth=2334 I0824 22:19:00.497917 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ञ queue_depth=2335 I0824 22:19:00.498318 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ट queue_depth=2336 I0824 22:19:00.498647 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ठ queue_depth=2337 I0824 22:19:00.498931 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ड queue_depth=2338 I0824 22:19:00.499210 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ढ queue_depth=2339 I0824 22:19:00.499461 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ण queue_depth=2340 I0824 22:19:00.499726 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=त queue_depth=2341 I0824 22:19:00.499988 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=थ queue_depth=2342 I0824 22:19:00.500287 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=द queue_depth=2343 I0824 22:19:00.500561 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ध queue_depth=2344 I0824 22:19:00.500856 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=न queue_depth=2345 I0824 22:19:00.501213 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ऩ queue_depth=2346 I0824 22:19:00.501526 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=प queue_depth=2347 I0824 22:19:00.501825 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=फ queue_depth=2348 I0824 22:19:00.502144 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ब queue_depth=2349 I0824 22:19:00.502452 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=भ queue_depth=2350 I0824 22:19:00.502783 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=म queue_depth=2351 I0824 22:19:00.503061 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=य queue_depth=2352 I0824 22:19:00.503398 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=र queue_depth=2353 I0824 22:19:00.503538 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ऱ queue_depth=2354 I0824 22:19:00.503632 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ल queue_depth=2355 I0824 22:19:00.503706 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ळ queue_depth=2356 I0824 22:19:00.503841 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ऴ queue_depth=2357 I0824 22:19:00.504113 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=व queue_depth=2358 I0824 22:19:00.504724 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=श queue_depth=2359 I0824 22:19:00.505047 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ष queue_depth=2360 I0824 22:19:00.505302 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=स queue_depth=2361 I0824 22:19:00.505402 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ह queue_depth=2362 I0824 22:19:00.505477 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ऺ queue_depth=2363 I0824 22:19:00.505595 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ऻ queue_depth=2364 I0824 22:19:00.505751 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=़ queue_depth=2365 I0824 22:19:00.505986 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ऽ queue_depth=2366 I0824 22:19:00.506191 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ा queue_depth=2367 I0824 22:19:00.506731 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ि queue_depth=2368 I0824 22:19:00.507014 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ी queue_depth=2369 I0824 22:19:00.507202 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ु queue_depth=2370 I0824 22:19:00.507418 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ू queue_depth=2371 I0824 22:19:00.507479 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ृ queue_depth=2372 I0824 22:19:00.507543 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ॄ queue_depth=2373 I0824 22:19:00.507606 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ॅ queue_depth=2374 I0824 22:19:00.507665 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ॆ queue_depth=2375 I0824 22:19:00.507763 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=े queue_depth=2376 I0824 22:19:00.508095 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ै queue_depth=2377 I0824 22:19:00.508318 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ॉ queue_depth=2378 I0824 22:19:00.508584 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ॊ queue_depth=2379 I0824 22:19:00.508829 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ो queue_depth=2380 I0824 22:19:00.509034 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ौ queue_depth=2381 I0824 22:19:00.509108 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=् queue_depth=2382 I0824 22:19:00.509180 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ॎ queue_depth=2383 I0824 22:19:00.509783 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ॏ queue_depth=2384 I0824 22:19:00.510130 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ॐ queue_depth=2385 I0824 22:19:00.510762 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=॑ queue_depth=2386 I0824 22:19:00.511126 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=॒ queue_depth=2387 I0824 22:19:00.511442 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=॓ queue_depth=2388 I0824 22:19:00.511772 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=॔ queue_depth=2389 I0824 22:19:00.512069 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ॕ queue_depth=2390 I0824 22:19:00.512324 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ॖ queue_depth=2391 I0824 22:19:00.512658 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ॗ queue_depth=2392 I0824 22:19:00.512994 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=क़ queue_depth=2393 I0824 22:19:00.513277 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ख़ queue_depth=2394 I0824 22:19:00.513547 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ग़ queue_depth=2395 I0824 22:19:00.513837 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ज़ queue_depth=2396 I0824 22:19:00.514172 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ड़ queue_depth=2397 I0824 22:19:00.514797 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ढ़ queue_depth=2398 I0824 22:19:00.515175 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=फ़ queue_depth=2399 I0824 22:19:00.515479 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=य़ queue_depth=2400 I0824 22:19:00.516048 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ॠ queue_depth=2401 I0824 22:19:00.516432 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ॡ queue_depth=2402 I0824 22:19:00.516724 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ॢ queue_depth=2403 I0824 22:19:00.517006 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ॣ queue_depth=2404 I0824 22:19:00.517260 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=। queue_depth=2405 I0824 22:19:00.517557 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=॥ queue_depth=2406 I0824 22:19:00.517888 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=० queue_depth=2407 I0824 22:19:00.518149 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=१ queue_depth=2408 I0824 22:19:00.518599 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=२ queue_depth=2409 I0824 22:19:00.518999 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=३ queue_depth=2410 I0824 22:19:00.519211 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=४ queue_depth=2411 I0824 22:19:00.519366 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=५ queue_depth=2412 I0824 22:19:00.519561 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=६ queue_depth=2413 I0824 22:19:00.519629 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=७ queue_depth=2414 I0824 22:19:00.519726 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=८ queue_depth=2415 I0824 22:19:00.519889 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=९ queue_depth=2416 I0824 22:19:00.519996 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=॰ queue_depth=2417 I0824 22:19:00.520226 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ॱ queue_depth=2418 I0824 22:19:00.520328 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ॲ queue_depth=2419 I0824 22:19:00.520602 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ॳ queue_depth=2420 I0824 22:19:00.520809 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ॴ queue_depth=2421 I0824 22:19:00.520913 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ॵ queue_depth=2422 I0824 22:19:00.521019 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ॶ queue_depth=2423 I0824 22:19:00.521218 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ॷ queue_depth=2424 I0824 22:19:00.521317 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ॸ queue_depth=2425 I0824 22:19:00.521487 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ॹ queue_depth=2426 I0824 22:19:00.521556 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ॺ queue_depth=2427 I0824 22:19:00.521829 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ॻ queue_depth=2428 I0824 22:19:00.522149 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ॼ queue_depth=2429 I0824 22:19:00.522402 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ॽ queue_depth=2430 I0824 22:19:00.522705 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ॾ queue_depth=2431 I0824 22:19:00.523057 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ॿ queue_depth=2432 I0824 22:19:00.523365 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ঀ queue_depth=2433 I0824 22:19:00.524852 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ঁ queue_depth=2434 I0824 22:19:00.525511 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ং queue_depth=2435 I0824 22:19:00.525804 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ঃ queue_depth=2436 I0824 22:19:00.526089 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=঄ queue_depth=2437 I0824 22:19:00.526220 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=অ queue_depth=2438 I0824 22:19:00.526517 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=আ queue_depth=2439 I0824 22:19:00.526819 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ই queue_depth=2440 I0824 22:19:00.527140 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ঈ queue_depth=2441 I0824 22:19:00.527458 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=উ queue_depth=2442 I0824 22:19:00.527796 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ঊ queue_depth=2443 I0824 22:19:00.528109 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ঋ queue_depth=2444 I0824 22:19:00.528341 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ঌ queue_depth=2445 I0824 22:19:00.528485 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=঍ queue_depth=2446 I0824 22:19:00.528790 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=঎ queue_depth=2447 I0824 22:19:00.529083 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=এ queue_depth=2448 I0824 22:19:00.529367 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ঐ queue_depth=2449 I0824 22:19:00.529638 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=঑ queue_depth=2450 I0824 22:19:00.529909 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=঒ queue_depth=2451 I0824 22:19:00.530171 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ও queue_depth=2452 I0824 22:19:00.530455 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ঔ queue_depth=2453 I0824 22:19:00.530689 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ক queue_depth=2454 I0824 22:19:00.530798 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=খ queue_depth=2455 I0824 22:19:00.531069 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=গ queue_depth=2456 I0824 22:19:00.531335 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ঘ queue_depth=2457 I0824 22:19:00.531528 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ঙ queue_depth=2458 I0824 22:19:00.531620 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=চ queue_depth=2459 I0824 22:19:00.531825 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ছ queue_depth=2460 I0824 22:19:00.532085 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=জ queue_depth=2461 I0824 22:19:00.532336 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ঝ queue_depth=2462 I0824 22:19:00.532596 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ঞ queue_depth=2463 I0824 22:19:00.532888 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ট queue_depth=2464 I0824 22:19:00.533121 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ঠ queue_depth=2465 I0824 22:19:00.533756 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ড queue_depth=2466 I0824 22:19:00.534601 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ঢ queue_depth=2467 I0824 22:19:00.535061 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ণ queue_depth=2468 I0824 22:19:00.535415 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ত queue_depth=2469 I0824 22:19:00.535705 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=থ queue_depth=2470 I0824 22:19:00.536038 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=দ queue_depth=2471 I0824 22:19:00.536631 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ধ queue_depth=2472 I0824 22:19:00.536928 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ন queue_depth=2473 I0824 22:19:00.537173 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=঩ queue_depth=2474 I0824 22:19:00.537449 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=প queue_depth=2475 I0824 22:19:00.537661 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ফ queue_depth=2476 I0824 22:19:00.537919 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ব queue_depth=2477 I0824 22:19:00.538167 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ভ queue_depth=2478 I0824 22:19:00.538403 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ম queue_depth=2479 I0824 22:19:00.538605 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=য queue_depth=2480 I0824 22:19:00.538760 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=র queue_depth=2481 I0824 22:19:00.539009 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=঱ queue_depth=2482 I0824 22:19:00.539098 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ল queue_depth=2483 I0824 22:19:00.539192 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=঳ queue_depth=2484 I0824 22:19:00.539431 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=঴ queue_depth=2485 I0824 22:19:00.539659 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=঵ queue_depth=2486 I0824 22:19:00.539770 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=শ queue_depth=2487 I0824 22:19:00.540000 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ষ queue_depth=2488 I0824 22:19:00.540271 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=স queue_depth=2489 I0824 22:19:00.540551 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=হ queue_depth=2490 I0824 22:19:00.540776 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=঺ queue_depth=2491 I0824 22:19:00.541364 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=঻ queue_depth=2492 I0824 22:19:00.541804 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=় queue_depth=2493 I0824 22:19:00.542111 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ঽ queue_depth=2494 I0824 22:19:00.542417 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=া queue_depth=2495 I0824 22:19:00.542683 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ি queue_depth=2496 I0824 22:19:00.542942 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ী queue_depth=2497 I0824 22:19:00.543329 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ু queue_depth=2498 I0824 22:19:00.543465 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ূ queue_depth=2499 I0824 22:19:00.544075 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ৃ queue_depth=2500 I0824 22:19:00.544819 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ৄ queue_depth=2501 I0824 22:19:00.545137 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=৅ queue_depth=2502 I0824 22:19:00.545370 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=৆ queue_depth=2503 I0824 22:19:00.545492 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ে queue_depth=2504 I0824 22:19:00.545695 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ৈ queue_depth=2505 I0824 22:19:00.545799 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=৉ queue_depth=2506 I0824 22:19:00.546050 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=৊ queue_depth=2507 I0824 22:19:00.546342 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ো queue_depth=2508 I0824 22:19:00.546646 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ৌ queue_depth=2509 I0824 22:19:00.546943 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=্ queue_depth=2510 I0824 22:19:00.547147 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ৎ queue_depth=2511 I0824 22:19:00.547339 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=৏ queue_depth=2512 I0824 22:19:00.547429 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=৐ queue_depth=2513 I0824 22:19:00.547490 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=৑ queue_depth=2514 I0824 22:19:00.547546 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=৒ queue_depth=2515 I0824 22:19:00.547611 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=৓ queue_depth=2516 I0824 22:19:00.547667 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=৔ queue_depth=2517 I0824 22:19:00.547752 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=৕ queue_depth=2518 I0824 22:19:00.547964 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=৖ queue_depth=2519 I0824 22:19:00.548237 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ৗ queue_depth=2520 I0824 22:19:00.548513 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=৘ queue_depth=2521 I0824 22:19:00.548793 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=৙ queue_depth=2522 I0824 22:19:00.549055 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=৚ queue_depth=2523 I0824 22:19:00.549327 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=৛ queue_depth=2524 I0824 22:19:00.549630 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ড় queue_depth=2525 I0824 22:19:00.549839 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ঢ় queue_depth=2526 I0824 22:19:00.550043 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=৞ queue_depth=2527 I0824 22:19:00.550135 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=য় queue_depth=2528 I0824 22:19:00.550283 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ৠ queue_depth=2529 I0824 22:19:00.550447 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ৡ queue_depth=2530 I0824 22:19:00.550527 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ৢ queue_depth=2531 I0824 22:19:00.550588 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ৣ queue_depth=2532 I0824 22:19:00.551019 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=৤ queue_depth=2533 I0824 22:19:00.551320 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=৥ queue_depth=2534 I0824 22:19:00.551624 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=০ queue_depth=2535 I0824 22:19:00.551867 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=১ queue_depth=2536 I0824 22:19:00.551966 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=২ queue_depth=2537 I0824 22:19:00.552204 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=৩ queue_depth=2538 I0824 22:19:00.552423 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=৪ queue_depth=2539 I0824 22:19:00.552641 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=৫ queue_depth=2540 I0824 22:19:00.552752 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=৬ queue_depth=2541 I0824 22:19:00.553001 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=৭ queue_depth=2542 I0824 22:19:00.553286 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=৮ queue_depth=2543 I0824 22:19:00.553405 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=৯ queue_depth=2544 I0824 22:19:00.553502 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ৰ queue_depth=2545 I0824 22:19:00.554826 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ৱ queue_depth=2546 I0824 22:19:00.555040 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=৲ queue_depth=2547 I0824 22:19:00.555364 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=৳ queue_depth=2548 I0824 22:19:00.555477 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=৴ queue_depth=2549 I0824 22:19:00.555561 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=৵ queue_depth=2550 I0824 22:19:00.555625 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=৶ queue_depth=2551 I0824 22:19:00.555661 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=৷ queue_depth=2552 I0824 22:19:00.555780 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=৸ queue_depth=2553 I0824 22:19:00.555816 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=৹ queue_depth=2554 I0824 22:19:00.555891 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=৺ queue_depth=2555 I0824 22:19:00.555979 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=৻ queue_depth=2556 I0824 22:19:00.556088 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ৼ queue_depth=2557 I0824 22:19:00.556148 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=৽ queue_depth=2558 I0824 22:19:00.556206 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=৾ queue_depth=2559 I0824 22:19:00.556335 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=৿ queue_depth=2560 I0824 22:19:00.556424 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=਀ queue_depth=2561 I0824 22:19:00.556526 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ਁ queue_depth=2562 I0824 22:19:00.556572 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ਂ queue_depth=2563 I0824 22:19:00.556609 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ਃ queue_depth=2564 I0824 22:19:00.556650 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=਄ queue_depth=2565 I0824 22:19:00.556683 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ਅ queue_depth=2566 I0824 22:19:00.556789 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ਆ queue_depth=2567 I0824 22:19:00.556844 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ਇ queue_depth=2568 I0824 22:19:00.556918 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ਈ queue_depth=2569 I0824 22:19:00.557033 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ਉ queue_depth=2570 I0824 22:19:00.557086 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ਊ queue_depth=2571 I0824 22:19:00.557239 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=਋ queue_depth=2572 I0824 22:19:00.557315 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=਌ queue_depth=2573 I0824 22:19:00.557366 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=਍ queue_depth=2574 I0824 22:19:00.557404 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=਎ queue_depth=2575 I0824 22:19:00.557449 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ਏ queue_depth=2576 I0824 22:19:00.557506 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ਐ queue_depth=2577 I0824 22:19:00.557569 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=਑ queue_depth=2578 I0824 22:19:00.557642 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=਒ queue_depth=2579 I0824 22:19:00.557704 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ਓ queue_depth=2580 I0824 22:19:00.557824 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ਔ queue_depth=2581 I0824 22:19:00.557898 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ਕ queue_depth=2582 I0824 22:19:00.557966 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ਖ queue_depth=2583 I0824 22:19:00.558003 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ਗ queue_depth=2584 I0824 22:19:00.558060 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ਘ queue_depth=2585 I0824 22:19:00.558096 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ਙ queue_depth=2586 I0824 22:19:00.558158 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ਚ queue_depth=2587 I0824 22:19:00.558194 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ਛ queue_depth=2588 I0824 22:19:00.558246 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ਜ queue_depth=2589 I0824 22:19:00.558339 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ਝ queue_depth=2590 I0824 22:19:00.558417 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ਞ queue_depth=2591 I0824 22:19:00.558530 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ਟ queue_depth=2592 I0824 22:19:00.558593 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ਠ queue_depth=2593 I0824 22:19:00.558651 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ਡ queue_depth=2594 I0824 22:19:00.558753 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ਢ queue_depth=2595 I0824 22:19:00.558806 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ਣ queue_depth=2596 I0824 22:19:00.558847 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ਤ queue_depth=2597 I0824 22:19:00.558905 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ਥ queue_depth=2598 I0824 22:19:00.558970 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ਦ queue_depth=2599 I0824 22:19:00.559014 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ਧ queue_depth=2600 I0824 22:19:00.559149 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ਨ queue_depth=2601 I0824 22:19:00.559283 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=਩ queue_depth=2602 I0824 22:19:00.559325 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ਪ queue_depth=2603 I0824 22:19:00.559435 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ਫ queue_depth=2604 I0824 22:19:00.559545 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ਬ queue_depth=2605 I0824 22:19:00.559648 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ਭ queue_depth=2606 I0824 22:19:00.559771 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ਮ queue_depth=2607 I0824 22:19:00.559842 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ਯ queue_depth=2608 I0824 22:19:00.559877 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ਰ queue_depth=2609 I0824 22:19:00.559942 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=਱ queue_depth=2610 I0824 22:19:00.560063 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ਲ queue_depth=2611 I0824 22:19:00.560149 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ਲ਼ queue_depth=2612 I0824 22:19:00.560203 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=਴ queue_depth=2613 I0824 22:19:00.560331 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ਵ queue_depth=2614 I0824 22:19:00.560457 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ਸ਼ queue_depth=2615 I0824 22:19:00.560509 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=਷ queue_depth=2616 I0824 22:19:00.560564 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ਸ queue_depth=2617 I0824 22:19:00.560608 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ਹ queue_depth=2618 I0824 22:19:00.560729 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=਺ queue_depth=2619 I0824 22:19:00.560783 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=਻ queue_depth=2620 I0824 22:19:00.560858 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=਼ queue_depth=2621 I0824 22:19:00.561011 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=਽ queue_depth=2622 I0824 22:19:00.561083 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ਾ queue_depth=2623 I0824 22:19:00.561165 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ਿ queue_depth=2624 I0824 22:19:00.561202 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ੀ queue_depth=2625 I0824 22:19:00.561237 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ੁ queue_depth=2626 I0824 22:19:00.561283 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ੂ queue_depth=2627 I0824 22:19:00.561338 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=੃ queue_depth=2628 I0824 22:19:00.561378 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=੄ queue_depth=2629 I0824 22:19:00.561436 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=੅ queue_depth=2630 I0824 22:19:00.561555 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=੆ queue_depth=2631 I0824 22:19:00.561602 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ੇ queue_depth=2632 I0824 22:19:00.561697 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ੈ queue_depth=2633 I0824 22:19:00.561810 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=੉ queue_depth=2634 I0824 22:19:00.561872 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=੊ queue_depth=2635 I0824 22:19:00.561916 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ੋ queue_depth=2636 I0824 22:19:00.561999 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ੌ queue_depth=2637 I0824 22:19:00.562050 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=੍ queue_depth=2638 I0824 22:19:00.562105 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=੎ queue_depth=2639 I0824 22:19:00.562146 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=੏ queue_depth=2640 I0824 22:19:00.562191 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=੐ queue_depth=2641 I0824 22:19:00.562245 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ੑ queue_depth=2642 I0824 22:19:00.562321 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=੒ queue_depth=2643 I0824 22:19:00.562427 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=੓ queue_depth=2644 I0824 22:19:00.562478 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=੔ queue_depth=2645 I0824 22:19:00.562520 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=੕ queue_depth=2646 I0824 22:19:00.562629 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=੖ queue_depth=2647 I0824 22:19:00.562734 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=੗ queue_depth=2648 I0824 22:19:00.562795 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=੘ queue_depth=2649 I0824 22:19:00.562859 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ਖ਼ queue_depth=2650 I0824 22:19:00.562906 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ਗ਼ queue_depth=2651 I0824 22:19:00.562938 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ਜ਼ queue_depth=2652 I0824 22:19:00.562974 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ੜ queue_depth=2653 I0824 22:19:00.563116 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=੝ queue_depth=2654 I0824 22:19:00.564815 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ਫ਼ queue_depth=2655 I0824 22:19:00.564856 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=੟ queue_depth=2656 I0824 22:19:00.564886 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=੠ queue_depth=2657 I0824 22:19:00.564969 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=੡ queue_depth=2658 I0824 22:19:00.565001 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=੢ queue_depth=2659 I0824 22:19:00.565024 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=੣ queue_depth=2660 I0824 22:19:00.565059 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=੤ queue_depth=2661 I0824 22:19:00.565199 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=੥ queue_depth=2662 I0824 22:19:00.565260 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=੦ queue_depth=2663 I0824 22:19:00.565296 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=੧ queue_depth=2664 I0824 22:19:00.565337 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=੨ queue_depth=2665 I0824 22:19:00.565386 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=੩ queue_depth=2666 I0824 22:19:00.565447 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=੪ queue_depth=2667 I0824 22:19:00.565513 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=੫ queue_depth=2668 I0824 22:19:00.565580 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=੬ queue_depth=2669 I0824 22:19:00.565638 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=੭ queue_depth=2670 I0824 22:19:00.565701 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=੮ queue_depth=2671 I0824 22:19:00.565805 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=੯ queue_depth=2672 I0824 22:19:00.565875 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ੰ queue_depth=2673 I0824 22:19:00.565926 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ੱ queue_depth=2674 I0824 22:19:00.565969 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ੲ queue_depth=2675 I0824 22:19:00.566014 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ੳ queue_depth=2676 I0824 22:19:00.566061 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ੴ queue_depth=2677 I0824 22:19:00.566100 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ੵ queue_depth=2678 I0824 22:19:00.566130 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=੶ queue_depth=2679 I0824 22:19:00.566260 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=੷ queue_depth=2680 I0824 22:19:00.566378 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=੸ queue_depth=2681 I0824 22:19:00.566465 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=੹ queue_depth=2682 I0824 22:19:00.566532 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=੺ queue_depth=2683 I0824 22:19:00.566654 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=੻ queue_depth=2684 I0824 22:19:00.566781 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=੼ queue_depth=2685 I0824 22:19:00.566835 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=੽ queue_depth=2686 I0824 22:19:00.566871 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=੾ queue_depth=2687 I0824 22:19:00.566902 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=੿ queue_depth=2688 I0824 22:19:00.566951 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=઀ queue_depth=2689 I0824 22:19:00.567025 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ઁ queue_depth=2690 I0824 22:19:00.567116 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ં queue_depth=2691 I0824 22:19:00.567184 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ઃ queue_depth=2692 I0824 22:19:00.567301 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=઄ queue_depth=2693 I0824 22:19:00.567389 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=અ queue_depth=2694 I0824 22:19:00.567452 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=આ queue_depth=2695 I0824 22:19:00.567512 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ઇ queue_depth=2696 I0824 22:19:00.567556 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ઈ queue_depth=2697 I0824 22:19:00.567603 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ઉ queue_depth=2698 I0824 22:19:00.567643 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ઊ queue_depth=2699 I0824 22:19:00.567682 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ઋ queue_depth=2700 I0824 22:19:00.567718 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ઌ queue_depth=2701 I0824 22:19:00.567753 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ઍ queue_depth=2702 I0824 22:19:00.567783 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=઎ queue_depth=2703 I0824 22:19:00.567830 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=એ queue_depth=2704 I0824 22:19:00.567931 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ઐ queue_depth=2705 I0824 22:19:00.568005 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ઑ queue_depth=2706 I0824 22:19:00.568070 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=઒ queue_depth=2707 I0824 22:19:00.568124 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ઓ queue_depth=2708 I0824 22:19:00.568186 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ઔ queue_depth=2709 I0824 22:19:00.568232 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ક queue_depth=2710 I0824 22:19:00.568299 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ખ queue_depth=2711 I0824 22:19:00.568338 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ગ queue_depth=2712 I0824 22:19:00.568485 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ઘ queue_depth=2713 I0824 22:19:00.568565 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ઙ queue_depth=2714 I0824 22:19:00.568604 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ચ queue_depth=2715 I0824 22:19:00.568675 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=છ queue_depth=2716 I0824 22:19:00.568781 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=જ queue_depth=2717 I0824 22:19:00.568832 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ઝ queue_depth=2718 I0824 22:19:00.568901 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ઞ queue_depth=2719 I0824 22:19:00.568941 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ટ queue_depth=2720 I0824 22:19:00.569004 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ઠ queue_depth=2721 I0824 22:19:00.569063 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ડ queue_depth=2722 I0824 22:19:00.569112 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ઢ queue_depth=2723 I0824 22:19:00.569155 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ણ queue_depth=2724 I0824 22:19:00.569220 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ત queue_depth=2725 I0824 22:19:00.569288 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=થ queue_depth=2726 I0824 22:19:00.569342 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=દ queue_depth=2727 I0824 22:19:00.569424 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ધ queue_depth=2728 I0824 22:19:00.569499 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ન queue_depth=2729 I0824 22:19:00.569567 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=઩ queue_depth=2730 I0824 22:19:00.569650 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=પ queue_depth=2731 I0824 22:19:00.569705 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ફ queue_depth=2732 I0824 22:19:00.569767 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=બ queue_depth=2733 I0824 22:19:00.569827 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ભ queue_depth=2734 I0824 22:19:00.569892 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=મ queue_depth=2735 I0824 22:19:00.569990 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ય queue_depth=2736 I0824 22:19:00.570075 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ર queue_depth=2737 I0824 22:19:00.570195 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=઱ queue_depth=2738 I0824 22:19:00.570295 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=લ queue_depth=2739 I0824 22:19:00.570346 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ળ queue_depth=2740 I0824 22:19:00.570438 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=઴ queue_depth=2741 I0824 22:19:00.570504 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=વ queue_depth=2742 I0824 22:19:00.570542 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=શ queue_depth=2743 I0824 22:19:00.570581 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ષ queue_depth=2744 I0824 22:19:00.570644 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=સ queue_depth=2745 I0824 22:19:00.570749 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=હ queue_depth=2746 I0824 22:19:00.570838 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=઺ queue_depth=2747 I0824 22:19:00.570920 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=઻ queue_depth=2748 I0824 22:19:00.570958 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=઼ queue_depth=2749 I0824 22:19:00.571020 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ઽ queue_depth=2750 I0824 22:19:00.571092 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ા queue_depth=2751 I0824 22:19:00.571165 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=િ queue_depth=2752 I0824 22:19:00.571305 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ી queue_depth=2753 I0824 22:19:00.571363 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ુ queue_depth=2754 I0824 22:19:00.571427 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ૂ queue_depth=2755 I0824 22:19:00.571515 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ૃ queue_depth=2756 I0824 22:19:00.571566 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ૄ queue_depth=2757 I0824 22:19:00.571621 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ૅ queue_depth=2758 I0824 22:19:00.571674 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=૆ queue_depth=2759 I0824 22:19:00.571721 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ે queue_depth=2760 I0824 22:19:00.571792 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ૈ queue_depth=2761 I0824 22:19:00.571872 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ૉ queue_depth=2762 I0824 22:19:00.571944 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=૊ queue_depth=2763 I0824 22:19:00.571991 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ો queue_depth=2764 I0824 22:19:00.572182 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ૌ queue_depth=2765 I0824 22:19:00.572285 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=્ queue_depth=2766 I0824 22:19:00.572446 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=૎ queue_depth=2767 I0824 22:19:00.572724 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=૏ queue_depth=2768 I0824 22:19:00.572988 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ૐ queue_depth=2769 I0824 22:19:00.575520 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=૑ queue_depth=2770 I0824 22:19:00.575678 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=૒ queue_depth=2771 I0824 22:19:00.575775 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=૓ queue_depth=2772 I0824 22:19:00.575898 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=૔ queue_depth=2773 I0824 22:19:00.575941 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=૕ queue_depth=2774 I0824 22:19:00.576316 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=૖ queue_depth=2775 I0824 22:19:00.576527 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=૗ queue_depth=2776 I0824 22:19:00.576591 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=૘ queue_depth=2777 I0824 22:19:00.576639 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=૙ queue_depth=2778 I0824 22:19:00.576842 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=૚ queue_depth=2779 I0824 22:19:00.577042 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=૛ queue_depth=2780 I0824 22:19:00.577181 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=૜ queue_depth=2781 I0824 22:19:00.577338 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=૝ queue_depth=2782 I0824 22:19:00.577392 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=૞ queue_depth=2783 I0824 22:19:00.577457 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=૟ queue_depth=2784 I0824 22:19:00.577501 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ૠ queue_depth=2785 I0824 22:19:00.577668 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ૡ queue_depth=2786 I0824 22:19:00.577837 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ૢ queue_depth=2787 I0824 22:19:00.577905 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ૣ queue_depth=2788 I0824 22:19:00.578057 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=૤ queue_depth=2789 I0824 22:19:00.578119 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=૥ queue_depth=2790 I0824 22:19:00.578170 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=૦ queue_depth=2791 I0824 22:19:00.578206 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=૧ queue_depth=2792 I0824 22:19:00.578392 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=૨ queue_depth=2793 I0824 22:19:00.578595 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=૩ queue_depth=2794 I0824 22:19:00.578714 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=૪ queue_depth=2795 I0824 22:19:00.578770 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=૫ queue_depth=2796 I0824 22:19:00.578828 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=૬ queue_depth=2797 I0824 22:19:00.578892 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=૭ queue_depth=2798 I0824 22:19:00.578981 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=૮ queue_depth=2799 I0824 22:19:00.579105 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=૯ queue_depth=2800 I0824 22:19:00.579300 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=૰ queue_depth=2801 I0824 22:19:00.579448 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=૱ queue_depth=2802 I0824 22:19:00.579682 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=૲ queue_depth=2803 I0824 22:19:00.579779 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=૳ queue_depth=2804 I0824 22:19:00.579848 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=૴ queue_depth=2805 I0824 22:19:00.579902 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=૵ queue_depth=2806 I0824 22:19:00.579992 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=૶ queue_depth=2807 I0824 22:19:00.580176 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=૷ queue_depth=2808 I0824 22:19:00.580297 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=૸ queue_depth=2809 I0824 22:19:00.580454 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ૹ queue_depth=2810 I0824 22:19:00.580512 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ૺ queue_depth=2811 I0824 22:19:00.580710 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ૻ queue_depth=2812 I0824 22:19:00.580906 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ૼ queue_depth=2813 I0824 22:19:00.581085 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=૽ queue_depth=2814 I0824 22:19:00.581230 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=૾ queue_depth=2815 I0824 22:19:00.581287 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=૿ queue_depth=2816 I0824 22:19:00.581356 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=଀ queue_depth=2817 I0824 22:19:00.581536 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ଁ queue_depth=2818 I0824 22:19:00.581746 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ଂ queue_depth=2819 I0824 22:19:00.581870 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ଃ queue_depth=2820 I0824 22:19:00.582021 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=଄ queue_depth=2821 I0824 22:19:00.582070 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ଅ queue_depth=2822 I0824 22:19:00.582286 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ଆ queue_depth=2823 I0824 22:19:00.582509 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ଇ queue_depth=2824 I0824 22:19:00.582632 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ଈ queue_depth=2825 I0824 22:19:00.582673 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ଉ queue_depth=2826 I0824 22:19:00.582719 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ଊ queue_depth=2827 I0824 22:19:00.582807 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ଋ queue_depth=2828 I0824 22:19:00.582858 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ଌ queue_depth=2829 I0824 22:19:00.582905 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=଍ queue_depth=2830 I0824 22:19:00.583072 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=଎ queue_depth=2831 I0824 22:19:00.585602 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ଏ queue_depth=2832 I0824 22:19:00.586057 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ଐ queue_depth=2833 I0824 22:19:00.586374 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=଑ queue_depth=2834 I0824 22:19:00.586497 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=଒ queue_depth=2835 I0824 22:19:00.586562 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ଓ queue_depth=2836 I0824 22:19:00.586614 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ଔ queue_depth=2837 I0824 22:19:00.586736 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=କ queue_depth=2838 I0824 22:19:00.586965 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ଖ queue_depth=2839 I0824 22:19:00.587143 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ଗ queue_depth=2840 I0824 22:19:00.587295 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ଘ queue_depth=2841 I0824 22:19:00.587350 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ଙ queue_depth=2842 I0824 22:19:00.587405 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ଚ queue_depth=2843 I0824 22:19:00.587488 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ଛ queue_depth=2844 I0824 22:19:00.587591 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ଜ queue_depth=2845 I0824 22:19:00.587686 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ଝ queue_depth=2846 I0824 22:19:00.587900 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ଞ queue_depth=2847 I0824 22:19:00.588045 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ଟ queue_depth=2848 I0824 22:19:00.588243 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ଠ queue_depth=2849 I0824 22:19:00.588346 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ଡ queue_depth=2850 I0824 22:19:00.588377 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ଢ queue_depth=2851 I0824 22:19:00.588473 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ଣ queue_depth=2852 I0824 22:19:00.588624 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ତ queue_depth=2853 I0824 22:19:00.588771 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ଥ queue_depth=2854 I0824 22:19:00.588950 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ଦ queue_depth=2855 I0824 22:19:00.589075 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ଧ queue_depth=2856 I0824 22:19:00.589151 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ନ queue_depth=2857 I0824 22:19:00.589200 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=଩ queue_depth=2858 I0824 22:19:00.589251 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ପ queue_depth=2859 I0824 22:19:00.589411 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ଫ queue_depth=2860 I0824 22:19:00.589577 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ବ queue_depth=2861 I0824 22:19:00.589720 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ଭ queue_depth=2862 I0824 22:19:00.589787 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ମ queue_depth=2863 I0824 22:19:00.589831 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ଯ queue_depth=2864 I0824 22:19:00.589866 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ର queue_depth=2865 I0824 22:19:00.589904 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=଱ queue_depth=2866 I0824 22:19:00.590014 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ଲ queue_depth=2867 I0824 22:19:00.590197 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ଳ queue_depth=2868 I0824 22:19:00.590342 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=଴ queue_depth=2869 I0824 22:19:00.590478 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ଵ queue_depth=2870 I0824 22:19:00.590550 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ଶ queue_depth=2871 I0824 22:19:00.590695 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ଷ queue_depth=2872 I0824 22:19:00.590847 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ସ queue_depth=2873 I0824 22:19:00.591026 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ହ queue_depth=2874 I0824 22:19:00.591187 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=଺ queue_depth=2875 I0824 22:19:00.591244 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=଻ queue_depth=2876 I0824 22:19:00.591363 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=଼ queue_depth=2877 I0824 22:19:00.591519 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ଽ queue_depth=2878 I0824 22:19:00.591720 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ା queue_depth=2879 I0824 22:19:00.591886 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ି queue_depth=2880 I0824 22:19:00.591988 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ୀ queue_depth=2881 I0824 22:19:00.592045 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ୁ queue_depth=2882 I0824 22:19:00.592073 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ୂ queue_depth=2883 I0824 22:19:00.592124 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ୃ queue_depth=2884 I0824 22:19:00.592272 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ୄ queue_depth=2885 I0824 22:19:00.592428 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=୅ queue_depth=2886 I0824 22:19:00.592574 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=୆ queue_depth=2887 I0824 22:19:00.592663 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=େ queue_depth=2888 I0824 22:19:00.592859 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ୈ queue_depth=2889 I0824 22:19:00.593027 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=୉ queue_depth=2890 I0824 22:19:00.593232 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=୊ queue_depth=2891 I0824 22:19:00.593419 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ୋ queue_depth=2892 I0824 22:19:00.593532 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ୌ queue_depth=2893 I0824 22:19:00.593576 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=୍ queue_depth=2894 I0824 22:19:00.593620 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=୎ queue_depth=2895 I0824 22:19:00.593744 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=୏ queue_depth=2896 I0824 22:19:00.593910 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=୐ queue_depth=2897 I0824 22:19:00.594057 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=୑ queue_depth=2898 I0824 22:19:00.594124 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=୒ queue_depth=2899 I0824 22:19:00.594344 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=୓ queue_depth=2900 I0824 22:19:00.594587 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=୔ queue_depth=2901 I0824 22:19:00.594793 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=୕ queue_depth=2902 I0824 22:19:00.594844 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ୖ queue_depth=2903 I0824 22:19:00.594971 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ୗ queue_depth=2904 I0824 22:19:00.595119 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=୘ queue_depth=2905 I0824 22:19:00.595197 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=୙ queue_depth=2906 I0824 22:19:00.595251 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=୚ queue_depth=2907 I0824 22:19:00.595403 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=୛ queue_depth=2908 I0824 22:19:00.595558 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ଡ଼ queue_depth=2909 I0824 22:19:00.595600 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ଢ଼ queue_depth=2910 I0824 22:19:00.595681 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=୞ queue_depth=2911 I0824 22:19:00.595800 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ୟ queue_depth=2912 I0824 22:19:00.595977 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ୠ queue_depth=2913 I0824 22:19:00.596131 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ୡ queue_depth=2914 I0824 22:19:00.596262 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ୢ queue_depth=2915 I0824 22:19:00.596305 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ୣ queue_depth=2916 I0824 22:19:00.596352 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=୤ queue_depth=2917 I0824 22:19:00.596398 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=୥ queue_depth=2918 I0824 22:19:00.596539 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=୦ queue_depth=2919 I0824 22:19:00.596689 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=୧ queue_depth=2920 I0824 22:19:00.596762 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=୨ queue_depth=2921 I0824 22:19:00.596936 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=୩ queue_depth=2922 I0824 22:19:00.596974 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=୪ queue_depth=2923 I0824 22:19:00.597005 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=୫ queue_depth=2924 I0824 22:19:00.597061 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=୬ queue_depth=2925 I0824 22:19:00.597097 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=୭ queue_depth=2926 I0824 22:19:00.597135 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=୮ queue_depth=2927 I0824 22:19:00.597276 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=୯ queue_depth=2928 I0824 22:19:00.597419 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=୰ queue_depth=2929 I0824 22:19:00.597585 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ୱ queue_depth=2930 I0824 22:19:00.597719 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=୲ queue_depth=2931 I0824 22:19:00.597934 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=୳ queue_depth=2932 I0824 22:19:00.598118 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=୴ queue_depth=2933 I0824 22:19:00.598282 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=୵ queue_depth=2934 I0824 22:19:00.598420 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=୶ queue_depth=2935 I0824 22:19:00.598468 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=୷ queue_depth=2936 I0824 22:19:00.598501 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=୸ queue_depth=2937 I0824 22:19:00.598650 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=୹ queue_depth=2938 I0824 22:19:00.598823 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=୺ queue_depth=2939 I0824 22:19:00.599027 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=୻ queue_depth=2940 I0824 22:19:00.599098 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=୼ queue_depth=2941 I0824 22:19:00.599173 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=୽ queue_depth=2942 I0824 22:19:00.599378 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=୾ queue_depth=2943 I0824 22:19:00.599708 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=୿ queue_depth=2944 I0824 22:19:00.599901 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=஀ queue_depth=2945 I0824 22:19:00.600084 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=஁ queue_depth=2946 I0824 22:19:00.600287 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ஂ queue_depth=2947 I0824 22:19:00.600478 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ஃ queue_depth=2948 I0824 22:19:00.600518 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=஄ queue_depth=2949 I0824 22:19:00.600557 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=அ queue_depth=2950 I0824 22:19:00.600584 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ஆ queue_depth=2951 I0824 22:19:00.600767 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=இ queue_depth=2952 I0824 22:19:00.600956 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ஈ queue_depth=2953 I0824 22:19:00.601117 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=உ queue_depth=2954 I0824 22:19:00.601221 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ஊ queue_depth=2955 I0824 22:19:00.601276 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=஋ queue_depth=2956 I0824 22:19:00.601437 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=஌ queue_depth=2957 I0824 22:19:00.601599 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=஍ queue_depth=2958 I0824 22:19:00.601761 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=எ queue_depth=2959 I0824 22:19:00.601812 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ஏ queue_depth=2960 I0824 22:19:00.601888 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ஐ queue_depth=2961 I0824 22:19:00.601922 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=஑ queue_depth=2962 I0824 22:19:00.601973 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ஒ queue_depth=2963 I0824 22:19:00.602155 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ஓ queue_depth=2964 I0824 22:19:00.602324 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ஔ queue_depth=2965 I0824 22:19:00.602480 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=க queue_depth=2966 I0824 22:19:00.602536 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=஖ queue_depth=2967 I0824 22:19:00.602610 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=஗ queue_depth=2968 I0824 22:19:00.602773 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=஘ queue_depth=2969 I0824 22:19:00.602965 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ங queue_depth=2970 I0824 22:19:00.603120 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ச queue_depth=2971 I0824 22:19:00.603293 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=஛ queue_depth=2972 I0824 22:19:00.603574 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ஜ queue_depth=2973 I0824 22:19:00.603754 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=஝ queue_depth=2974 I0824 22:19:00.603902 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ஞ queue_depth=2975 I0824 22:19:00.603953 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ட queue_depth=2976 I0824 22:19:00.604418 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=஠ queue_depth=2977 I0824 22:19:00.604586 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=஡ queue_depth=2978 I0824 22:19:00.604719 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=஢ queue_depth=2979 I0824 22:19:00.604882 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ண queue_depth=2980 I0824 22:19:00.605026 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=த queue_depth=2981 I0824 22:19:00.605266 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=஥ queue_depth=2982 I0824 22:19:00.605390 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=஦ queue_depth=2983 I0824 22:19:00.606642 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=஧ queue_depth=2984 I0824 22:19:00.606984 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ந queue_depth=2985 I0824 22:19:00.607123 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ன queue_depth=2986 I0824 22:19:00.607175 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ப queue_depth=2987 I0824 22:19:00.607354 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=஫ queue_depth=2988 I0824 22:19:00.607470 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=஬ queue_depth=2989 I0824 22:19:00.607577 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=஭ queue_depth=2990 I0824 22:19:00.607694 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ம queue_depth=2991 I0824 22:19:00.607858 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ய queue_depth=2992 I0824 22:19:00.608029 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ர queue_depth=2993 I0824 22:19:00.608091 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ற queue_depth=2994 I0824 22:19:00.608240 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ல queue_depth=2995 I0824 22:19:00.608409 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ள queue_depth=2996 I0824 22:19:00.608581 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ழ queue_depth=2997 I0824 22:19:00.608739 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=வ queue_depth=2998 I0824 22:19:00.608850 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ஶ queue_depth=2999 I0824 22:19:00.609025 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ஷ queue_depth=3000 I0824 22:19:00.609238 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ஸ queue_depth=3001 I0824 22:19:00.609407 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ஹ queue_depth=3002 I0824 22:19:00.609456 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=஺ queue_depth=3003 I0824 22:19:00.609666 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=஻ queue_depth=3004 I0824 22:19:00.609823 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=஼ queue_depth=3005 I0824 22:19:00.609968 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=஽ queue_depth=3006 I0824 22:19:00.610074 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ா queue_depth=3007 I0824 22:19:00.610160 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ி queue_depth=3008 I0824 22:19:00.610313 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ீ queue_depth=3009 I0824 22:19:00.610573 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ு queue_depth=3010 I0824 22:19:00.610738 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ூ queue_depth=3011 I0824 22:19:00.610894 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=௃ queue_depth=3012 I0824 22:19:00.611041 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=௄ queue_depth=3013 I0824 22:19:00.611177 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=௅ queue_depth=3014 I0824 22:19:00.611379 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ெ queue_depth=3015 I0824 22:19:00.611606 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ே queue_depth=3016 I0824 22:19:00.611745 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ை queue_depth=3017 I0824 22:19:00.611863 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=௉ queue_depth=3018 I0824 22:19:00.611932 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ொ queue_depth=3019 I0824 22:19:00.612098 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ோ queue_depth=3020 I0824 22:19:00.612253 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ௌ queue_depth=3021 I0824 22:19:00.612372 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=் queue_depth=3022 I0824 22:19:00.612455 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=௎ queue_depth=3023 I0824 22:19:00.612519 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=௏ queue_depth=3024 I0824 22:19:00.612563 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ௐ queue_depth=3025 I0824 22:19:00.612623 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=௑ queue_depth=3026 I0824 22:19:00.612866 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=௒ queue_depth=3027 I0824 22:19:00.613036 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=௓ queue_depth=3028 I0824 22:19:00.613164 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=௔ queue_depth=3029 I0824 22:19:00.613334 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=௕ queue_depth=3030 I0824 22:19:00.613493 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=௖ queue_depth=3031 I0824 22:19:00.613677 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ௗ queue_depth=3032 I0824 22:19:00.613737 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=௘ queue_depth=3033 I0824 22:19:00.613877 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=௙ queue_depth=3034 I0824 22:19:00.614057 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=௚ queue_depth=3035 I0824 22:19:00.614204 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=௛ queue_depth=3036 I0824 22:19:00.614382 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=௜ queue_depth=3037 I0824 22:19:00.614591 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=௝ queue_depth=3038 I0824 22:19:00.614759 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=௞ queue_depth=3039 I0824 22:19:00.614958 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=௟ queue_depth=3040 I0824 22:19:00.615065 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=௠ queue_depth=3041 I0824 22:19:00.615262 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=௡ queue_depth=3042 I0824 22:19:00.615580 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=௢ queue_depth=3043 I0824 22:19:00.615676 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=௣ queue_depth=3044 I0824 22:19:00.615750 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=௤ queue_depth=3045 I0824 22:19:00.615913 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=௥ queue_depth=3046 I0824 22:19:00.616071 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=௦ queue_depth=3047 I0824 22:19:00.616187 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=௧ queue_depth=3048 I0824 22:19:00.616248 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=௨ queue_depth=3049 I0824 22:19:00.616282 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=௩ queue_depth=3050 I0824 22:19:00.616322 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=௪ queue_depth=3051 I0824 22:19:00.616473 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=௫ queue_depth=3052 I0824 22:19:00.616604 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=௬ queue_depth=3053 I0824 22:19:00.616717 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=௭ queue_depth=3054 I0824 22:19:00.616845 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=௮ queue_depth=3055 I0824 22:19:00.616975 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=௯ queue_depth=3056 I0824 22:19:00.617175 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=௰ queue_depth=3057 I0824 22:19:00.617282 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=௱ queue_depth=3058 I0824 22:19:00.617439 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=௲ queue_depth=3059 I0824 22:19:00.617611 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=௳ queue_depth=3060 I0824 22:19:00.617673 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=௴ queue_depth=3061 I0824 22:19:00.617717 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=௵ queue_depth=3062 I0824 22:19:00.617771 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=௶ queue_depth=3063 I0824 22:19:00.617890 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=௷ queue_depth=3064 I0824 22:19:00.618146 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=௸ queue_depth=3065 I0824 22:19:00.618293 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=௹ queue_depth=3066 I0824 22:19:00.618417 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=௺ queue_depth=3067 I0824 22:19:00.618460 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=௻ queue_depth=3068 I0824 22:19:00.618592 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=௼ queue_depth=3069 I0824 22:19:00.618699 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=௽ queue_depth=3070 I0824 22:19:00.618828 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=௾ queue_depth=3071 I0824 22:19:00.618950 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=௿ queue_depth=3072 I0824 22:19:00.619098 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ఀ queue_depth=3073 I0824 22:19:00.619260 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ఁ queue_depth=3074 I0824 22:19:00.619386 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ం queue_depth=3075 I0824 22:19:00.619545 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ః queue_depth=3076 I0824 22:19:00.619725 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ఄ queue_depth=3077 I0824 22:19:00.619872 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=అ queue_depth=3078 I0824 22:19:00.619944 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ఆ queue_depth=3079 I0824 22:19:00.620046 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ఇ queue_depth=3080 I0824 22:19:00.620296 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ఈ queue_depth=3081 I0824 22:19:00.620459 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ఉ queue_depth=3082 I0824 22:19:00.620556 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ఊ queue_depth=3083 I0824 22:19:00.620618 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ఋ queue_depth=3084 I0824 22:19:00.620775 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ఌ queue_depth=3085 I0824 22:19:00.620931 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=఍ queue_depth=3086 I0824 22:19:00.621042 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ఎ queue_depth=3087 I0824 22:19:00.621096 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ఏ queue_depth=3088 I0824 22:19:00.621245 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ఐ queue_depth=3089 I0824 22:19:00.621365 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=఑ queue_depth=3090 I0824 22:19:00.621503 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ఒ queue_depth=3091 I0824 22:19:00.621548 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ఓ queue_depth=3092 I0824 22:19:00.621639 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ఔ queue_depth=3093 I0824 22:19:00.621779 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=క queue_depth=3094 I0824 22:19:00.621923 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ఖ queue_depth=3095 I0824 22:19:00.622100 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=గ queue_depth=3096 I0824 22:19:00.622430 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ఘ queue_depth=3097 I0824 22:19:00.622551 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ఙ queue_depth=3098 I0824 22:19:00.622701 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=చ queue_depth=3099 I0824 22:19:00.622889 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ఛ queue_depth=3100 I0824 22:19:00.622982 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=జ queue_depth=3101 I0824 22:19:00.623111 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ఝ queue_depth=3102 I0824 22:19:00.623244 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ఞ queue_depth=3103 I0824 22:19:00.624500 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ట queue_depth=3104 I0824 22:19:00.624725 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ఠ queue_depth=3105 I0824 22:19:00.624863 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=డ queue_depth=3106 I0824 22:19:00.624996 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ఢ queue_depth=3107 I0824 22:19:00.625199 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ణ queue_depth=3108 I0824 22:19:00.625442 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=త queue_depth=3109 I0824 22:19:00.625574 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=థ queue_depth=3110 I0824 22:19:00.625823 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ద queue_depth=3111 I0824 22:19:00.625973 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ధ queue_depth=3112 I0824 22:19:00.626119 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=న queue_depth=3113 I0824 22:19:00.626196 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=఩ queue_depth=3114 I0824 22:19:00.626356 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ప queue_depth=3115 I0824 22:19:00.626445 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ఫ queue_depth=3116 I0824 22:19:00.626534 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=బ queue_depth=3117 I0824 22:19:00.626671 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=భ queue_depth=3118 I0824 22:19:00.626828 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=మ queue_depth=3119 I0824 22:19:00.626874 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=య queue_depth=3120 I0824 22:19:00.627471 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ర queue_depth=3121 I0824 22:19:00.627602 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ఱ queue_depth=3122 I0824 22:19:00.627655 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ల queue_depth=3123 I0824 22:19:00.627711 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ళ queue_depth=3124 I0824 22:19:00.627771 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ఴ queue_depth=3125 I0824 22:19:00.627903 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=వ queue_depth=3126 I0824 22:19:00.628008 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=శ queue_depth=3127 I0824 22:19:00.628070 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ష queue_depth=3128 I0824 22:19:00.628121 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=స queue_depth=3129 I0824 22:19:00.628199 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=హ queue_depth=3130 I0824 22:19:00.628254 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=఺ queue_depth=3131 I0824 22:19:00.628298 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=఻ queue_depth=3132 I0824 22:19:00.628329 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=఼ queue_depth=3133 I0824 22:19:00.628368 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ఽ queue_depth=3134 I0824 22:19:00.628418 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ా queue_depth=3135 I0824 22:19:00.628466 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ి queue_depth=3136 I0824 22:19:00.628529 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ీ queue_depth=3137 I0824 22:19:00.628561 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ు queue_depth=3138 I0824 22:19:00.628616 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ూ queue_depth=3139 I0824 22:19:00.628674 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ృ queue_depth=3140 I0824 22:19:00.628767 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ౄ queue_depth=3141 I0824 22:19:00.628937 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=౅ queue_depth=3142 I0824 22:19:00.629018 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ె queue_depth=3143 I0824 22:19:00.629124 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ే queue_depth=3144 I0824 22:19:00.629178 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ై queue_depth=3145 I0824 22:19:00.629311 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=౉ queue_depth=3146 I0824 22:19:00.629421 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ొ queue_depth=3147 I0824 22:19:00.629530 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ో queue_depth=3148 I0824 22:19:00.629591 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ౌ queue_depth=3149 I0824 22:19:00.629632 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=్ queue_depth=3150 I0824 22:19:00.629674 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=౎ queue_depth=3151 I0824 22:19:00.629712 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=౏ queue_depth=3152 I0824 22:19:00.629834 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=౐ queue_depth=3153 I0824 22:19:00.629922 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=౑ queue_depth=3154 I0824 22:19:00.629992 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=౒ queue_depth=3155 I0824 22:19:00.630036 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=౓ queue_depth=3156 I0824 22:19:00.630099 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=౔ queue_depth=3157 I0824 22:19:00.630245 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ౕ queue_depth=3158 I0824 22:19:00.630353 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ౖ queue_depth=3159 I0824 22:19:00.630479 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=౗ queue_depth=3160 I0824 22:19:00.630598 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ౘ queue_depth=3161 I0824 22:19:00.630692 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ౙ queue_depth=3162 I0824 22:19:00.630755 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ౚ queue_depth=3163 I0824 22:19:00.630815 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=౛ queue_depth=3164 I0824 22:19:00.630870 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=౜ queue_depth=3165 I0824 22:19:00.630935 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ౝ queue_depth=3166 I0824 22:19:00.630984 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=౞ queue_depth=3167 I0824 22:19:00.631029 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=౟ queue_depth=3168 I0824 22:19:00.631064 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ౠ queue_depth=3169 I0824 22:19:00.631108 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ౡ queue_depth=3170 I0824 22:19:00.631166 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ౢ queue_depth=3171 I0824 22:19:00.631222 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ౣ queue_depth=3172 I0824 22:19:00.631289 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=౤ queue_depth=3173 I0824 22:19:00.631369 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=౥ queue_depth=3174 I0824 22:19:00.631493 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=౦ queue_depth=3175 I0824 22:19:00.631606 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=౧ queue_depth=3176 I0824 22:19:00.631707 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=౨ queue_depth=3177 I0824 22:19:00.631778 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=౩ queue_depth=3178 I0824 22:19:00.631822 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=౪ queue_depth=3179 I0824 22:19:00.631955 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=౫ queue_depth=3180 I0824 22:19:00.632060 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=౬ queue_depth=3181 I0824 22:19:00.632103 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=౭ queue_depth=3182 I0824 22:19:00.632155 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=౮ queue_depth=3183 I0824 22:19:00.632206 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=౯ queue_depth=3184 I0824 22:19:00.632258 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=౰ queue_depth=3185 I0824 22:19:00.632312 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=౱ queue_depth=3186 I0824 22:19:00.632380 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=౲ queue_depth=3187 I0824 22:19:00.632440 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=౳ queue_depth=3188 I0824 22:19:00.632475 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=౴ queue_depth=3189 I0824 22:19:00.632514 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=౵ queue_depth=3190 I0824 22:19:00.632572 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=౶ queue_depth=3191 I0824 22:19:00.632618 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=౷ queue_depth=3192 I0824 22:19:00.632676 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=౸ queue_depth=3193 I0824 22:19:00.632726 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=౹ queue_depth=3194 I0824 22:19:00.632782 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=౺ queue_depth=3195 I0824 22:19:00.632859 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=౻ queue_depth=3196 I0824 22:19:00.633012 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=౼ queue_depth=3197 I0824 22:19:00.633122 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=౽ queue_depth=3198 I0824 22:19:00.633188 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=౾ queue_depth=3199 I0824 22:19:00.633309 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=౿ queue_depth=3200 I0824 22:19:00.633450 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ಀ queue_depth=3201 I0824 22:19:00.633496 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ಁ queue_depth=3202 I0824 22:19:00.633544 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ಂ queue_depth=3203 I0824 22:19:00.633600 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ಃ queue_depth=3204 I0824 22:19:00.633686 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=಄ queue_depth=3205 I0824 22:19:00.633744 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ಅ queue_depth=3206 I0824 22:19:00.633784 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ಆ queue_depth=3207 I0824 22:19:00.633941 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ಇ queue_depth=3208 I0824 22:19:00.634064 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ಈ queue_depth=3209 I0824 22:19:00.634139 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ಉ queue_depth=3210 I0824 22:19:00.634186 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ಊ queue_depth=3211 I0824 22:19:00.634239 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ಋ queue_depth=3212 I0824 22:19:00.634293 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ಌ queue_depth=3213 I0824 22:19:00.634422 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=಍ queue_depth=3214 I0824 22:19:00.634483 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ಎ queue_depth=3215 I0824 22:19:00.634551 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ಏ queue_depth=3216 I0824 22:19:00.634677 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ಐ queue_depth=3217 I0824 22:19:00.634716 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=಑ queue_depth=3218 I0824 22:19:00.634788 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ಒ queue_depth=3219 I0824 22:19:00.634859 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ಓ queue_depth=3220 I0824 22:19:00.634912 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ಔ queue_depth=3221 I0824 22:19:00.634950 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ಕ queue_depth=3222 I0824 22:19:00.634982 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ಖ queue_depth=3223 I0824 22:19:00.635023 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ಗ queue_depth=3224 I0824 22:19:00.635065 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ಘ queue_depth=3225 I0824 22:19:00.635121 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ಙ queue_depth=3226 I0824 22:19:00.635159 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ಚ queue_depth=3227 I0824 22:19:00.635303 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ಛ queue_depth=3228 I0824 22:19:00.635393 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ಜ queue_depth=3229 I0824 22:19:00.635425 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ಝ queue_depth=3230 I0824 22:19:00.635454 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ಞ queue_depth=3231 I0824 22:19:00.635485 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ಟ queue_depth=3232 I0824 22:19:00.635517 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ಠ queue_depth=3233 I0824 22:19:00.635551 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ಡ queue_depth=3234 I0824 22:19:00.635595 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ಢ queue_depth=3235 I0824 22:19:00.635629 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ಣ queue_depth=3236 I0824 22:19:00.635655 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ತ queue_depth=3237 I0824 22:19:00.635689 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ಥ queue_depth=3238 I0824 22:19:00.635727 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ದ queue_depth=3239 I0824 22:19:00.635765 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ಧ queue_depth=3240 I0824 22:19:00.635852 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ನ queue_depth=3241 I0824 22:19:00.635929 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=಩ queue_depth=3242 I0824 22:19:00.635957 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ಪ queue_depth=3243 I0824 22:19:00.635983 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ಫ queue_depth=3244 I0824 22:19:00.636009 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ಬ queue_depth=3245 I0824 22:19:00.636032 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ಭ queue_depth=3246 I0824 22:19:00.636056 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ಮ queue_depth=3247 I0824 22:19:00.636081 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ಯ queue_depth=3248 I0824 22:19:00.636105 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ರ queue_depth=3249 I0824 22:19:00.636128 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ಱ queue_depth=3250 I0824 22:19:00.636149 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ಲ queue_depth=3251 I0824 22:19:00.636172 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ಳ queue_depth=3252 I0824 22:19:00.636196 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=಴ queue_depth=3253 I0824 22:19:00.636218 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ವ queue_depth=3254 I0824 22:19:00.636241 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ಶ queue_depth=3255 I0824 22:19:00.636263 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ಷ queue_depth=3256 I0824 22:19:00.636286 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ಸ queue_depth=3257 I0824 22:19:00.636311 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ಹ queue_depth=3258 I0824 22:19:00.636343 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=಺ queue_depth=3259 I0824 22:19:00.636370 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=಻ queue_depth=3260 I0824 22:19:00.636399 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=಼ queue_depth=3261 I0824 22:19:00.636425 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ಽ queue_depth=3262 I0824 22:19:00.636482 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ಾ queue_depth=3263 I0824 22:19:00.636511 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ಿ queue_depth=3264 I0824 22:19:00.636533 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ೀ queue_depth=3265 I0824 22:19:00.636560 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ು queue_depth=3266 I0824 22:19:00.636586 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ೂ queue_depth=3267 I0824 22:19:00.636618 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ೃ queue_depth=3268 I0824 22:19:00.636647 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ೄ queue_depth=3269 I0824 22:19:00.636691 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=೅ queue_depth=3270 I0824 22:19:00.636715 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ೆ queue_depth=3271 I0824 22:19:00.636740 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ೇ queue_depth=3272 I0824 22:19:00.636763 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ೈ queue_depth=3273 I0824 22:19:00.636810 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=೉ queue_depth=3274 I0824 22:19:00.636840 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ೊ queue_depth=3275 I0824 22:19:00.636867 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ೋ queue_depth=3276 I0824 22:19:00.636896 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ೌ queue_depth=3277 I0824 22:19:00.636932 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=್ queue_depth=3278 I0824 22:19:00.636991 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=೎ queue_depth=3279 I0824 22:19:00.637066 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=೏ queue_depth=3280 I0824 22:19:00.637095 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=೐ queue_depth=3281 I0824 22:19:00.637120 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=೑ queue_depth=3282 I0824 22:19:00.637148 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=೒ queue_depth=3283 I0824 22:19:00.637173 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=೓ queue_depth=3284 I0824 22:19:00.637204 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=೔ queue_depth=3285 I0824 22:19:00.637231 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ೕ queue_depth=3286 I0824 22:19:00.637259 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ೖ queue_depth=3287 I0824 22:19:00.637290 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=೗ queue_depth=3288 I0824 22:19:00.637315 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=೘ queue_depth=3289 I0824 22:19:00.637360 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=೙ queue_depth=3290 I0824 22:19:00.637385 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=೚ queue_depth=3291 I0824 22:19:00.637411 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=೛ queue_depth=3292 I0824 22:19:00.637443 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=೜ queue_depth=3293 I0824 22:19:00.637467 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ೝ queue_depth=3294 I0824 22:19:00.637491 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ೞ queue_depth=3295 I0824 22:19:00.637519 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=೟ queue_depth=3296 I0824 22:19:00.637545 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ೠ queue_depth=3297 I0824 22:19:00.637574 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ೡ queue_depth=3298 I0824 22:19:00.637602 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ೢ queue_depth=3299 I0824 22:19:00.637644 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ೣ queue_depth=3300 I0824 22:19:00.637672 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=೤ queue_depth=3301 I0824 22:19:00.637694 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=೥ queue_depth=3302 I0824 22:19:00.637721 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=೦ queue_depth=3303 I0824 22:19:00.637743 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=೧ queue_depth=3304 I0824 22:19:00.637768 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=೨ queue_depth=3305 I0824 22:19:00.637796 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=೩ queue_depth=3306 I0824 22:19:00.637872 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=೪ queue_depth=3307 I0824 22:19:00.637955 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=೫ queue_depth=3308 I0824 22:19:00.637984 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=೬ queue_depth=3309 I0824 22:19:00.638030 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=೭ queue_depth=3310 I0824 22:19:00.638059 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=೮ queue_depth=3311 I0824 22:19:00.638083 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=೯ queue_depth=3312 I0824 22:19:00.638106 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=೰ queue_depth=3313 I0824 22:19:00.638128 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ೱ queue_depth=3314 I0824 22:19:00.638152 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ೲ queue_depth=3315 I0824 22:19:00.638175 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ೳ queue_depth=3316 I0824 22:19:00.638198 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=೴ queue_depth=3317 I0824 22:19:00.638280 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=೵ queue_depth=3318 I0824 22:19:00.638316 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=೶ queue_depth=3319 I0824 22:19:00.638348 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=೷ queue_depth=3320 I0824 22:19:00.638379 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=೸ queue_depth=3321 I0824 22:19:00.638412 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=೹ queue_depth=3322 I0824 22:19:00.638444 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=೺ queue_depth=3323 I0824 22:19:00.638472 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=೻ queue_depth=3324 I0824 22:19:00.638514 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=೼ queue_depth=3325 I0824 22:19:00.638541 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=೽ queue_depth=3326 I0824 22:19:00.638565 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=೾ queue_depth=3327 I0824 22:19:00.638589 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=೿ queue_depth=3328 I0824 22:19:00.638614 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ഀ queue_depth=3329 I0824 22:19:00.638643 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ഁ queue_depth=3330 I0824 22:19:00.638672 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ം queue_depth=3331 I0824 22:19:00.638704 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ഃ queue_depth=3332 I0824 22:19:00.638733 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ഄ queue_depth=3333 I0824 22:19:00.638759 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=അ queue_depth=3334 I0824 22:19:00.638808 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ആ queue_depth=3335 I0824 22:19:00.638833 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ഇ queue_depth=3336 I0824 22:19:00.638857 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ഈ queue_depth=3337 I0824 22:19:00.638882 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ഉ queue_depth=3338 I0824 22:19:00.638911 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ഊ queue_depth=3339 I0824 22:19:00.638974 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ഋ queue_depth=3340 I0824 22:19:00.639009 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ഌ queue_depth=3341 I0824 22:19:00.639040 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=഍ queue_depth=3342 I0824 22:19:00.639088 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=എ queue_depth=3343 I0824 22:19:00.639125 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ഏ queue_depth=3344 I0824 22:19:00.639240 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ഐ queue_depth=3345 I0824 22:19:00.639318 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=഑ queue_depth=3346 I0824 22:19:00.639354 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ഒ queue_depth=3347 I0824 22:19:00.639382 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ഓ queue_depth=3348 I0824 22:19:00.639412 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ഔ queue_depth=3349 I0824 22:19:00.639441 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ക queue_depth=3350 I0824 22:19:00.639474 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ഖ queue_depth=3351 I0824 22:19:00.639516 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ഗ queue_depth=3352 I0824 22:19:00.639567 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ഘ queue_depth=3353 I0824 22:19:00.639604 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ങ queue_depth=3354 I0824 22:19:00.639631 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ച queue_depth=3355 I0824 22:19:00.639656 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ഛ queue_depth=3356 I0824 22:19:00.639678 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ജ queue_depth=3357 I0824 22:19:00.639704 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ഝ queue_depth=3358 I0824 22:19:00.639728 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ഞ queue_depth=3359 I0824 22:19:00.639749 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ട queue_depth=3360 I0824 22:19:00.639774 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ഠ queue_depth=3361 I0824 22:19:00.639798 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ഡ queue_depth=3362 I0824 22:19:00.639828 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ഢ queue_depth=3363 I0824 22:19:00.639868 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ണ queue_depth=3364 I0824 22:19:00.639892 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ത queue_depth=3365 I0824 22:19:00.639916 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ഥ queue_depth=3366 I0824 22:19:00.639944 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ദ queue_depth=3367 I0824 22:19:00.639972 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ധ queue_depth=3368 I0824 22:19:00.639996 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ന queue_depth=3369 I0824 22:19:00.640025 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ഩ queue_depth=3370 I0824 22:19:00.640053 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=പ queue_depth=3371 I0824 22:19:00.640079 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ഫ queue_depth=3372 I0824 22:19:00.640156 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ബ queue_depth=3373 I0824 22:19:00.640255 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ഭ queue_depth=3374 I0824 22:19:00.640291 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=മ queue_depth=3375 I0824 22:19:00.640321 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=യ queue_depth=3376 I0824 22:19:00.640348 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ര queue_depth=3377 I0824 22:19:00.640373 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=റ queue_depth=3378 I0824 22:19:00.640401 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ല queue_depth=3379 I0824 22:19:00.640429 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ള queue_depth=3380 I0824 22:19:00.640459 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ഴ queue_depth=3381 I0824 22:19:00.640485 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=വ queue_depth=3382 I0824 22:19:00.640516 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ശ queue_depth=3383 I0824 22:19:00.640562 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ഷ queue_depth=3384 I0824 22:19:00.640599 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=സ queue_depth=3385 I0824 22:19:00.640632 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ഹ queue_depth=3386 I0824 22:19:00.640661 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ഺ queue_depth=3387 I0824 22:19:00.640690 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=഻ queue_depth=3388 I0824 22:19:00.640719 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=഼ queue_depth=3389 I0824 22:19:00.640741 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ഽ queue_depth=3390 I0824 22:19:00.640766 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ാ queue_depth=3391 I0824 22:19:00.640826 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ി queue_depth=3392 I0824 22:19:00.640899 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ീ queue_depth=3393 I0824 22:19:00.640925 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ു queue_depth=3394 I0824 22:19:00.640949 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ൂ queue_depth=3395 I0824 22:19:00.640979 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ൃ queue_depth=3396 I0824 22:19:00.641003 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ൄ queue_depth=3397 I0824 22:19:00.641027 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=൅ queue_depth=3398 I0824 22:19:00.641052 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=െ queue_depth=3399 I0824 22:19:00.641075 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=േ queue_depth=3400 I0824 22:19:00.641103 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ൈ queue_depth=3401 I0824 22:19:00.641126 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=൉ queue_depth=3402 I0824 22:19:00.641161 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ൊ queue_depth=3403 I0824 22:19:00.641186 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ോ queue_depth=3404 I0824 22:19:00.641208 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ൌ queue_depth=3405 I0824 22:19:00.641252 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=് queue_depth=3406 I0824 22:19:00.641282 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ൎ queue_depth=3407 I0824 22:19:00.641307 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=൏ queue_depth=3408 I0824 22:19:00.641330 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=൐ queue_depth=3409 I0824 22:19:00.641357 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=൑ queue_depth=3410 I0824 22:19:00.641381 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=൒ queue_depth=3411 I0824 22:19:00.641404 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=൓ queue_depth=3412 I0824 22:19:00.641428 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ൔ queue_depth=3413 I0824 22:19:00.641465 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ൕ queue_depth=3414 I0824 22:19:00.641491 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ൖ queue_depth=3415 I0824 22:19:00.641516 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ൗ queue_depth=3416 I0824 22:19:00.641544 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=൘ queue_depth=3417 I0824 22:19:00.641575 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=൙ queue_depth=3418 I0824 22:19:00.641605 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=൚ queue_depth=3419 I0824 22:19:00.641632 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=൛ queue_depth=3420 I0824 22:19:00.641659 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=൜ queue_depth=3421 I0824 22:19:00.641687 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=൝ queue_depth=3422 I0824 22:19:00.641712 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=൞ queue_depth=3423 I0824 22:19:00.641740 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ൟ queue_depth=3424 I0824 22:19:00.641783 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ൠ queue_depth=3425 I0824 22:19:00.641813 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ൡ queue_depth=3426 I0824 22:19:00.641838 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ൢ queue_depth=3427 I0824 22:19:00.641899 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ൣ queue_depth=3428 I0824 22:19:00.641934 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=൤ queue_depth=3429 I0824 22:19:00.641966 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=൥ queue_depth=3430 I0824 22:19:00.641995 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=൦ queue_depth=3431 I0824 22:19:00.642024 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=൧ queue_depth=3432 I0824 22:19:00.642055 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=൨ queue_depth=3433 I0824 22:19:00.642101 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=൩ queue_depth=3434 I0824 22:19:00.642129 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=൪ queue_depth=3435 I0824 22:19:00.642155 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=൫ queue_depth=3436 I0824 22:19:00.642185 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=൬ queue_depth=3437 I0824 22:19:00.642208 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=൭ queue_depth=3438 I0824 22:19:00.642258 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=൮ queue_depth=3439 I0824 22:19:00.642881 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=൯ queue_depth=3440 I0824 22:19:00.642977 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=൰ queue_depth=3441 I0824 22:19:00.643017 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=൱ queue_depth=3442 I0824 22:19:00.643049 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=൲ queue_depth=3443 I0824 22:19:00.643075 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=൳ queue_depth=3444 I0824 22:19:00.643105 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=൴ queue_depth=3445 I0824 22:19:00.643161 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=൵ queue_depth=3446 I0824 22:19:00.643354 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=൶ queue_depth=3447 I0824 22:19:00.643453 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=൷ queue_depth=3448 I0824 22:19:00.643559 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=൸ queue_depth=3449 I0824 22:19:00.643599 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=൹ queue_depth=3450 I0824 22:19:00.643632 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ൺ queue_depth=3451 I0824 22:19:00.643685 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ൻ queue_depth=3452 I0824 22:19:00.643717 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ർ queue_depth=3453 I0824 22:19:00.643748 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ൽ queue_depth=3454 I0824 22:19:00.643778 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ൾ queue_depth=3455 I0824 22:19:00.643808 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ൿ queue_depth=3456 I0824 22:19:00.643840 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=඀ queue_depth=3457 I0824 22:19:00.643871 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ඁ queue_depth=3458 I0824 22:19:00.643900 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ං queue_depth=3459 I0824 22:19:00.643930 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ඃ queue_depth=3460 I0824 22:19:00.644135 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=඄ queue_depth=3461 I0824 22:19:00.644220 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=අ queue_depth=3462 I0824 22:19:00.644254 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ආ queue_depth=3463 I0824 22:19:00.644300 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ඇ queue_depth=3464 I0824 22:19:00.644333 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ඈ queue_depth=3465 I0824 22:19:00.644366 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ඉ queue_depth=3466 I0824 22:19:00.644397 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ඊ queue_depth=3467 I0824 22:19:00.644432 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=උ queue_depth=3468 I0824 22:19:00.644471 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ඌ queue_depth=3469 I0824 22:19:00.644504 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ඍ queue_depth=3470 I0824 22:19:00.644537 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ඎ queue_depth=3471 I0824 22:19:00.644678 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ඏ queue_depth=3472 I0824 22:19:00.644748 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ඐ queue_depth=3473 I0824 22:19:00.644816 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=එ queue_depth=3474 I0824 22:19:00.644844 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ඒ queue_depth=3475 I0824 22:19:00.644867 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ඓ queue_depth=3476 I0824 22:19:00.644892 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ඔ queue_depth=3477 I0824 22:19:00.644930 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ඕ queue_depth=3478 I0824 22:19:00.644956 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ඖ queue_depth=3479 I0824 22:19:00.644979 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=඗ queue_depth=3480 I0824 22:19:00.645000 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=඘ queue_depth=3481 I0824 22:19:00.645029 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=඙ queue_depth=3482 I0824 22:19:00.645079 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ක queue_depth=3483 I0824 22:19:00.645105 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ඛ queue_depth=3484 I0824 22:19:00.645128 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ග queue_depth=3485 I0824 22:19:00.645151 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ඝ queue_depth=3486 I0824 22:19:00.645174 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ඞ queue_depth=3487 I0824 22:19:00.645197 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ඟ queue_depth=3488 I0824 22:19:00.645221 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ච queue_depth=3489 I0824 22:19:00.645258 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ඡ queue_depth=3490 I0824 22:19:00.645286 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ජ queue_depth=3491 I0824 22:19:00.645820 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ඣ queue_depth=3492 I0824 22:19:00.645912 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ඤ queue_depth=3493 I0824 22:19:00.645939 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ඥ queue_depth=3494 I0824 22:19:00.645964 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ඦ queue_depth=3495 I0824 22:19:00.645990 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ට queue_depth=3496 I0824 22:19:00.646012 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ඨ queue_depth=3497 I0824 22:19:00.646036 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ඩ queue_depth=3498 I0824 22:19:00.646060 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ඪ queue_depth=3499 I0824 22:19:00.646082 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ණ queue_depth=3500 I0824 22:19:00.646106 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ඬ queue_depth=3501 I0824 22:19:00.646128 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ත queue_depth=3502 I0824 22:19:00.646153 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ථ queue_depth=3503 I0824 22:19:00.646190 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ද queue_depth=3504 I0824 22:19:00.646247 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ධ queue_depth=3505 I0824 22:19:00.646279 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=න queue_depth=3506 I0824 22:19:00.646304 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=඲ queue_depth=3507 I0824 22:19:00.646328 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ඳ queue_depth=3508 I0824 22:19:00.646351 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ප queue_depth=3509 I0824 22:19:00.646373 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ඵ queue_depth=3510 I0824 22:19:00.646420 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=බ queue_depth=3511 I0824 22:19:00.646446 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=භ queue_depth=3512 I0824 22:19:00.646484 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ම queue_depth=3513 I0824 22:19:00.646507 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ඹ queue_depth=3514 I0824 22:19:00.646536 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ය queue_depth=3515 I0824 22:19:00.646571 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ර queue_depth=3516 I0824 22:19:00.646604 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=඼ queue_depth=3517 I0824 22:19:00.646641 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ල queue_depth=3518 I0824 22:19:00.646666 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=඾ queue_depth=3519 I0824 22:19:00.646689 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=඿ queue_depth=3520 I0824 22:19:00.646712 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ව queue_depth=3521 I0824 22:19:00.646736 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ශ queue_depth=3522 I0824 22:19:00.646758 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ෂ queue_depth=3523 I0824 22:19:00.646797 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ස queue_depth=3524 I0824 22:19:00.646819 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=හ queue_depth=3525 I0824 22:19:00.646844 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ළ queue_depth=3526 I0824 22:19:00.646868 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ෆ queue_depth=3527 I0824 22:19:00.646890 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=෇ queue_depth=3528 I0824 22:19:00.646912 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=෈ queue_depth=3529 I0824 22:19:00.646933 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=෉ queue_depth=3530 I0824 22:19:00.646957 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=් queue_depth=3531 I0824 22:19:00.646980 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=෋ queue_depth=3532 I0824 22:19:00.647007 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=෌ queue_depth=3533 I0824 22:19:00.647038 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=෍ queue_depth=3534 I0824 22:19:00.647087 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=෎ queue_depth=3535 I0824 22:19:00.647144 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ා queue_depth=3536 I0824 22:19:00.647196 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ැ queue_depth=3537 I0824 22:19:00.647947 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ෑ queue_depth=3538 I0824 22:19:00.648035 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ි queue_depth=3539 I0824 22:19:00.648086 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ී queue_depth=3540 I0824 22:19:00.648116 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ු queue_depth=3541 I0824 22:19:00.648141 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=෕ queue_depth=3542 I0824 22:19:00.648166 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ූ queue_depth=3543 I0824 22:19:00.648197 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=෗ queue_depth=3544 I0824 22:19:00.648224 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ෘ queue_depth=3545 I0824 22:19:00.648254 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ෙ queue_depth=3546 I0824 22:19:00.648279 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ේ queue_depth=3547 I0824 22:19:00.648307 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ෛ queue_depth=3548 I0824 22:19:00.648353 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ො queue_depth=3549 I0824 22:19:00.648382 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ෝ queue_depth=3550 I0824 22:19:00.648403 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ෞ queue_depth=3551 I0824 22:19:00.648428 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ෟ queue_depth=3552 I0824 22:19:00.648454 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=෠ queue_depth=3553 I0824 22:19:00.648490 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=෡ queue_depth=3554 I0824 22:19:00.648516 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=෢ queue_depth=3555 I0824 22:19:00.648574 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=෣ queue_depth=3556 I0824 22:19:00.648606 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=෤ queue_depth=3557 I0824 22:19:00.648630 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=෥ queue_depth=3558 I0824 22:19:00.648657 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=෦ queue_depth=3559 I0824 22:19:00.648684 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=෧ queue_depth=3560 I0824 22:19:00.648712 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=෨ queue_depth=3561 I0824 22:19:00.648743 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=෩ queue_depth=3562 I0824 22:19:00.648773 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=෪ queue_depth=3563 I0824 22:19:00.648803 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=෫ queue_depth=3564 I0824 22:19:00.648837 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=෬ queue_depth=3565 I0824 22:19:00.648865 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=෭ queue_depth=3566 I0824 22:19:00.648906 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=෮ queue_depth=3567 I0824 22:19:00.648928 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=෯ queue_depth=3568 I0824 22:19:00.648950 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=෰ queue_depth=3569 I0824 22:19:00.648973 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=෱ queue_depth=3570 I0824 22:19:00.649020 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ෲ queue_depth=3571 I0824 22:19:00.649046 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ෳ queue_depth=3572 I0824 22:19:00.649068 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=෴ queue_depth=3573 I0824 22:19:00.649094 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=෵ queue_depth=3574 I0824 22:19:00.649118 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=෶ queue_depth=3575 I0824 22:19:00.649140 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=෷ queue_depth=3576 I0824 22:19:00.649162 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=෸ queue_depth=3577 I0824 22:19:00.649204 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=෹ queue_depth=3578 I0824 22:19:00.649233 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=෺ queue_depth=3579 I0824 22:19:00.649262 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=෻ queue_depth=3580 I0824 22:19:00.649286 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=෼ queue_depth=3581 I0824 22:19:00.649785 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=෽ queue_depth=3582 I0824 22:19:00.649860 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=෾ queue_depth=3583 I0824 22:19:00.649892 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=෿ queue_depth=3584 I0824 22:19:00.649925 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=฀ queue_depth=3585 I0824 22:19:00.649953 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ก queue_depth=3586 I0824 22:19:00.649977 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ข queue_depth=3587 I0824 22:19:00.650002 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ฃ queue_depth=3588 I0824 22:19:00.650029 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ค queue_depth=3589 I0824 22:19:00.650059 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ฅ queue_depth=3590 I0824 22:19:00.650099 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ฆ queue_depth=3591 I0824 22:19:00.650123 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ง queue_depth=3592 I0824 22:19:00.650183 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=จ queue_depth=3593 I0824 22:19:00.650214 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ฉ queue_depth=3594 I0824 22:19:00.650238 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ช queue_depth=3595 I0824 22:19:00.650262 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ซ queue_depth=3596 I0824 22:19:00.650284 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ฌ queue_depth=3597 I0824 22:19:00.650307 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ญ queue_depth=3598 I0824 22:19:00.650329 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ฎ queue_depth=3599 I0824 22:19:00.650351 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ฏ queue_depth=3600 I0824 22:19:00.650386 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ฐ queue_depth=3601 I0824 22:19:00.650409 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ฑ queue_depth=3602 I0824 22:19:00.650431 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ฒ queue_depth=3603 I0824 22:19:00.650497 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ณ queue_depth=3604 I0824 22:19:00.650525 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ด queue_depth=3605 I0824 22:19:00.650547 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ต queue_depth=3606 I0824 22:19:00.650572 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ถ queue_depth=3607 I0824 22:19:00.650593 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ท queue_depth=3608 I0824 22:19:00.650618 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ธ queue_depth=3609 I0824 22:19:00.650639 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=น queue_depth=3610 I0824 22:19:00.650663 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=บ queue_depth=3611 I0824 22:19:00.650700 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ป queue_depth=3612 I0824 22:19:00.650722 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ผ queue_depth=3613 I0824 22:19:00.650746 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ฝ queue_depth=3614 I0824 22:19:00.650789 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=พ queue_depth=3615 I0824 22:19:00.650816 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ฟ queue_depth=3616 I0824 22:19:00.650837 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ภ queue_depth=3617 I0824 22:19:00.650859 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ม queue_depth=3618 I0824 22:19:00.650881 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ย queue_depth=3619 I0824 22:19:00.650903 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ร queue_depth=3620 I0824 22:19:00.650926 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ฤ queue_depth=3621 I0824 22:19:00.650946 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ล queue_depth=3622 I0824 22:19:00.650991 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ฦ queue_depth=3623 I0824 22:19:00.651016 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ว queue_depth=3624 I0824 22:19:00.651038 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ศ queue_depth=3625 I0824 22:19:00.651062 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ษ queue_depth=3626 I0824 22:19:00.651083 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ส queue_depth=3627 I0824 22:19:00.651107 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ห queue_depth=3628 I0824 22:19:00.651129 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ฬ queue_depth=3629 I0824 22:19:00.651150 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=อ queue_depth=3630 I0824 22:19:00.651175 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ฮ queue_depth=3631 I0824 22:19:00.651196 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ฯ queue_depth=3632 I0824 22:19:00.651218 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ะ queue_depth=3633 I0824 22:19:00.651241 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ั queue_depth=3634 I0824 22:19:00.651264 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=า queue_depth=3635 I0824 22:19:00.651302 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ำ queue_depth=3636 I0824 22:19:00.651342 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ิ queue_depth=3637 I0824 22:19:00.651410 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ี queue_depth=3638 I0824 22:19:00.651437 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ึ queue_depth=3639 I0824 22:19:00.651460 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ื queue_depth=3640 I0824 22:19:00.651484 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ุ queue_depth=3641 I0824 22:19:00.651506 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ู queue_depth=3642 I0824 22:19:00.651529 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ฺ queue_depth=3643 I0824 22:19:00.651551 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=฻ queue_depth=3644 I0824 22:19:00.651573 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=฼ queue_depth=3645 I0824 22:19:00.651609 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=฽ queue_depth=3646 I0824 22:19:00.651631 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=฾ queue_depth=3647 I0824 22:19:00.651676 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=฿ queue_depth=3648 I0824 22:19:00.651704 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=เ queue_depth=3649 I0824 22:19:00.651727 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=แ queue_depth=3650 I0824 22:19:00.651750 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=โ queue_depth=3651 I0824 22:19:00.651772 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ใ queue_depth=3652 I0824 22:19:00.651795 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ไ queue_depth=3653 I0824 22:19:00.651816 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ๅ queue_depth=3654 I0824 22:19:00.651839 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ๆ queue_depth=3655 I0824 22:19:00.651861 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=็ queue_depth=3656 I0824 22:19:00.651896 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=่ queue_depth=3657 I0824 22:19:00.651920 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=้ queue_depth=3658 I0824 22:19:00.651941 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=๊ queue_depth=3659 I0824 22:19:00.651967 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=๋ queue_depth=3660 I0824 22:19:00.651998 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=์ queue_depth=3661 I0824 22:19:00.652026 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ํ queue_depth=3662 I0824 22:19:00.652055 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=๎ queue_depth=3663 I0824 22:19:00.652080 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=๏ queue_depth=3664 I0824 22:19:00.652106 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=๐ queue_depth=3665 I0824 22:19:00.652132 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=๑ queue_depth=3666 I0824 22:19:00.652162 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=๒ queue_depth=3667 I0824 22:19:00.652186 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=๓ queue_depth=3668 I0824 22:19:00.652213 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=๔ queue_depth=3669 I0824 22:19:00.652277 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=๕ queue_depth=3670 I0824 22:19:00.652308 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=๖ queue_depth=3671 I0824 22:19:00.652337 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=๗ queue_depth=3672 I0824 22:19:00.652363 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=๘ queue_depth=3673 I0824 22:19:00.652391 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=๙ queue_depth=3674 I0824 22:19:00.652418 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=๚ queue_depth=3675 I0824 22:19:00.652469 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=๛ queue_depth=3676 I0824 22:19:00.652504 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=๜ queue_depth=3677 I0824 22:19:00.652546 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=๝ queue_depth=3678 I0824 22:19:00.652569 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=๞ queue_depth=3679 I0824 22:19:00.652592 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=๟ queue_depth=3680 I0824 22:19:00.652613 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=๠ queue_depth=3681 I0824 22:19:00.652639 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=๡ queue_depth=3682 I0824 22:19:00.652662 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=๢ queue_depth=3683 I0824 22:19:00.652683 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=๣ queue_depth=3684 I0824 22:19:00.652706 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=๤ queue_depth=3685 I0824 22:19:00.652729 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=๥ queue_depth=3686 I0824 22:19:00.652751 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=๦ queue_depth=3687 I0824 22:19:00.652777 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=๧ queue_depth=3688 I0824 22:19:00.652799 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=๨ queue_depth=3689 I0824 22:19:00.652835 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=๩ queue_depth=3690 I0824 22:19:00.652858 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=๪ queue_depth=3691 I0824 22:19:00.652880 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=๫ queue_depth=3692 I0824 22:19:00.652905 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=๬ queue_depth=3693 I0824 22:19:00.652928 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=๭ queue_depth=3694 I0824 22:19:00.652955 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=๮ queue_depth=3695 I0824 22:19:00.652979 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=๯ queue_depth=3696 I0824 22:19:00.653006 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=๰ queue_depth=3697 I0824 22:19:00.653033 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=๱ queue_depth=3698 I0824 22:19:00.653055 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=๲ queue_depth=3699 I0824 22:19:00.653080 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=๳ queue_depth=3700 I0824 22:19:00.653166 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=๴ queue_depth=3701 I0824 22:19:00.653244 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=๵ queue_depth=3702 I0824 22:19:00.653449 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=๶ queue_depth=3703 I0824 22:19:00.653541 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=๷ queue_depth=3704 I0824 22:19:00.653583 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=๸ queue_depth=3705 I0824 22:19:00.653617 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=๹ queue_depth=3706 I0824 22:19:00.653659 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=๺ queue_depth=3707 I0824 22:19:00.653697 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=๻ queue_depth=3708 I0824 22:19:00.653732 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=๼ queue_depth=3709 I0824 22:19:00.653763 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=๽ queue_depth=3710 I0824 22:19:00.653802 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=๾ queue_depth=3711 I0824 22:19:00.653836 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=๿ queue_depth=3712 I0824 22:19:00.653866 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=຀ queue_depth=3713 I0824 22:19:00.653893 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ກ queue_depth=3714 I0824 22:19:00.653922 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ຂ queue_depth=3715 I0824 22:19:00.653973 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=຃ queue_depth=3716 I0824 22:19:00.654003 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ຄ queue_depth=3717 I0824 22:19:00.654031 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=຅ queue_depth=3718 I0824 22:19:00.654059 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ຆ queue_depth=3719 I0824 22:19:00.654126 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ງ queue_depth=3720 I0824 22:19:00.654194 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ຈ queue_depth=3721 I0824 22:19:00.654256 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ຉ queue_depth=3722 I0824 22:19:00.654287 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ຊ queue_depth=3723 I0824 22:19:00.654319 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=຋ queue_depth=3724 I0824 22:19:00.654348 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ຌ queue_depth=3725 I0824 22:19:00.654381 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ຍ queue_depth=3726 I0824 22:19:00.654414 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ຎ queue_depth=3727 I0824 22:19:00.654449 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ຏ queue_depth=3728 I0824 22:19:00.654484 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ຐ queue_depth=3729 I0824 22:19:00.654524 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ຑ queue_depth=3730 I0824 22:19:00.654561 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ຒ queue_depth=3731 I0824 22:19:00.654587 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ຓ queue_depth=3732 I0824 22:19:00.654612 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ດ queue_depth=3733 I0824 22:19:00.654633 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ຕ queue_depth=3734 I0824 22:19:00.654672 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ຖ queue_depth=3735 I0824 22:19:00.654718 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ທ queue_depth=3736 I0824 22:19:00.654749 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ຘ queue_depth=3737 I0824 22:19:00.654771 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ນ queue_depth=3738 I0824 22:19:00.654794 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ບ queue_depth=3739 I0824 22:19:00.654816 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ປ queue_depth=3740 I0824 22:19:00.654838 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ຜ queue_depth=3741 I0824 22:19:00.654862 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ຝ queue_depth=3742 I0824 22:19:00.654885 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ພ queue_depth=3743 I0824 22:19:00.654907 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ຟ queue_depth=3744 I0824 22:19:00.654933 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ຠ queue_depth=3745 I0824 22:19:00.654955 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ມ queue_depth=3746 I0824 22:19:00.654991 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ຢ queue_depth=3747 I0824 22:19:00.655014 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ຣ queue_depth=3748 I0824 22:19:00.655036 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=຤ queue_depth=3749 I0824 22:19:00.655058 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ລ queue_depth=3750 I0824 22:19:00.655080 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=຦ queue_depth=3751 I0824 22:19:00.655124 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ວ queue_depth=3752 I0824 22:19:00.655147 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ຨ queue_depth=3753 I0824 22:19:00.655171 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ຩ queue_depth=3754 I0824 22:19:00.655193 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ສ queue_depth=3755 I0824 22:19:00.655214 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ຫ queue_depth=3756 I0824 22:19:00.655237 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ຬ queue_depth=3757 I0824 22:19:00.655260 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ອ queue_depth=3758 I0824 22:19:00.655319 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ຮ queue_depth=3759 I0824 22:19:00.655346 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ຯ queue_depth=3760 I0824 22:19:00.655369 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ະ queue_depth=3761 I0824 22:19:00.655390 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ັ queue_depth=3762 I0824 22:19:00.655413 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=າ queue_depth=3763 I0824 22:19:00.655436 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ຳ queue_depth=3764 I0824 22:19:00.655458 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ິ queue_depth=3765 I0824 22:19:00.655482 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ີ queue_depth=3766 I0824 22:19:00.655503 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ຶ queue_depth=3767 I0824 22:19:00.655525 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ື queue_depth=3768 I0824 22:19:00.655566 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ຸ queue_depth=3769 I0824 22:19:00.655609 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ູ queue_depth=3770 I0824 22:19:00.655634 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=຺ queue_depth=3771 I0824 22:19:00.655654 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ົ queue_depth=3772 I0824 22:19:00.655677 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ຼ queue_depth=3773 I0824 22:19:00.655698 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ຽ queue_depth=3774 I0824 22:19:00.655719 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=຾ queue_depth=3775 I0824 22:19:00.655742 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=຿ queue_depth=3776 I0824 22:19:00.655763 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ເ queue_depth=3777 I0824 22:19:00.655787 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ແ queue_depth=3778 I0824 22:19:00.655810 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ໂ queue_depth=3779 I0824 22:19:00.655831 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ໃ queue_depth=3780 I0824 22:19:00.655854 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ໄ queue_depth=3781 I0824 22:19:00.655875 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=໅ queue_depth=3782 I0824 22:19:00.655927 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ໆ queue_depth=3783 I0824 22:19:00.655955 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=໇ queue_depth=3784 I0824 22:19:00.655980 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=່ queue_depth=3785 I0824 22:19:00.656004 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=້ queue_depth=3786 I0824 22:19:00.656025 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=໊ queue_depth=3787 I0824 22:19:00.656048 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=໋ queue_depth=3788 I0824 22:19:00.656071 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=໌ queue_depth=3789 I0824 22:19:00.656093 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ໍ queue_depth=3790 I0824 22:19:00.656116 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=໎ queue_depth=3791 I0824 22:19:00.656137 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=໏ queue_depth=3792 I0824 22:19:00.656161 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=໐ queue_depth=3793 I0824 22:19:00.656183 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=໑ queue_depth=3794 I0824 22:19:00.656204 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=໒ queue_depth=3795 I0824 22:19:00.656240 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=໓ queue_depth=3796 I0824 22:19:00.656263 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=໔ queue_depth=3797 I0824 22:19:00.656284 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=໕ queue_depth=3798 I0824 22:19:00.656306 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=໖ queue_depth=3799 I0824 22:19:00.656328 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=໗ queue_depth=3800 I0824 22:19:00.656351 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=໘ queue_depth=3801 I0824 22:19:00.656418 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=໙ queue_depth=3802 I0824 22:19:00.656445 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=໚ queue_depth=3803 I0824 22:19:00.656467 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=໛ queue_depth=3804 I0824 22:19:00.656489 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ໜ queue_depth=3805 I0824 22:19:00.656511 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ໝ queue_depth=3806 I0824 22:19:00.656548 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ໞ queue_depth=3807 I0824 22:19:00.656570 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ໟ queue_depth=3808 I0824 22:19:00.656592 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=໠ queue_depth=3809 I0824 22:19:00.656615 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=໡ queue_depth=3810 I0824 22:19:00.656637 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=໢ queue_depth=3811 I0824 22:19:00.656662 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=໣ queue_depth=3812 I0824 22:19:00.656705 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=໤ queue_depth=3813 I0824 22:19:00.656732 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=໥ queue_depth=3814 I0824 22:19:00.656754 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=໦ queue_depth=3815 I0824 22:19:00.656776 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=໧ queue_depth=3816 I0824 22:19:00.656799 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=໨ queue_depth=3817 I0824 22:19:00.656833 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=໩ queue_depth=3818 I0824 22:19:00.656857 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=໪ queue_depth=3819 I0824 22:19:00.656880 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=໫ queue_depth=3820 I0824 22:19:00.656902 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=໬ queue_depth=3821 I0824 22:19:00.656924 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=໭ queue_depth=3822 I0824 22:19:00.656947 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=໮ queue_depth=3823 I0824 22:19:00.656968 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=໯ queue_depth=3824 I0824 22:19:00.656991 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=໰ queue_depth=3825 I0824 22:19:00.657012 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=໱ queue_depth=3826 I0824 22:19:00.657041 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=໲ queue_depth=3827 I0824 22:19:00.657063 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=໳ queue_depth=3828 I0824 22:19:00.657085 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=໴ queue_depth=3829 I0824 22:19:00.657121 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=໵ queue_depth=3830 I0824 22:19:00.657143 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=໶ queue_depth=3831 I0824 22:19:00.657164 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=໷ queue_depth=3832 I0824 22:19:00.657186 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=໸ queue_depth=3833 I0824 22:19:00.657207 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=໹ queue_depth=3834 I0824 22:19:00.657246 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=໺ queue_depth=3835 I0824 22:19:00.657271 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=໻ queue_depth=3836 I0824 22:19:00.657296 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=໼ queue_depth=3837 I0824 22:19:00.657323 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=໽ queue_depth=3838 I0824 22:19:00.657344 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=໾ queue_depth=3839 I0824 22:19:00.657366 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=໿ queue_depth=3840 I0824 22:19:00.657404 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ༀ queue_depth=3841 I0824 22:19:00.657447 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=༁ queue_depth=3842 I0824 22:19:00.657473 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=༂ queue_depth=3843 I0824 22:19:00.657495 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=༃ queue_depth=3844 I0824 22:19:00.657516 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=༄ queue_depth=3845 I0824 22:19:00.657538 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=༅ queue_depth=3846 I0824 22:19:00.657563 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=༆ queue_depth=3847 I0824 22:19:00.657586 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=༇ queue_depth=3848 I0824 22:19:00.657614 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=༈ queue_depth=3849 I0824 22:19:00.657636 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=༉ queue_depth=3850 I0824 22:19:00.657658 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=༊ queue_depth=3851 I0824 22:19:00.657681 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=་ queue_depth=3852 I0824 22:19:00.657717 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=༌ queue_depth=3853 I0824 22:19:00.657740 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=། queue_depth=3854 I0824 22:19:00.657762 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=༎ queue_depth=3855 I0824 22:19:00.657784 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=༏ queue_depth=3856 I0824 22:19:00.657807 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=༐ queue_depth=3857 I0824 22:19:00.657831 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=༑ queue_depth=3858 I0824 22:19:00.657854 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=༒ queue_depth=3859 I0824 22:19:00.657876 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=༓ queue_depth=3860 I0824 22:19:00.657898 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=༔ queue_depth=3861 I0824 22:19:00.657921 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=༕ queue_depth=3862 I0824 22:19:00.657943 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=༖ queue_depth=3863 I0824 22:19:00.657966 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=༗ queue_depth=3864 I0824 22:19:00.658005 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=༘ queue_depth=3865 I0824 22:19:00.658030 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=༙ queue_depth=3866 I0824 22:19:00.658058 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=༚ queue_depth=3867 I0824 22:19:00.658101 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=༛ queue_depth=3868 I0824 22:19:00.658162 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=༜ queue_depth=3869 I0824 22:19:00.658210 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=༝ queue_depth=3870 I0824 22:19:00.658233 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=༞ queue_depth=3871 I0824 22:19:00.658254 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=༟ queue_depth=3872 I0824 22:19:00.658288 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=༠ queue_depth=3873 I0824 22:19:00.658311 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=༡ queue_depth=3874 I0824 22:19:00.658333 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=༢ queue_depth=3875 I0824 22:19:00.658356 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=༣ queue_depth=3876 I0824 22:19:00.658379 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=༤ queue_depth=3877 I0824 22:19:00.658404 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=༥ queue_depth=3878 I0824 22:19:00.658432 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=༦ queue_depth=3879 I0824 22:19:00.658455 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=༧ queue_depth=3880 I0824 22:19:00.658495 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=༨ queue_depth=3881 I0824 22:19:00.658522 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=༩ queue_depth=3882 I0824 22:19:00.658544 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=༪ queue_depth=3883 I0824 22:19:00.658609 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=༫ queue_depth=3884 I0824 22:19:00.658660 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=༬ queue_depth=3885 I0824 22:19:00.658684 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=༭ queue_depth=3886 I0824 22:19:00.658707 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=༮ queue_depth=3887 I0824 22:19:00.658728 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=༯ queue_depth=3888 I0824 22:19:00.658777 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=༰ queue_depth=3889 I0824 22:19:00.658802 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=༱ queue_depth=3890 I0824 22:19:00.658827 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=༲ queue_depth=3891 I0824 22:19:00.658849 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=༳ queue_depth=3892 I0824 22:19:00.658878 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=༴ queue_depth=3893 I0824 22:19:00.658904 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=༵ queue_depth=3894 I0824 22:19:00.658928 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=༶ queue_depth=3895 I0824 22:19:00.658951 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=༷ queue_depth=3896 I0824 22:19:00.658987 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=༸ queue_depth=3897 I0824 22:19:00.659009 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=༹ queue_depth=3898 I0824 22:19:00.659031 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=༺ queue_depth=3899 I0824 22:19:00.659054 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=༻ queue_depth=3900 I0824 22:19:00.659093 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=༼ queue_depth=3901 I0824 22:19:00.659119 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=༽ queue_depth=3902 I0824 22:19:00.659159 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=༾ queue_depth=3903 I0824 22:19:00.659186 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=༿ queue_depth=3904 I0824 22:19:00.659213 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ཀ queue_depth=3905 I0824 22:19:00.659241 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ཁ queue_depth=3906 I0824 22:19:00.659301 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ག queue_depth=3907 I0824 22:19:00.659325 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=གྷ queue_depth=3908 I0824 22:19:00.659349 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ང queue_depth=3909 I0824 22:19:00.659371 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ཅ queue_depth=3910 I0824 22:19:00.659393 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ཆ queue_depth=3911 I0824 22:19:00.659415 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ཇ queue_depth=3912 I0824 22:19:00.659436 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=཈ queue_depth=3913 I0824 22:19:00.659459 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ཉ queue_depth=3914 I0824 22:19:00.659480 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ཊ queue_depth=3915 I0824 22:19:00.659503 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ཋ queue_depth=3916 I0824 22:19:00.659990 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ཌ queue_depth=3917 I0824 22:19:00.660027 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ཌྷ queue_depth=3918 I0824 22:19:00.660055 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ཎ queue_depth=3919 I0824 22:19:00.660081 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ཏ queue_depth=3920 I0824 22:19:00.660103 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ཐ queue_depth=3921 I0824 22:19:00.660125 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ད queue_depth=3922 I0824 22:19:00.660147 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=དྷ queue_depth=3923 I0824 22:19:00.660194 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ན queue_depth=3924 I0824 22:19:00.660221 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=པ queue_depth=3925 I0824 22:19:00.660242 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ཕ queue_depth=3926 I0824 22:19:00.660273 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=བ queue_depth=3927 I0824 22:19:00.660301 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=བྷ queue_depth=3928 I0824 22:19:00.660335 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=མ queue_depth=3929 I0824 22:19:00.660361 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ཙ queue_depth=3930 I0824 22:19:00.660411 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ཚ queue_depth=3931 I0824 22:19:00.660432 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ཛ queue_depth=3932 I0824 22:19:00.660457 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ཛྷ queue_depth=3933 I0824 22:19:00.660512 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ཝ queue_depth=3934 I0824 22:19:00.660545 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ཞ queue_depth=3935 I0824 22:19:00.660588 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ཟ queue_depth=3936 I0824 22:19:00.660614 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=འ queue_depth=3937 I0824 22:19:00.660635 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ཡ queue_depth=3938 I0824 22:19:00.660668 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ར queue_depth=3939 I0824 22:19:00.660694 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ལ queue_depth=3940 I0824 22:19:00.660715 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ཤ queue_depth=3941 I0824 22:19:00.660742 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ཥ queue_depth=3942 I0824 22:19:00.660772 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ས queue_depth=3943 I0824 22:19:00.660799 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ཧ queue_depth=3944 I0824 22:19:00.660821 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ཨ queue_depth=3945 I0824 22:19:00.660866 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ཀྵ queue_depth=3946 I0824 22:19:00.660892 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ཪ queue_depth=3947 I0824 22:19:00.660915 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ཫ queue_depth=3948 I0824 22:19:00.660941 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ཬ queue_depth=3949 I0824 22:19:00.660962 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=཭ queue_depth=3950 I0824 22:19:00.661007 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=཮ queue_depth=3951 I0824 22:19:00.661035 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=཯ queue_depth=3952 I0824 22:19:00.661074 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=཰ queue_depth=3953 I0824 22:19:00.661102 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ཱ queue_depth=3954 I0824 22:19:00.661123 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ི queue_depth=3955 I0824 22:19:00.661170 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ཱི queue_depth=3956 I0824 22:19:00.661197 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ུ queue_depth=3957 I0824 22:19:00.661218 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ཱུ queue_depth=3958 I0824 22:19:00.661248 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ྲྀ queue_depth=3959 I0824 22:19:00.661273 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ཷ queue_depth=3960 I0824 22:19:00.661294 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ླྀ queue_depth=3961 I0824 22:19:00.661328 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ཹ queue_depth=3962 I0824 22:19:00.661355 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ེ queue_depth=3963 I0824 22:19:00.661380 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ཻ queue_depth=3964 I0824 22:19:00.661401 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ོ queue_depth=3965 I0824 22:19:00.661444 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ཽ queue_depth=3966 I0824 22:19:00.661499 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ཾ queue_depth=3967 I0824 22:19:00.661527 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ཿ queue_depth=3968 I0824 22:19:00.661550 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ྀ queue_depth=3969 I0824 22:19:00.661575 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ཱྀ queue_depth=3970 I0824 22:19:00.661597 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ྂ queue_depth=3971 I0824 22:19:00.661623 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ྃ queue_depth=3972 I0824 22:19:00.661652 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=྄ queue_depth=3973 I0824 22:19:00.661673 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=྅ queue_depth=3974 I0824 22:19:00.661694 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=྆ queue_depth=3975 I0824 22:19:00.661727 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=྇ queue_depth=3976 I0824 22:19:00.661754 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ྈ queue_depth=3977 I0824 22:19:00.661808 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ྉ queue_depth=3978 I0824 22:19:00.661837 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ྊ queue_depth=3979 I0824 22:19:00.661863 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ྋ queue_depth=3980 I0824 22:19:00.661889 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ྌ queue_depth=3981 I0824 22:19:00.661909 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ྍ queue_depth=3982 I0824 22:19:00.661942 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ྎ queue_depth=3983 I0824 22:19:00.661970 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ྏ queue_depth=3984 I0824 22:19:00.661996 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ྐ queue_depth=3985 I0824 22:19:00.662017 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ྑ queue_depth=3986 I0824 22:19:00.662043 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ྒ queue_depth=3987 I0824 22:19:00.662082 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ྒྷ queue_depth=3988 I0824 22:19:00.662106 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ྔ queue_depth=3989 I0824 22:19:00.662132 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ྕ queue_depth=3990 I0824 22:19:00.662153 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ྖ queue_depth=3991 I0824 22:19:00.662175 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ྗ queue_depth=3992 I0824 22:19:00.662202 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=྘ queue_depth=3993 I0824 22:19:00.662223 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ྙ queue_depth=3994 I0824 22:19:00.662248 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ྚ queue_depth=3995 I0824 22:19:00.662274 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ྛ queue_depth=3996 I0824 22:19:00.662300 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ྜ queue_depth=3997 I0824 22:19:00.662321 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ྜྷ queue_depth=3998 I0824 22:19:00.662348 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ྞ queue_depth=3999 I0824 22:19:00.662405 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ྟ queue_depth=4000 I0824 22:19:00.662434 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ྠ queue_depth=4001 I0824 22:19:00.662455 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ྡ queue_depth=4002 I0824 22:19:00.662488 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ྡྷ queue_depth=4003 I0824 22:19:00.662513 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ྣ queue_depth=4004 I0824 22:19:00.662535 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ྤ queue_depth=4005 I0824 22:19:00.662563 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ྥ queue_depth=4006 I0824 22:19:00.662616 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ྦ queue_depth=4007 I0824 22:19:00.662644 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ྦྷ queue_depth=4008 I0824 22:19:00.662670 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ྨ queue_depth=4009 I0824 22:19:00.662705 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ྩ queue_depth=4010 I0824 22:19:00.662731 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ྪ queue_depth=4011 I0824 22:19:00.662752 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ྫ queue_depth=4012 I0824 22:19:00.662774 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ྫྷ queue_depth=4013 I0824 22:19:00.662800 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ྭ queue_depth=4014 I0824 22:19:00.662822 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ྮ queue_depth=4015 I0824 22:19:00.662846 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ྯ queue_depth=4016 I0824 22:19:00.662872 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ྰ queue_depth=4017 I0824 22:19:00.662893 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ྱ queue_depth=4018 I0824 22:19:00.662914 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ྲ queue_depth=4019 I0824 22:19:00.662948 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ླ queue_depth=4020 I0824 22:19:00.662973 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ྴ queue_depth=4021 I0824 22:19:00.662995 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ྵ queue_depth=4022 I0824 22:19:00.663035 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ྶ queue_depth=4023 I0824 22:19:00.663057 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ྷ queue_depth=4024 I0824 22:19:00.663083 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ྸ queue_depth=4025 I0824 22:19:00.663109 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ྐྵ queue_depth=4026 I0824 22:19:00.663133 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ྺ queue_depth=4027 I0824 22:19:00.663173 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ྻ queue_depth=4028 I0824 22:19:00.663213 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=ྼ queue_depth=4029 I0824 22:19:00.663310 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=྽ queue_depth=4030 I0824 22:19:00.663423 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=྾ queue_depth=4031 I0824 22:19:00.663455 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=྿ queue_depth=4032 I0824 22:19:00.663513 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=࿀ queue_depth=4033 I0824 22:19:00.663543 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=࿁ queue_depth=4034 I0824 22:19:00.663579 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=࿂ queue_depth=4035 I0824 22:19:00.663606 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=࿃ queue_depth=4036 I0824 22:19:00.663643 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=࿄ queue_depth=4037 I0824 22:19:00.663676 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=࿅ queue_depth=4038 I0824 22:19:00.663719 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=࿆ queue_depth=4039 I0824 22:19:00.663751 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=࿇ queue_depth=4040 I0824 22:19:00.663779 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=࿈ queue_depth=4041 I0824 22:19:00.663809 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=࿉ queue_depth=4042 I0824 22:19:00.663840 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=࿊ queue_depth=4043 I0824 22:19:00.663865 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=࿋ queue_depth=4044 I0824 22:19:00.663893 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=࿌ queue_depth=4045 I0824 22:19:00.663924 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=࿍ queue_depth=4046 I0824 22:19:00.663952 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=࿎ queue_depth=4047 I0824 22:19:00.664038 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=࿏ queue_depth=4048 I0824 22:19:00.664076 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=࿐ queue_depth=4049 I0824 22:19:00.664113 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=࿑ queue_depth=4050 I0824 22:19:00.664144 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=࿒ queue_depth=4051 I0824 22:19:00.664186 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=࿓ queue_depth=4052 I0824 22:19:00.664222 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=࿔ queue_depth=4053 I0824 22:19:00.664267 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=࿕ queue_depth=4054 I0824 22:19:00.664300 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=࿖ queue_depth=4055 I0824 22:19:00.664335 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=࿗ queue_depth=4056 I0824 22:19:00.664375 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=࿘ queue_depth=4057 I0824 22:19:00.664407 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=࿙ queue_depth=4058 I0824 22:19:00.664429 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=࿚ queue_depth=4059 I0824 22:19:00.664451 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=࿛ queue_depth=4060 I0824 22:19:00.664472 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=࿜ queue_depth=4061 I0824 22:19:00.664493 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=࿝ queue_depth=4062 I0824 22:19:00.664514 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=࿞ queue_depth=4063 I0824 22:19:00.664537 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=࿟ queue_depth=4064 I0824 22:19:00.664572 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=࿠ queue_depth=4065 I0824 22:19:00.664619 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=࿡ queue_depth=4066 I0824 22:19:00.664654 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=࿢ queue_depth=4067 I0824 22:19:00.664679 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=࿣ queue_depth=4068 I0824 22:19:00.664705 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=࿤ queue_depth=4069 I0824 22:19:00.664733 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=࿥ queue_depth=4070 I0824 22:19:00.664755 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=࿦ queue_depth=4071 I0824 22:19:00.664776 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=࿧ queue_depth=4072 I0824 22:19:00.664797 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=࿨ queue_depth=4073 I0824 22:19:00.664825 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=࿩ queue_depth=4074 I0824 22:19:00.664864 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=࿪ queue_depth=4075 I0824 22:19:00.664888 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=࿫ queue_depth=4076 I0824 22:19:00.664949 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=࿬ queue_depth=4077 I0824 22:19:00.664978 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=࿭ queue_depth=4078 I0824 22:19:00.665004 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=࿮ queue_depth=4079 I0824 22:19:00.665025 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=࿯ queue_depth=4080 I0824 22:19:00.665047 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=࿰ queue_depth=4081 I0824 22:19:00.665076 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=࿱ queue_depth=4082 I0824 22:19:00.665101 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=࿲ queue_depth=4083 I0824 22:19:00.665122 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=࿳ queue_depth=4084 I0824 22:19:00.665146 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=࿴ queue_depth=4085 I0824 22:19:00.665186 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=࿵ queue_depth=4086 I0824 22:19:00.665210 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=࿶ queue_depth=4087 I0824 22:19:00.665235 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=࿷ queue_depth=4088 I0824 22:19:00.666090 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=࿸ queue_depth=4089 I0824 22:19:00.666174 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=࿹ queue_depth=4090 I0824 22:19:00.666216 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=࿺ queue_depth=4091 I0824 22:19:00.666253 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=࿻ queue_depth=4092 I0824 22:19:00.666278 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=࿼ queue_depth=4093 I0824 22:19:00.666300 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=࿽ queue_depth=4094 I0824 22:19:00.666325 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=࿾ queue_depth=4095 I0824 22:19:00.666356 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=࿿ queue_depth=4096 W0824 22:19:00.666388 s3api_versioning_reconciler.go:50 [versioning-heal] event=queue_full bucket=b key=က capacity=4096 (candidate dropped; read-path heal still covers it) W0824 22:19:00.666453 s3api_versioning_reconciler.go:50 [versioning-heal] event=queue_full bucket=b key=ခ capacity=4096 (candidate dropped; read-path heal still covers it) W0824 22:19:00.666542 s3api_versioning_reconciler.go:50 [versioning-heal] event=queue_full bucket=b key=ဂ capacity=4096 (candidate dropped; read-path heal still covers it) W0824 22:19:00.666576 s3api_versioning_reconciler.go:50 [versioning-heal] event=queue_full bucket=b key=ဃ capacity=4096 (candidate dropped; read-path heal still covers it) W0824 22:19:00.666630 s3api_versioning_reconciler.go:50 [versioning-heal] event=queue_full bucket=b key=င capacity=4096 (candidate dropped; read-path heal still covers it) W0824 22:19:00.666658 s3api_versioning_reconciler.go:50 [versioning-heal] event=queue_full bucket=b key=စ capacity=4096 (candidate dropped; read-path heal still covers it) W0824 22:19:00.666679 s3api_versioning_reconciler.go:50 [versioning-heal] event=queue_full bucket=b key=ဆ capacity=4096 (candidate dropped; read-path heal still covers it) W0824 22:19:00.666703 s3api_versioning_reconciler.go:50 [versioning-heal] event=queue_full bucket=b key=ဇ capacity=4096 (candidate dropped; read-path heal still covers it) W0824 22:19:00.666734 s3api_versioning_reconciler.go:50 [versioning-heal] event=queue_full bucket=b key=ဈ capacity=4096 (candidate dropped; read-path heal still covers it) W0824 22:19:00.666762 s3api_versioning_reconciler.go:50 [versioning-heal] event=queue_full bucket=b key=ဉ capacity=4096 (candidate dropped; read-path heal still covers it) W0824 22:19:00.666783 s3api_versioning_reconciler.go:50 [versioning-heal] event=queue_full bucket=b key=ည capacity=4096 (candidate dropped; read-path heal still covers it) W0824 22:19:00.666817 s3api_versioning_reconciler.go:50 [versioning-heal] event=queue_full bucket=b key=ဋ capacity=4096 (candidate dropped; read-path heal still covers it) W0824 22:19:00.666856 s3api_versioning_reconciler.go:50 [versioning-heal] event=queue_full bucket=b key=ဌ capacity=4096 (candidate dropped; read-path heal still covers it) W0824 22:19:00.666880 s3api_versioning_reconciler.go:50 [versioning-heal] event=queue_full bucket=b key=ဍ capacity=4096 (candidate dropped; read-path heal still covers it) W0824 22:19:00.666931 s3api_versioning_reconciler.go:50 [versioning-heal] event=queue_full bucket=b key=ဎ capacity=4096 (candidate dropped; read-path heal still covers it) W0824 22:19:00.666961 s3api_versioning_reconciler.go:50 [versioning-heal] event=queue_full bucket=b key=ဏ capacity=4096 (candidate dropped; read-path heal still covers it) W0824 22:19:00.666987 s3api_versioning_reconciler.go:50 [versioning-heal] event=queue_full bucket=b key=တ capacity=4096 (candidate dropped; read-path heal still covers it) W0824 22:19:00.667007 s3api_versioning_reconciler.go:50 [versioning-heal] event=queue_full bucket=b key=ထ capacity=4096 (candidate dropped; read-path heal still covers it) W0824 22:19:00.667028 s3api_versioning_reconciler.go:50 [versioning-heal] event=queue_full bucket=b key=ဒ capacity=4096 (candidate dropped; read-path heal still covers it) W0824 22:19:00.667049 s3api_versioning_reconciler.go:50 [versioning-heal] event=queue_full bucket=b key=ဓ capacity=4096 (candidate dropped; read-path heal still covers it) W0824 22:19:00.667074 s3api_versioning_reconciler.go:50 [versioning-heal] event=queue_full bucket=b key=န capacity=4096 (candidate dropped; read-path heal still covers it) W0824 22:19:00.667095 s3api_versioning_reconciler.go:50 [versioning-heal] event=queue_full bucket=b key=ပ capacity=4096 (candidate dropped; read-path heal still covers it) W0824 22:19:00.667115 s3api_versioning_reconciler.go:50 [versioning-heal] event=queue_full bucket=b key=ဖ capacity=4096 (candidate dropped; read-path heal still covers it) W0824 22:19:00.667141 s3api_versioning_reconciler.go:50 [versioning-heal] event=queue_full bucket=b key=ဗ capacity=4096 (candidate dropped; read-path heal still covers it) W0824 22:19:00.667175 s3api_versioning_reconciler.go:50 [versioning-heal] event=queue_full bucket=b key=ဘ capacity=4096 (candidate dropped; read-path heal still covers it) W0824 22:19:00.667200 s3api_versioning_reconciler.go:50 [versioning-heal] event=queue_full bucket=b key=မ capacity=4096 (candidate dropped; read-path heal still covers it) W0824 22:19:00.667221 s3api_versioning_reconciler.go:50 [versioning-heal] event=queue_full bucket=b key=ယ capacity=4096 (candidate dropped; read-path heal still covers it) W0824 22:19:00.667251 s3api_versioning_reconciler.go:50 [versioning-heal] event=queue_full bucket=b key=ရ capacity=4096 (candidate dropped; read-path heal still covers it) W0824 22:19:00.667277 s3api_versioning_reconciler.go:50 [versioning-heal] event=queue_full bucket=b key=လ capacity=4096 (candidate dropped; read-path heal still covers it) W0824 22:19:00.667298 s3api_versioning_reconciler.go:50 [versioning-heal] event=queue_full bucket=b key=ဝ capacity=4096 (candidate dropped; read-path heal still covers it) W0824 22:19:00.667328 s3api_versioning_reconciler.go:50 [versioning-heal] event=queue_full bucket=b key=သ capacity=4096 (candidate dropped; read-path heal still covers it) W0824 22:19:00.667354 s3api_versioning_reconciler.go:50 [versioning-heal] event=queue_full bucket=b key=ဟ capacity=4096 (candidate dropped; read-path heal still covers it) W0824 22:19:00.667374 s3api_versioning_reconciler.go:50 [versioning-heal] event=queue_full bucket=b key=ဠ capacity=4096 (candidate dropped; read-path heal still covers it) W0824 22:19:00.667394 s3api_versioning_reconciler.go:50 [versioning-heal] event=queue_full bucket=b key=အ capacity=4096 (candidate dropped; read-path heal still covers it) W0824 22:19:00.667419 s3api_versioning_reconciler.go:50 [versioning-heal] event=queue_full bucket=b key=ဢ capacity=4096 (candidate dropped; read-path heal still covers it) W0824 22:19:00.667457 s3api_versioning_reconciler.go:50 [versioning-heal] event=queue_full bucket=b key=ဣ capacity=4096 (candidate dropped; read-path heal still covers it) W0824 22:19:00.667495 s3api_versioning_reconciler.go:50 [versioning-heal] event=queue_full bucket=b key=ဤ capacity=4096 (candidate dropped; read-path heal still covers it) W0824 22:19:00.667515 s3api_versioning_reconciler.go:50 [versioning-heal] event=queue_full bucket=b key=ဥ capacity=4096 (candidate dropped; read-path heal still covers it) W0824 22:19:00.667548 s3api_versioning_reconciler.go:50 [versioning-heal] event=queue_full bucket=b key=ဦ capacity=4096 (candidate dropped; read-path heal still covers it) W0824 22:19:00.667580 s3api_versioning_reconciler.go:50 [versioning-heal] event=queue_full bucket=b key=ဧ capacity=4096 (candidate dropped; read-path heal still covers it) W0824 22:19:00.667612 s3api_versioning_reconciler.go:50 [versioning-heal] event=queue_full bucket=b key=ဨ capacity=4096 (candidate dropped; read-path heal still covers it) W0824 22:19:00.667646 s3api_versioning_reconciler.go:50 [versioning-heal] event=queue_full bucket=b key=ဩ capacity=4096 (candidate dropped; read-path heal still covers it) W0824 22:19:00.667674 s3api_versioning_reconciler.go:50 [versioning-heal] event=queue_full bucket=b key=ဪ capacity=4096 (candidate dropped; read-path heal still covers it) W0824 22:19:00.667701 s3api_versioning_reconciler.go:50 [versioning-heal] event=queue_full bucket=b key=ါ capacity=4096 (candidate dropped; read-path heal still covers it) W0824 22:19:00.667726 s3api_versioning_reconciler.go:50 [versioning-heal] event=queue_full bucket=b key=ာ capacity=4096 (candidate dropped; read-path heal still covers it) W0824 22:19:00.667746 s3api_versioning_reconciler.go:50 [versioning-heal] event=queue_full bucket=b key=ိ capacity=4096 (candidate dropped; read-path heal still covers it) W0824 22:19:00.667819 s3api_versioning_reconciler.go:50 [versioning-heal] event=queue_full bucket=b key=ီ capacity=4096 (candidate dropped; read-path heal still covers it) W0824 22:19:00.667881 s3api_versioning_reconciler.go:50 [versioning-heal] event=queue_full bucket=b key=ု capacity=4096 (candidate dropped; read-path heal still covers it) W0824 22:19:00.667903 s3api_versioning_reconciler.go:50 [versioning-heal] event=queue_full bucket=b key=ူ capacity=4096 (candidate dropped; read-path heal still covers it) W0824 22:19:00.667937 s3api_versioning_reconciler.go:50 [versioning-heal] event=queue_full bucket=b key=ေ capacity=4096 (candidate dropped; read-path heal still covers it) --- PASS: TestVersionsHealQueue_CapacityCap (0.47s) === RUN TestVersionsHealQueue_PopReadyOnlyDueItems I0824 22:19:00.668452 s3api_versioning_reconciler.go:46 [versioning-heal] event=enqueue bucket=b key=due queue_depth=1 --- PASS: TestVersionsHealQueue_PopReadyOnlyDueItems (0.00s) === RUN TestVersionsHealQueue_RequeueWithBackoff --- PASS: TestVersionsHealQueue_RequeueWithBackoff (0.00s) === RUN TestVersionsHealQueue_GiveUpAfterMaxAttempts W0824 22:19:00.668933 s3api_versioning_reconciler.go:50 [versioning-heal] event=gave_up bucket=b key=obj attempts=6 (read-path heal will still recover) --- PASS: TestVersionsHealQueue_GiveUpAfterMaxAttempts (0.00s) === RUN TestRetryFilerOp_SucceedsBeforeExhaustion --- PASS: TestRetryFilerOp_SucceedsBeforeExhaustion (0.31s) === RUN TestRetryFilerOp_PropagatesAfterExhaustion --- PASS: TestRetryFilerOp_PropagatesAfterExhaustion (3.14s) === RUN TestRetryFilerOp_ContextCancelInterruptsBackoff --- PASS: TestRetryFilerOp_ContextCancelInterruptsBackoff (0.04s) === RUN TestRetryFilerOp_TerminalErrorsShortCircuit === RUN TestRetryFilerOp_TerminalErrorsShortCircuit/filer_pb.ErrNotFound === RUN TestRetryFilerOp_TerminalErrorsShortCircuit/wrapped_filer_pb.ErrNotFound === RUN TestRetryFilerOp_TerminalErrorsShortCircuit/grpc_NotFound_status === RUN TestRetryFilerOp_TerminalErrorsShortCircuit/context_canceled === RUN TestRetryFilerOp_TerminalErrorsShortCircuit/context_deadline_exceeded --- PASS: TestRetryFilerOp_TerminalErrorsShortCircuit (0.00s) --- PASS: TestRetryFilerOp_TerminalErrorsShortCircuit/filer_pb.ErrNotFound (0.00s) --- PASS: TestRetryFilerOp_TerminalErrorsShortCircuit/wrapped_filer_pb.ErrNotFound (0.00s) --- PASS: TestRetryFilerOp_TerminalErrorsShortCircuit/grpc_NotFound_status (0.00s) --- PASS: TestRetryFilerOp_TerminalErrorsShortCircuit/context_canceled (0.00s) --- PASS: TestRetryFilerOp_TerminalErrorsShortCircuit/context_deadline_exceeded (0.00s) === RUN TestMultipartSSES3RealisticEndToEnd --- PASS: TestMultipartSSES3RealisticEndToEnd (6.20s) === RUN TestBuildMultipartSSES3Reader_LazyChunkFetch --- PASS: TestBuildMultipartSSES3Reader_LazyChunkFetch (0.01s) === RUN TestTrackAuditFallbackForDirectWriteHeader --- PASS: TestTrackAuditFallbackForDirectWriteHeader (0.00s) === RUN TestTrackAuditSkipsFallbackWhenHandlerEmits --- PASS: TestTrackAuditSkipsFallbackWhenHandlerEmits (0.00s) === RUN TestComputePackedPolicySize === RUN TestComputePackedPolicySize/empty_->_nil === RUN TestComputePackedPolicySize/tiny_policy_->_0% === RUN TestComputePackedPolicySize/half_budget_->_50% === RUN TestComputePackedPolicySize/full_budget_->_100% === RUN TestComputePackedPolicySize/oversized_->_capped_at_100 --- PASS: TestComputePackedPolicySize (0.00s) --- PASS: TestComputePackedPolicySize/empty_->_nil (0.00s) --- PASS: TestComputePackedPolicySize/tiny_policy_->_0% (0.00s) --- PASS: TestComputePackedPolicySize/half_budget_->_50% (0.00s) --- PASS: TestComputePackedPolicySize/full_budget_->_100% (0.00s) --- PASS: TestComputePackedPolicySize/oversized_->_capped_at_100 (0.00s) === RUN TestSTSAssumeRolePostBody === RUN TestSTSAssumeRolePostBody/ActionInQuery === RUN TestSTSAssumeRolePostBody/ActionInBody === RUN TestSTSAssumeRolePostBody/ActionInBodyWithSigV4Style --- PASS: TestSTSAssumeRolePostBody (0.06s) --- PASS: TestSTSAssumeRolePostBody/ActionInQuery (0.01s) --- PASS: TestSTSAssumeRolePostBody/ActionInBody (0.00s) --- PASS: TestSTSAssumeRolePostBody/ActionInBodyWithSigV4Style (0.00s) === RUN TestValidateRoleSessionName === RUN TestValidateRoleSessionName/empty_rejected === RUN TestValidateRoleSessionName/single_char_rejected_(below_min_len_2) === RUN TestValidateRoleSessionName/min_length_2_accepted === RUN TestValidateRoleSessionName/plain_ascii_accepted === RUN TestValidateRoleSessionName/all_special_chars_allowed === RUN TestValidateRoleSessionName/email-style_accepted === RUN TestValidateRoleSessionName/max_length_64_accepted === RUN TestValidateRoleSessionName/max_length_64_valid_charset_accepted === RUN TestValidateRoleSessionName/length_65_rejected === RUN TestValidateRoleSessionName/space_rejected === RUN TestValidateRoleSessionName/slash_rejected === RUN TestValidateRoleSessionName/colon_rejected === RUN TestValidateRoleSessionName/unicode_rejected --- PASS: TestValidateRoleSessionName (0.01s) --- PASS: TestValidateRoleSessionName/empty_rejected (0.00s) --- PASS: TestValidateRoleSessionName/single_char_rejected_(below_min_len_2) (0.00s) --- PASS: TestValidateRoleSessionName/min_length_2_accepted (0.00s) --- PASS: TestValidateRoleSessionName/plain_ascii_accepted (0.00s) --- PASS: TestValidateRoleSessionName/all_special_chars_allowed (0.00s) --- PASS: TestValidateRoleSessionName/email-style_accepted (0.00s) --- PASS: TestValidateRoleSessionName/max_length_64_accepted (0.00s) --- PASS: TestValidateRoleSessionName/max_length_64_valid_charset_accepted (0.00s) --- PASS: TestValidateRoleSessionName/length_65_rejected (0.00s) --- PASS: TestValidateRoleSessionName/space_rejected (0.00s) --- PASS: TestValidateRoleSessionName/slash_rejected (0.00s) --- PASS: TestValidateRoleSessionName/colon_rejected (0.00s) --- PASS: TestValidateRoleSessionName/unicode_rejected (0.00s) === RUN TestParseTagsHeader === RUN TestParseTagsHeader/simple_tags === RUN TestParseTagsHeader/URL_encoded_timestamp_-_issue_#7040_scenario === RUN TestParseTagsHeader/URL_encoded_key_and_value === RUN TestParseTagsHeader/empty_value === RUN TestParseTagsHeader/special_characters_encoded === RUN TestParseTagsHeader/invalid_URL_encoding === RUN TestParseTagsHeader/plus_signs_and_equals_in_values === RUN TestParseTagsHeader/unescaped_equals_in_value --- PASS: TestParseTagsHeader (0.01s) --- PASS: TestParseTagsHeader/simple_tags (0.00s) --- PASS: TestParseTagsHeader/URL_encoded_timestamp_-_issue_#7040_scenario (0.00s) --- PASS: TestParseTagsHeader/URL_encoded_key_and_value (0.00s) --- PASS: TestParseTagsHeader/empty_value (0.00s) --- PASS: TestParseTagsHeader/special_characters_encoded (0.00s) --- PASS: TestParseTagsHeader/invalid_URL_encoding (0.00s) --- PASS: TestParseTagsHeader/plus_signs_and_equals_in_values (0.00s) --- PASS: TestParseTagsHeader/unescaped_equals_in_value (0.00s) PASS ok github.com/seaweedfs/seaweedfs/weed/s3api 61.517s === RUN TestValidateConfiguration === RUN TestValidateConfiguration/nil_config === RUN TestValidateConfiguration/empty_rules === RUN TestValidateConfiguration/valid_single_rule === RUN TestValidateConfiguration/too_many_rules === RUN TestValidateConfiguration/invalid_method === RUN TestValidateConfiguration/empty_origins === RUN TestValidateConfiguration/invalid_origin_with_multiple_wildcards === RUN TestValidateConfiguration/negative_MaxAgeSeconds --- PASS: TestValidateConfiguration (0.04s) --- PASS: TestValidateConfiguration/nil_config (0.00s) --- PASS: TestValidateConfiguration/empty_rules (0.00s) --- PASS: TestValidateConfiguration/valid_single_rule (0.00s) --- PASS: TestValidateConfiguration/too_many_rules (0.00s) --- PASS: TestValidateConfiguration/invalid_method (0.00s) --- PASS: TestValidateConfiguration/empty_origins (0.00s) --- PASS: TestValidateConfiguration/invalid_origin_with_multiple_wildcards (0.00s) --- PASS: TestValidateConfiguration/negative_MaxAgeSeconds (0.00s) === RUN TestValidateOrigin === RUN TestValidateOrigin/empty_origin === RUN TestValidateOrigin/valid_origin === RUN TestValidateOrigin/wildcard_origin === RUN TestValidateOrigin/valid_wildcard_origin === RUN TestValidateOrigin/https_wildcard_origin === RUN TestValidateOrigin/invalid_wildcard_origin === RUN TestValidateOrigin/multiple_wildcards --- PASS: TestValidateOrigin (0.00s) --- PASS: TestValidateOrigin/empty_origin (0.00s) --- PASS: TestValidateOrigin/valid_origin (0.00s) --- PASS: TestValidateOrigin/wildcard_origin (0.00s) --- PASS: TestValidateOrigin/valid_wildcard_origin (0.00s) --- PASS: TestValidateOrigin/https_wildcard_origin (0.00s) --- PASS: TestValidateOrigin/invalid_wildcard_origin (0.00s) --- PASS: TestValidateOrigin/multiple_wildcards (0.00s) === RUN TestParseRequest === RUN TestParseRequest/simple_GET_request === RUN TestParseRequest/OPTIONS_preflight_request === RUN TestParseRequest/request_without_origin --- PASS: TestParseRequest (0.00s) --- PASS: TestParseRequest/simple_GET_request (0.00s) --- PASS: TestParseRequest/OPTIONS_preflight_request (0.00s) --- PASS: TestParseRequest/request_without_origin (0.00s) === RUN TestMatchesOrigin === RUN TestMatchesOrigin/wildcard_match === RUN TestMatchesOrigin/exact_match === RUN TestMatchesOrigin/no_match === RUN TestMatchesOrigin/wildcard_subdomain_match === RUN TestMatchesOrigin/wildcard_subdomain_no_match === RUN TestMatchesOrigin/multiple_origins_with_match === RUN TestMatchesOrigin/https_exact_match === RUN TestMatchesOrigin/https_no_match === RUN TestMatchesOrigin/https_wildcard_subdomain_match === RUN TestMatchesOrigin/https_wildcard_subdomain_no_match_-_base_domain === RUN TestMatchesOrigin/https_wildcard_subdomain_no_match_-_different_domain === RUN TestMatchesOrigin/protocol_mismatch_-_http_pattern_https_origin === RUN TestMatchesOrigin/protocol_mismatch_-_https_pattern_http_origin --- PASS: TestMatchesOrigin (0.00s) --- PASS: TestMatchesOrigin/wildcard_match (0.00s) --- PASS: TestMatchesOrigin/exact_match (0.00s) --- PASS: TestMatchesOrigin/no_match (0.00s) --- PASS: TestMatchesOrigin/wildcard_subdomain_match (0.00s) --- PASS: TestMatchesOrigin/wildcard_subdomain_no_match (0.00s) --- PASS: TestMatchesOrigin/multiple_origins_with_match (0.00s) --- PASS: TestMatchesOrigin/https_exact_match (0.00s) --- PASS: TestMatchesOrigin/https_no_match (0.00s) --- PASS: TestMatchesOrigin/https_wildcard_subdomain_match (0.00s) --- PASS: TestMatchesOrigin/https_wildcard_subdomain_no_match_-_base_domain (0.00s) --- PASS: TestMatchesOrigin/https_wildcard_subdomain_no_match_-_different_domain (0.00s) --- PASS: TestMatchesOrigin/protocol_mismatch_-_http_pattern_https_origin (0.00s) --- PASS: TestMatchesOrigin/protocol_mismatch_-_https_pattern_http_origin (0.00s) === RUN TestMatchesHeader === RUN TestMatchesHeader/empty_allowed_headers === RUN TestMatchesHeader/wildcard_match === RUN TestMatchesHeader/exact_match === RUN TestMatchesHeader/case_insensitive_match === RUN TestMatchesHeader/no_match === RUN TestMatchesHeader/wildcard_prefix_match --- PASS: TestMatchesHeader (0.00s) --- PASS: TestMatchesHeader/empty_allowed_headers (0.00s) --- PASS: TestMatchesHeader/wildcard_match (0.00s) --- PASS: TestMatchesHeader/exact_match (0.00s) --- PASS: TestMatchesHeader/case_insensitive_match (0.00s) --- PASS: TestMatchesHeader/no_match (0.00s) --- PASS: TestMatchesHeader/wildcard_prefix_match (0.00s) === RUN TestEvaluateRequest === RUN TestEvaluateRequest/matching_first_rule === RUN TestEvaluateRequest/matching_second_rule === RUN TestEvaluateRequest/no_matching_rule === RUN TestEvaluateRequest/preflight_request === RUN TestEvaluateRequest/preflight_request_with_forbidden_header === RUN TestEvaluateRequest/request_without_origin --- PASS: TestEvaluateRequest (0.00s) --- PASS: TestEvaluateRequest/matching_first_rule (0.00s) --- PASS: TestEvaluateRequest/matching_second_rule (0.00s) --- PASS: TestEvaluateRequest/no_matching_rule (0.00s) --- PASS: TestEvaluateRequest/preflight_request (0.00s) --- PASS: TestEvaluateRequest/preflight_request_with_forbidden_header (0.00s) --- PASS: TestEvaluateRequest/request_without_origin (0.00s) === RUN TestApplyHeaders === RUN TestApplyHeaders/nil_response === RUN TestApplyHeaders/complete_response === RUN TestApplyHeaders/with_credentials --- PASS: TestApplyHeaders (0.00s) --- PASS: TestApplyHeaders/nil_response (0.00s) --- PASS: TestApplyHeaders/complete_response (0.00s) --- PASS: TestApplyHeaders/with_credentials (0.00s) === RUN TestMiddlewareNonExistentBucket === RUN TestMiddlewareNonExistentBucket/Preflight_request_to_non-existent_bucket_with_global_CORS_config === RUN TestMiddlewareNonExistentBucket/Actual_request_to_non-existent_bucket_with_global_CORS_config === RUN TestMiddlewareNonExistentBucket/Preflight_to_non-existent_bucket_with_specific_origin === RUN TestMiddlewareNonExistentBucket/Preflight_to_non-existent_bucket_with_non-matching_origin === RUN TestMiddlewareNonExistentBucket/Preflight_to_non-existent_bucket_without_CORS_config === RUN TestMiddlewareNonExistentBucket/Bucket_listing_request_to_non-existent_bucket_with_CORS --- PASS: TestMiddlewareNonExistentBucket (0.01s) --- PASS: TestMiddlewareNonExistentBucket/Preflight_request_to_non-existent_bucket_with_global_CORS_config (0.00s) --- PASS: TestMiddlewareNonExistentBucket/Actual_request_to_non-existent_bucket_with_global_CORS_config (0.00s) --- PASS: TestMiddlewareNonExistentBucket/Preflight_to_non-existent_bucket_with_specific_origin (0.00s) --- PASS: TestMiddlewareNonExistentBucket/Preflight_to_non-existent_bucket_with_non-matching_origin (0.01s) --- PASS: TestMiddlewareNonExistentBucket/Preflight_to_non-existent_bucket_without_CORS_config (0.00s) --- PASS: TestMiddlewareNonExistentBucket/Bucket_listing_request_to_non-existent_bucket_with_CORS (0.00s) === RUN TestMiddlewareConsistentBehavior === RUN TestMiddlewareConsistentBehavior/existing_bucket === RUN TestMiddlewareConsistentBehavior/non-existent_bucket --- PASS: TestMiddlewareConsistentBehavior (0.00s) --- PASS: TestMiddlewareConsistentBehavior/existing_bucket (0.00s) --- PASS: TestMiddlewareConsistentBehavior/non-existent_bucket (0.00s) === RUN TestMiddlewareFallbackConfig === RUN TestMiddlewareFallbackConfig/No_bucket_config,_fallback_to_global_config_with_wildcard === RUN TestMiddlewareFallbackConfig/No_bucket_config,_fallback_to_global_config_with_specific_origin === RUN TestMiddlewareFallbackConfig/No_bucket_config,_fallback_rejects_non-matching_origin === RUN TestMiddlewareFallbackConfig/Bucket_config_takes_precedence_over_fallback === RUN TestMiddlewareFallbackConfig/Bucket_config_rejects,_even_though_fallback_would_allow === RUN TestMiddlewareFallbackConfig/No_config_at_all,_no_CORS_headers === RUN TestMiddlewareFallbackConfig/OPTIONS_preflight_with_fallback_config === RUN TestMiddlewareFallbackConfig/OPTIONS_preflight_without_any_config_should_fail --- PASS: TestMiddlewareFallbackConfig (0.01s) --- PASS: TestMiddlewareFallbackConfig/No_bucket_config,_fallback_to_global_config_with_wildcard (0.00s) --- PASS: TestMiddlewareFallbackConfig/No_bucket_config,_fallback_to_global_config_with_specific_origin (0.00s) --- PASS: TestMiddlewareFallbackConfig/No_bucket_config,_fallback_rejects_non-matching_origin (0.00s) --- PASS: TestMiddlewareFallbackConfig/Bucket_config_takes_precedence_over_fallback (0.00s) --- PASS: TestMiddlewareFallbackConfig/Bucket_config_rejects,_even_though_fallback_would_allow (0.00s) --- PASS: TestMiddlewareFallbackConfig/No_config_at_all,_no_CORS_headers (0.00s) --- PASS: TestMiddlewareFallbackConfig/OPTIONS_preflight_with_fallback_config (0.00s) --- PASS: TestMiddlewareFallbackConfig/OPTIONS_preflight_without_any_config_should_fail (0.00s) === RUN TestMiddlewareFallbackConfigWithMultipleOrigins === RUN TestMiddlewareFallbackConfigWithMultipleOrigins/First_allowed_origin_should_match === RUN TestMiddlewareFallbackConfigWithMultipleOrigins/Second_allowed_origin_should_match === RUN TestMiddlewareFallbackConfigWithMultipleOrigins/Non-allowed_origin_should_not_match --- PASS: TestMiddlewareFallbackConfigWithMultipleOrigins (0.00s) --- PASS: TestMiddlewareFallbackConfigWithMultipleOrigins/First_allowed_origin_should_match (0.00s) --- PASS: TestMiddlewareFallbackConfigWithMultipleOrigins/Second_allowed_origin_should_match (0.00s) --- PASS: TestMiddlewareFallbackConfigWithMultipleOrigins/Non-allowed_origin_should_not_match (0.00s) === RUN TestMiddlewareFallbackWithError === RUN TestMiddlewareFallbackWithError/ErrAccessDenied_should_not_trigger_fallback === RUN TestMiddlewareFallbackWithError/ErrInternalError_should_not_trigger_fallback === RUN TestMiddlewareFallbackWithError/ErrNoSuchBucket_should_trigger_fallback === RUN TestMiddlewareFallbackWithError/ErrNoSuchCORSConfiguration_should_trigger_fallback --- PASS: TestMiddlewareFallbackWithError (0.00s) --- PASS: TestMiddlewareFallbackWithError/ErrAccessDenied_should_not_trigger_fallback (0.00s) --- PASS: TestMiddlewareFallbackWithError/ErrInternalError_should_not_trigger_fallback (0.00s) --- PASS: TestMiddlewareFallbackWithError/ErrNoSuchBucket_should_trigger_fallback (0.00s) --- PASS: TestMiddlewareFallbackWithError/ErrNoSuchCORSConfiguration_should_trigger_fallback (0.00s) === RUN TestMiddlewareVaryHeader === RUN TestMiddlewareVaryHeader/Specific_allowed_origin === RUN TestMiddlewareVaryHeader/Wildcard_allowed_origin === RUN TestMiddlewareVaryHeader/No_CORS_configuration --- PASS: TestMiddlewareVaryHeader (0.00s) --- PASS: TestMiddlewareVaryHeader/Specific_allowed_origin (0.00s) --- PASS: TestMiddlewareVaryHeader/Wildcard_allowed_origin (0.00s) --- PASS: TestMiddlewareVaryHeader/No_CORS_configuration (0.00s) === RUN TestHandleOptionsRequestVaryHeader --- PASS: TestHandleOptionsRequestVaryHeader (0.00s) PASS ok github.com/seaweedfs/seaweedfs/weed/s3api/cors 11.353s === RUN TestHandleReportMetricsAcceptsScanReport --- PASS: TestHandleReportMetricsAcceptsScanReport (0.04s) === RUN TestHandleReportMetricsRejectsGarbage --- PASS: TestHandleReportMetricsRejectsGarbage (0.00s) === RUN TestHandleReportMetricsRequiresReportType --- PASS: TestHandleReportMetricsRequiresReportType (0.00s) === RUN TestHandleReportMetricsAcceptsOversizedReport --- PASS: TestHandleReportMetricsAcceptsOversizedReport (0.16s) === RUN TestHandleReportMetricsRequiresTable --- PASS: TestHandleReportMetricsRequiresTable (0.00s) === RUN TestHandleOAuthTokens_Success --- PASS: TestHandleOAuthTokens_Success (0.02s) === RUN TestHandleOAuthTokens_InvalidCredentials --- PASS: TestHandleOAuthTokens_InvalidCredentials (0.00s) === RUN TestHandleOAuthTokens_UnsupportedGrantType --- PASS: TestHandleOAuthTokens_UnsupportedGrantType (0.00s) === RUN TestBearerTokenRoundTrip --- PASS: TestBearerTokenRoundTrip (0.02s) === RUN TestBearerTokenInvalid --- PASS: TestBearerTokenInvalid (0.00s) === RUN TestBearerTokenNone --- PASS: TestBearerTokenNone (0.00s) === RUN TestWantsVendedCredentials === RUN TestWantsVendedCredentials/no_header === RUN TestWantsVendedCredentials/vended_credentials === RUN TestWantsVendedCredentials/mechanism_list === RUN TestWantsVendedCredentials/spaced_and_mixed_case === RUN TestWantsVendedCredentials/repeated_header === RUN TestWantsVendedCredentials/remote_signing_only --- PASS: TestWantsVendedCredentials (0.01s) --- PASS: TestWantsVendedCredentials/no_header (0.00s) --- PASS: TestWantsVendedCredentials/vended_credentials (0.00s) --- PASS: TestWantsVendedCredentials/mechanism_list (0.00s) --- PASS: TestWantsVendedCredentials/spaced_and_mixed_case (0.00s) --- PASS: TestWantsVendedCredentials/repeated_header (0.00s) --- PASS: TestWantsVendedCredentials/remote_signing_only (0.00s) === RUN TestBuildFileIOConfigWithholdsEndpointFromCredentialVendingClients --- PASS: TestBuildFileIOConfigWithholdsEndpointFromCredentialVendingClients (0.00s) === RUN TestWriteLoadResultVariesOnTheDelegationHeader --- PASS: TestWriteLoadResultVariesOnTheDelegationHeader (0.00s) === RUN TestLoadTableResultOmitsConfigForCredentialVendingClients --- PASS: TestLoadTableResultOmitsConfigForCredentialVendingClients (0.10s) === RUN TestParseCommitUpdatesSeparatesStatistics --- PASS: TestParseCommitUpdatesSeparatesStatistics (0.00s) === RUN TestParseCommitUpdatesRejectsIncompleteSetStatistics --- PASS: TestParseCommitUpdatesRejectsIncompleteSetStatistics (0.00s) === RUN TestApplyStatisticsUpdatesUpsertAndRemove --- PASS: TestApplyStatisticsUpdatesUpsertAndRemove (0.00s) === RUN TestIsS3TablesConflict --- PASS: TestIsS3TablesConflict (0.00s) === RUN TestValidateCreateTableRequestRequiresName --- PASS: TestValidateCreateTableRequestRequiresName (0.00s) === RUN TestValidateCreateTableRequestAcceptsWithName --- PASS: TestValidateCreateTableRequestAcceptsWithName (0.00s) === RUN TestIsStageCreateEnabledDefaultsToTrue --- PASS: TestIsStageCreateEnabledDefaultsToTrue (0.00s) === RUN TestIsStageCreateEnabledFalseValues --- PASS: TestIsStageCreateEnabledFalseValues (0.00s) === RUN TestNewTableMetadataRejectsV3TypeBelowV3 --- PASS: TestNewTableMetadataRejectsV3TypeBelowV3 (0.00s) === RUN TestNewTableMetadataAcceptsV3TypeAtV3 --- PASS: TestNewTableMetadataAcceptsV3TypeAtV3 (0.01s) === RUN TestBuildLoadTableResultNeverReturnsNilMetadata === RUN TestBuildLoadTableResultNeverReturnsNilMetadata/empty_full_metadata === RUN TestBuildLoadTableResultNeverReturnsNilMetadata/unparseable_metadata W0824 22:18:35.447115 handlers_table.go:556 Iceberg: Failed to parse persisted metadata for t: unexpected end of JSON input === RUN TestBuildLoadTableResultNeverReturnsNilMetadata/no_stored_metadata --- PASS: TestBuildLoadTableResultNeverReturnsNilMetadata (0.03s) --- PASS: TestBuildLoadTableResultNeverReturnsNilMetadata/empty_full_metadata (0.00s) --- PASS: TestBuildLoadTableResultNeverReturnsNilMetadata/unparseable_metadata (0.02s) --- PASS: TestBuildLoadTableResultNeverReturnsNilMetadata/no_stored_metadata (0.00s) === RUN TestCreateViewMissingNamespaceReturns404 E0824 22:18:35.491877 handler.go:218 S3Tables: error handling GetView: filer: no entry is found in filer store E0824 22:18:35.501602 handler.go:218 S3Tables: error handling CreateView: filer: no entry is found in filer store --- PASS: TestCreateViewMissingNamespaceReturns404 (0.03s) === RUN TestCreateViewTagsEntryAsView E0824 22:18:35.504587 handler.go:218 S3Tables: error handling GetView: filer: no entry is found in filer store --- PASS: TestCreateViewTagsEntryAsView (0.00s) === RUN TestCreateViewDuplicateDoesNotClobberMetadata E0824 22:18:35.507812 handler.go:218 S3Tables: error handling GetView: filer: no entry is found in filer store --- PASS: TestCreateViewDuplicateDoesNotClobberMetadata (0.02s) === RUN TestCreateViewRollsBackEntryWhenMetadataWriteFails E0824 22:18:35.530442 handler.go:218 S3Tables: error handling GetView: filer: no entry is found in filer store --- PASS: TestCreateViewRollsBackEntryWhenMetadataWriteFails (0.00s) === RUN TestBuildFileIOConfigVendingVendsScopedCredentials --- PASS: TestBuildFileIOConfigVendingVendsScopedCredentials (0.01s) === RUN TestBuildFileIOConfigVendingWithoutVendorSaysNothing --- PASS: TestBuildFileIOConfigVendingWithoutVendorSaysNothing (0.00s) === RUN TestBuildFileIOConfigVendingFallsBackWhenVendingFails W0824 22:18:35.544389 handlers_table.go:684 Iceberg: failed to vend credentials for s3://warehouse/ns/t: no role configured --- PASS: TestBuildFileIOConfigVendingFallsBackWhenVendingFails (0.00s) === RUN TestBuildFileIOConfigVendingWithoutDelegationHeader --- PASS: TestBuildFileIOConfigVendingWithoutDelegationHeader (0.00s) === RUN TestNameValidationError --- PASS: TestNameValidationError (0.00s) === RUN TestWriteManagerError === RUN TestWriteManagerError/invalid_name_is_a_client_error === RUN TestWriteManagerError/rejected_schema_is_a_client_error === RUN TestWriteManagerError/bad_format_version_is_a_client_error === RUN TestWriteManagerError/missing_table_bucket_is_a_client_error === RUN TestWriteManagerError/everything_else_is_a_server_fault --- PASS: TestWriteManagerError (0.00s) --- PASS: TestWriteManagerError/invalid_name_is_a_client_error (0.00s) --- PASS: TestWriteManagerError/rejected_schema_is_a_client_error (0.00s) --- PASS: TestWriteManagerError/bad_format_version_is_a_client_error (0.00s) --- PASS: TestWriteManagerError/missing_table_bucket_is_a_client_error (0.00s) --- PASS: TestWriteManagerError/everything_else_is_a_server_fault (0.00s) === RUN TestGetBucketFromPrefix_WarehouseQueryFallback === RUN TestGetBucketFromPrefix_WarehouseQueryFallback/warehouse_query_routes_to_its_bucket_when_no_prefix_in_path === RUN TestGetBucketFromPrefix_WarehouseQueryFallback/warehouse_query_with_sub-path_still_picks_the_bucket === RUN TestGetBucketFromPrefix_WarehouseQueryFallback/bare_bucket_name_is_taken_as_the_table_bucket === RUN TestGetBucketFromPrefix_WarehouseQueryFallback/table_bucket_ARN_routes_to_its_bucket === RUN TestGetBucketFromPrefix_WarehouseQueryFallback/unusable_warehouse_value_falls_through_to_default === RUN TestGetBucketFromPrefix_WarehouseQueryFallback/no_warehouse_and_no_prefix_returns_default --- PASS: TestGetBucketFromPrefix_WarehouseQueryFallback (0.00s) --- PASS: TestGetBucketFromPrefix_WarehouseQueryFallback/warehouse_query_routes_to_its_bucket_when_no_prefix_in_path (0.00s) --- PASS: TestGetBucketFromPrefix_WarehouseQueryFallback/warehouse_query_with_sub-path_still_picks_the_bucket (0.00s) --- PASS: TestGetBucketFromPrefix_WarehouseQueryFallback/bare_bucket_name_is_taken_as_the_table_bucket (0.00s) --- PASS: TestGetBucketFromPrefix_WarehouseQueryFallback/table_bucket_ARN_routes_to_its_bucket (0.00s) --- PASS: TestGetBucketFromPrefix_WarehouseQueryFallback/unusable_warehouse_value_falls_through_to_default (0.00s) --- PASS: TestGetBucketFromPrefix_WarehouseQueryFallback/no_warehouse_and_no_prefix_returns_default (0.00s) === RUN TestBuildFileIOConfig === RUN TestBuildFileIOConfig/no_endpoint_configured_yields_empty_config === RUN TestBuildFileIOConfig/endpoint_is_advertised_with_path-style-access_and_region --- PASS: TestBuildFileIOConfig (0.00s) --- PASS: TestBuildFileIOConfig/no_endpoint_configured_yields_empty_config (0.00s) --- PASS: TestBuildFileIOConfig/endpoint_is_advertised_with_path-style-access_and_region (0.00s) === RUN TestGetBucketFromPrefix_TableBucketEnvFallback --- PASS: TestGetBucketFromPrefix_TableBucketEnvFallback (0.00s) === RUN TestNormalizeNamespacePropertiesNil --- PASS: TestNormalizeNamespacePropertiesNil (0.00s) === RUN TestNormalizeNamespacePropertiesReturnsInputWhenSet --- PASS: TestNormalizeNamespacePropertiesReturnsInputWhenSet (0.00s) === RUN TestParsePaginationDefaultValues --- PASS: TestParsePaginationDefaultValues (0.00s) === RUN TestParsePaginationUsesCamelCaseParameters --- PASS: TestParsePaginationUsesCamelCaseParameters (0.00s) === RUN TestParsePaginationSupportsHyphenatedFallback --- PASS: TestParsePaginationSupportsHyphenatedFallback (0.00s) === RUN TestParsePaginationRejectsInvalidPageSize === RUN TestParsePaginationRejectsInvalidPageSize//v1/namespaces?pageSize=0 === RUN TestParsePaginationRejectsInvalidPageSize//v1/namespaces?pageSize=-1 === RUN TestParsePaginationRejectsInvalidPageSize//v1/namespaces?pageSize=foo === RUN TestParsePaginationRejectsInvalidPageSize//v1/namespaces?pageSize=1001 --- PASS: TestParsePaginationRejectsInvalidPageSize (0.00s) --- PASS: TestParsePaginationRejectsInvalidPageSize//v1/namespaces?pageSize=0 (0.00s) --- PASS: TestParsePaginationRejectsInvalidPageSize//v1/namespaces?pageSize=-1 (0.00s) --- PASS: TestParsePaginationRejectsInvalidPageSize//v1/namespaces?pageSize=foo (0.00s) --- PASS: TestParsePaginationRejectsInvalidPageSize//v1/namespaces?pageSize=1001 (0.00s) === RUN TestRegisterTableRequestWireShape --- PASS: TestRegisterTableRequestWireShape (0.00s) === RUN TestRegisterTableRejectsOutOfBoundsLocation === RUN TestRegisterTableRejectsOutOfBoundsLocation/cross_bucket === RUN TestRegisterTableRejectsOutOfBoundsLocation/traversal --- PASS: TestRegisterTableRejectsOutOfBoundsLocation (0.00s) --- PASS: TestRegisterTableRejectsOutOfBoundsLocation/cross_bucket (0.00s) --- PASS: TestRegisterTableRejectsOutOfBoundsLocation/traversal (0.00s) === RUN TestHandleRenameTableRejectsIncompleteIdentifiers === RUN TestHandleRenameTableRejectsIncompleteIdentifiers/missing_source_name === RUN TestHandleRenameTableRejectsIncompleteIdentifiers/missing_source_namespace === RUN TestHandleRenameTableRejectsIncompleteIdentifiers/missing_destination_name === RUN TestHandleRenameTableRejectsIncompleteIdentifiers/missing_destination_namespace === RUN TestHandleRenameTableRejectsIncompleteIdentifiers/empty_body === RUN TestHandleRenameTableRejectsIncompleteIdentifiers/malformed_json --- PASS: TestHandleRenameTableRejectsIncompleteIdentifiers (0.00s) --- PASS: TestHandleRenameTableRejectsIncompleteIdentifiers/missing_source_name (0.00s) --- PASS: TestHandleRenameTableRejectsIncompleteIdentifiers/missing_source_namespace (0.00s) --- PASS: TestHandleRenameTableRejectsIncompleteIdentifiers/missing_destination_name (0.00s) --- PASS: TestHandleRenameTableRejectsIncompleteIdentifiers/missing_destination_namespace (0.00s) --- PASS: TestHandleRenameTableRejectsIncompleteIdentifiers/empty_body (0.00s) --- PASS: TestHandleRenameTableRejectsIncompleteIdentifiers/malformed_json (0.00s) === RUN TestApplySnapshotsParamRefsKeepsOnlyReferenced --- PASS: TestApplySnapshotsParamRefsKeepsOnlyReferenced (0.00s) === RUN TestApplySnapshotsParamDefaultsToAll --- PASS: TestApplySnapshotsParamDefaultsToAll (0.00s) === RUN TestDecodeCommitTransactionRequest --- PASS: TestDecodeCommitTransactionRequest (0.00s) === RUN TestParseTableChangeSeparatesStatistics --- PASS: TestParseTableChangeSeparatesStatistics (0.00s) === RUN TestParseTableChangeRejectsInvalidRequirements --- PASS: TestParseTableChangeRejectsInvalidRequirements (0.00s) === RUN TestCloneTableMetadataIsIndependent --- PASS: TestCloneTableMetadataIsIndependent (0.00s) === RUN TestBuildTableRestoreRequestCarriesPriorState --- PASS: TestBuildTableRestoreRequestCarriesPriorState (0.00s) === RUN TestRollbackRevertsEveryFlippedField --- PASS: TestRollbackRevertsEveryFlippedField (0.00s) === RUN TestCleanupSafeMetadataKeepsFailedRollback --- PASS: TestCleanupSafeMetadataKeepsFailedRollback (0.00s) === RUN TestCleanupSafeMetadataFirstFlipFails --- PASS: TestCleanupSafeMetadataFirstFlipFails (0.00s) === RUN TestCleanupSafeMetadataAllRolledBack --- PASS: TestCleanupSafeMetadataAllRolledBack (0.00s) === RUN TestApplyNamespacePropertyUpdates --- PASS: TestApplyNamespacePropertyUpdates (0.00s) === RUN TestApplyNamespacePropertyUpdatesRemoveExisting --- PASS: TestApplyNamespacePropertyUpdatesRemoveExisting (0.00s) === RUN TestApplyNamespacePropertyUpdatesDuplicateRemovals --- PASS: TestApplyNamespacePropertyUpdatesDuplicateRemovals (0.00s) === RUN TestHandleUpdateNamespacePropertiesConflict --- PASS: TestHandleUpdateNamespacePropertiesConflict (0.00s) === RUN TestIsViewNotFound --- PASS: TestIsViewNotFound (0.00s) === RUN TestIsViewAlreadyExists --- PASS: TestIsViewAlreadyExists (0.00s) === RUN TestViewPropertiesCopiesAndIsNonNil --- PASS: TestViewPropertiesCopiesAndIsNonNil (0.00s) === RUN TestViewResponseRoundTrip --- PASS: TestViewResponseRoundTrip (0.00s) === RUN TestHandleConfigWarehouseSpellings === RUN TestHandleConfigWarehouseSpellings/s3_location === RUN TestHandleConfigWarehouseSpellings/bare_bucket_name === RUN TestHandleConfigWarehouseSpellings/table_bucket_ARN === RUN TestHandleConfigWarehouseSpellings/no_warehouse === RUN TestHandleConfigWarehouseSpellings/unusable_value --- PASS: TestHandleConfigWarehouseSpellings (0.00s) --- PASS: TestHandleConfigWarehouseSpellings/s3_location (0.00s) --- PASS: TestHandleConfigWarehouseSpellings/bare_bucket_name (0.00s) --- PASS: TestHandleConfigWarehouseSpellings/table_bucket_ARN (0.00s) --- PASS: TestHandleConfigWarehouseSpellings/no_warehouse (0.00s) --- PASS: TestHandleConfigWarehouseSpellings/unusable_value (0.00s) === RUN TestRepairClickHouseRelativeManifests --- PASS: TestRepairClickHouseRelativeManifests (0.03s) === RUN TestRepairClickHouseAbsoluteManifests --- PASS: TestRepairClickHouseAbsoluteManifests (0.04s) === RUN TestRepairedManifestsAreLeftAlone --- PASS: TestRepairedManifestsAreLeftAlone (0.02s) === RUN TestRepairKeepsDeleteManifestContent --- PASS: TestRepairKeepsDeleteManifestContent (0.03s) === RUN TestManifestContentValue --- PASS: TestManifestContentValue (0.00s) === RUN TestRepairSkipsLocationsOutsideMetadataDir --- PASS: TestRepairSkipsLocationsOutsideMetadataDir (0.00s) === RUN TestRepairLeavesUnreadableListAlone W0824 22:18:35.683956 manifest_repair.go:503 Iceberg: manifest repair for s3://iceberg-tables/wprobe/target_rel/metadata/junk.avro skipped: open avro: cannot create OCFReader: cannot read OCF header with invalid magic bytes: `not ` --- PASS: TestRepairLeavesUnreadableListAlone (0.00s) === RUN TestEnsureMetadataSpecCompliance_BackfillsMissingFields --- PASS: TestEnsureMetadataSpecCompliance_BackfillsMissingFields (0.00s) === RUN TestEnsureMetadataSpecCompliance_PreservesExistingFields --- PASS: TestEnsureMetadataSpecCompliance_PreservesExistingFields (0.00s) === RUN TestEnsureMetadataSpecCompliance_ReplacesExplicitNullsWithSentinels --- PASS: TestEnsureMetadataSpecCompliance_ReplacesExplicitNullsWithSentinels (0.00s) === RUN TestEnsureMetadataSpecCompliance_InvalidJSONReturnedUnchanged --- PASS: TestEnsureMetadataSpecCompliance_InvalidJSONReturnedUnchanged (0.00s) === RUN TestEnsureMetadataSpecCompliance_EmptyInputReturnedUnchanged --- PASS: TestEnsureMetadataSpecCompliance_EmptyInputReturnedUnchanged (0.00s) === RUN TestEnsureMetadataSpecCompliance_PreservesOriginalKeyOrder --- PASS: TestEnsureMetadataSpecCompliance_PreservesOriginalKeyOrder (0.00s) === RUN TestEnsureMetadataSpecCompliance_EmptyObjectBackfilled --- PASS: TestEnsureMetadataSpecCompliance_EmptyObjectBackfilled (0.00s) === RUN TestEnsureMetadataSpecCompliance_AllPresentReturnsSameBytes --- PASS: TestEnsureMetadataSpecCompliance_AllPresentReturnsSameBytes (0.00s) === RUN TestSaveMetadataFileExclusiveRefusesToOverwrite --- PASS: TestSaveMetadataFileExclusiveRefusesToOverwrite (0.00s) === RUN TestStageCommitMetadataFallsBackToAUniqueName --- PASS: TestStageCommitMetadataFallsBackToAUniqueName (0.00s) === RUN TestSaveMetadataFileOverwrites --- PASS: TestSaveMetadataFileOverwrites (0.00s) === RUN TestNameMappingFromSchema --- PASS: TestNameMappingFromSchema (0.00s) === RUN TestRefreshDefaultNameMappingDropsReassignedNames --- PASS: TestRefreshDefaultNameMappingDropsReassignedNames (0.00s) === RUN TestRefreshDefaultNameMappingOnSchemaEvolution --- PASS: TestRefreshDefaultNameMappingOnSchemaEvolution (0.00s) === RUN TestRefreshDefaultNameMappingKeepsUserNames --- PASS: TestRefreshDefaultNameMappingKeepsUserNames (0.00s) === RUN TestCreateTableSetsDefaultNameMapping --- PASS: TestCreateTableSetsDefaultNameMapping (0.00s) === RUN TestIsValidTablePath --- PASS: TestIsValidTablePath (0.00s) === RUN TestValidateRequestPath_RejectsTraversal === RUN TestValidateRequestPath_RejectsTraversal/clean_namespace+table_passes === RUN TestValidateRequestPath_RejectsTraversal/clean_prefixed_passes === RUN TestValidateRequestPath_RejectsTraversal/clean_namespace_only_passes === RUN TestValidateRequestPath_RejectsTraversal/dotdot_as_prefix_var_rejected === RUN TestValidateRequestPath_RejectsTraversal/dotdot_as_namespace_var_rejected === RUN TestValidateRequestPath_RejectsTraversal/dotdot_as_namespace_var_prefixed_rejected === RUN TestValidateRequestPath_RejectsTraversal/dotdot_as_table_var_rejected === RUN TestValidateRequestPath_RejectsTraversal/dot_as_table_var_rejected === RUN TestValidateRequestPath_RejectsTraversal/unit-sep_namespace_with_dotdot_part_rejected === RUN TestValidateRequestPath_RejectsTraversal/leading_unit-sep_namespace_rejected === RUN TestValidateRequestPath_RejectsTraversal/trailing_unit-sep_namespace_rejected === RUN TestValidateRequestPath_RejectsTraversal/consecutive_unit-sep_namespace_rejected --- PASS: TestValidateRequestPath_RejectsTraversal (0.09s) --- PASS: TestValidateRequestPath_RejectsTraversal/clean_namespace+table_passes (0.00s) --- PASS: TestValidateRequestPath_RejectsTraversal/clean_prefixed_passes (0.00s) --- PASS: TestValidateRequestPath_RejectsTraversal/clean_namespace_only_passes (0.02s) --- PASS: TestValidateRequestPath_RejectsTraversal/dotdot_as_prefix_var_rejected (0.01s) --- PASS: TestValidateRequestPath_RejectsTraversal/dotdot_as_namespace_var_rejected (0.00s) --- PASS: TestValidateRequestPath_RejectsTraversal/dotdot_as_namespace_var_prefixed_rejected (0.00s) --- PASS: TestValidateRequestPath_RejectsTraversal/dotdot_as_table_var_rejected (0.00s) --- PASS: TestValidateRequestPath_RejectsTraversal/dot_as_table_var_rejected (0.00s) --- PASS: TestValidateRequestPath_RejectsTraversal/unit-sep_namespace_with_dotdot_part_rejected (0.00s) --- PASS: TestValidateRequestPath_RejectsTraversal/leading_unit-sep_namespace_rejected (0.01s) --- PASS: TestValidateRequestPath_RejectsTraversal/trailing_unit-sep_namespace_rejected (0.00s) --- PASS: TestValidateRequestPath_RejectsTraversal/consecutive_unit-sep_namespace_rejected (0.00s) === RUN TestValidateRequestPath_RejectsEmptyCapturedVars === RUN TestValidateRequestPath_RejectsEmptyCapturedVars/empty_prefix === RUN TestValidateRequestPath_RejectsEmptyCapturedVars/empty_table === RUN TestValidateRequestPath_RejectsEmptyCapturedVars/empty_namespace --- PASS: TestValidateRequestPath_RejectsEmptyCapturedVars (0.01s) --- PASS: TestValidateRequestPath_RejectsEmptyCapturedVars/empty_prefix (0.00s) --- PASS: TestValidateRequestPath_RejectsEmptyCapturedVars/empty_table (0.00s) --- PASS: TestValidateRequestPath_RejectsEmptyCapturedVars/empty_namespace (0.00s) === RUN TestIsValidNameSegment === RUN TestIsValidNameSegment/empty_ok === RUN TestIsValidNameSegment/plain === RUN TestIsValidNameSegment/with_dot_inside === RUN TestIsValidNameSegment/hidden === RUN TestIsValidNameSegment/bare_dot === RUN TestIsValidNameSegment/bare_dotdot === RUN TestIsValidNameSegment/contains_slash === RUN TestIsValidNameSegment/contains_backslash === RUN TestIsValidNameSegment/contains_nul --- PASS: TestIsValidNameSegment (0.01s) --- PASS: TestIsValidNameSegment/empty_ok (0.00s) --- PASS: TestIsValidNameSegment/plain (0.00s) --- PASS: TestIsValidNameSegment/with_dot_inside (0.00s) --- PASS: TestIsValidNameSegment/hidden (0.00s) --- PASS: TestIsValidNameSegment/bare_dot (0.00s) --- PASS: TestIsValidNameSegment/bare_dotdot (0.00s) --- PASS: TestIsValidNameSegment/contains_slash (0.00s) --- PASS: TestIsValidNameSegment/contains_backslash (0.00s) --- PASS: TestIsValidNameSegment/contains_nul (0.00s) === RUN TestRegisteredRoutesReachAHandler --- PASS: TestRegisteredRoutesReachAHandler (0.03s) PASS ok github.com/seaweedfs/seaweedfs/weed/s3api/iceberg 26.732s === RUN TestTableLifecycle E0824 22:18:29.229927 handler.go:218 S3Tables: error handling GetNamespace: filer: no entry is found in filer store E0824 22:18:29.264888 handler_namespace.go:18 S3Tables: handleCreateNamespace called I0824 22:18:29.291595 handler_namespace.go:115 S3Tables: CreateNamespace permission check - principal=000000000000, owner=000000000000, actions=[] --- PASS: TestTableLifecycle (1.05s) === RUN TestRegisterOverwriteKeepsTheOldDataset E0824 22:18:29.978959 handler.go:218 S3Tables: error handling GetNamespace: filer: no entry is found in filer store E0824 22:18:29.992755 handler_namespace.go:18 S3Tables: handleCreateNamespace called I0824 22:18:30.008469 handler_namespace.go:115 S3Tables: CreateNamespace permission check - principal=000000000000, owner=000000000000, actions=[] --- PASS: TestRegisterOverwriteKeepsTheOldDataset (0.48s) === RUN TestListAllTablesIsNeverNull --- PASS: TestListAllTablesIsNeverNull (0.10s) === RUN TestNamespaceListing E0824 22:18:30.531844 handler.go:218 S3Tables: error handling GetNamespace: filer: no entry is found in filer store E0824 22:18:30.544545 handler_namespace.go:18 S3Tables: handleCreateNamespace called I0824 22:18:30.558760 handler_namespace.go:115 S3Tables: CreateNamespace permission check - principal=000000000000, owner=000000000000, actions=[] E0824 22:18:30.582655 handler.go:218 S3Tables: error handling GetNamespace: filer: no entry is found in filer store E0824 22:18:30.595363 handler_namespace.go:18 S3Tables: handleCreateNamespace called I0824 22:18:30.609655 handler_namespace.go:115 S3Tables: CreateNamespace permission check - principal=000000000000, owner=000000000000, actions=[] E0824 22:18:30.687019 handler.go:218 S3Tables: error handling GetNamespace: filer: no entry is found in filer store E0824 22:18:30.695181 handler.go:218 S3Tables: error handling DeleteNamespace: filer: no entry is found in filer store E0824 22:18:30.700609 handler.go:218 S3Tables: error handling DeleteNamespace: filer: no entry is found in filer store --- PASS: TestNamespaceListing (0.25s) === RUN TestCreateNamespaceRequiresItsParent E0824 22:18:30.789541 handler.go:218 S3Tables: error handling GetNamespace: filer: no entry is found in filer store E0824 22:18:30.794178 handler.go:218 S3Tables: error handling GetNamespace: filer: no entry is found in filer store E0824 22:18:30.806113 handler.go:218 S3Tables: error handling GetNamespace: filer: no entry is found in filer store E0824 22:18:30.817096 handler_namespace.go:18 S3Tables: handleCreateNamespace called I0824 22:18:30.831281 handler_namespace.go:115 S3Tables: CreateNamespace permission check - principal=000000000000, owner=000000000000, actions=[] E0824 22:18:30.850953 handler.go:218 S3Tables: error handling GetNamespace: filer: no entry is found in filer store E0824 22:18:30.872396 handler_namespace.go:18 S3Tables: handleCreateNamespace called I0824 22:18:30.883703 handler_namespace.go:115 S3Tables: CreateNamespace permission check - principal=000000000000, owner=000000000000, actions=[] --- PASS: TestCreateNamespaceRequiresItsParent (0.24s) === RUN TestCreateNamespaceModes E0824 22:18:31.035164 handler.go:218 S3Tables: error handling GetNamespace: filer: no entry is found in filer store E0824 22:18:31.044679 handler_namespace.go:18 S3Tables: handleCreateNamespace called I0824 22:18:31.057822 handler_namespace.go:115 S3Tables: CreateNamespace permission check - principal=000000000000, owner=000000000000, actions=[] --- PASS: TestCreateNamespaceModes (0.20s) === RUN TestIcebergTablesAreInvisible E0824 22:18:31.310348 handler.go:218 S3Tables: error handling GetNamespace: filer: no entry is found in filer store E0824 22:18:31.325794 handler_namespace.go:18 S3Tables: handleCreateNamespace called I0824 22:18:31.355457 handler_namespace.go:115 S3Tables: CreateNamespace permission check - principal=000000000000, owner=000000000000, actions=[] E0824 22:18:31.945465 handler.go:218 S3Tables: error handling CreateTable: format conflict: ledger --- PASS: TestIcebergTablesAreInvisible (0.81s) === RUN TestRouteAndBodyMustAgree E0824 22:18:32.106158 handler.go:218 S3Tables: error handling GetNamespace: filer: no entry is found in filer store E0824 22:18:32.114643 handler_namespace.go:18 S3Tables: handleCreateNamespace called I0824 22:18:32.145746 handler_namespace.go:115 S3Tables: CreateNamespace permission check - principal=000000000000, owner=000000000000, actions=[] --- PASS: TestRouteAndBodyMustAgree (0.21s) === RUN TestDataPlaneIsUnsupported --- PASS: TestDataPlaneIsUnsupported (0.10s) === RUN TestParseIdentifier === RUN TestParseIdentifier/root_is_the_delimiter === RUN TestParseIdentifier/empty_is_the_root === RUN TestParseIdentifier/bucket === RUN TestParseIdentifier/namespace === RUN TestParseIdentifier/table === RUN TestParseIdentifier/custom_delimiter === RUN TestParseIdentifier/empty_part_is_rejected === RUN TestParseIdentifier/trailing_delimiter_is_rejected --- PASS: TestParseIdentifier (0.01s) --- PASS: TestParseIdentifier/root_is_the_delimiter (0.00s) --- PASS: TestParseIdentifier/empty_is_the_root (0.00s) --- PASS: TestParseIdentifier/bucket (0.00s) --- PASS: TestParseIdentifier/namespace (0.00s) --- PASS: TestParseIdentifier/table (0.00s) --- PASS: TestParseIdentifier/custom_delimiter (0.00s) --- PASS: TestParseIdentifier/empty_part_is_rejected (0.00s) --- PASS: TestParseIdentifier/trailing_delimiter_is_rejected (0.00s) === RUN TestIdentifierNamespace --- PASS: TestIdentifierNamespace (0.00s) === RUN TestIdentifierTable --- PASS: TestIdentifierTable (0.00s) === RUN TestIdentifierMatchesBody --- PASS: TestIdentifierMatchesBody (0.00s) === RUN TestIdentifierString --- PASS: TestIdentifierString (0.00s) PASS ok github.com/seaweedfs/seaweedfs/weed/s3api/lance 23.141s === RUN TestLifecycleToCanonical_TopLevelPrefix --- PASS: TestLifecycleToCanonical_TopLevelPrefix (0.07s) === RUN TestLifecycleToCanonical_FilterPrefix --- PASS: TestLifecycleToCanonical_FilterPrefix (0.00s) === RUN TestLifecycleToCanonical_FilterTagAndSize --- PASS: TestLifecycleToCanonical_FilterTagAndSize (0.00s) === RUN TestLifecycleToCanonical_SingleTagFilter --- PASS: TestLifecycleToCanonical_SingleTagFilter (0.00s) === RUN TestLifecycleToCanonical_MultipleActions --- PASS: TestLifecycleToCanonical_MultipleActions (0.00s) === RUN TestLifecycleToCanonical_ExpirationDate --- PASS: TestLifecycleToCanonical_ExpirationDate (0.00s) === RUN TestLifecycleToCanonical_ExpiredObjectDeleteMarker --- PASS: TestLifecycleToCanonical_ExpiredObjectDeleteMarker (0.00s) === RUN TestLifecycleToCanonical_DisabledRulePreserved --- PASS: TestLifecycleToCanonical_DisabledRulePreserved (0.04s) === RUN TestLifecycleToCanonical_NilSafe --- PASS: TestLifecycleToCanonical_NilSafe (0.00s) === RUN TestLifecycleToCanonical_EmptyRules --- PASS: TestLifecycleToCanonical_EmptyRules (0.00s) === RUN TestLifecycleXMLRoundTrip_NoncurrentVersionExpiration --- PASS: TestLifecycleXMLRoundTrip_NoncurrentVersionExpiration (0.01s) === RUN TestLifecycleXMLRoundTrip_AbortIncompleteMultipartUpload --- PASS: TestLifecycleXMLRoundTrip_AbortIncompleteMultipartUpload (0.01s) === RUN TestLifecycleXMLRoundTrip_FilterWithTag --- PASS: TestLifecycleXMLRoundTrip_FilterWithTag (0.00s) === RUN TestLifecycleXMLRoundTrip_FilterWithAnd --- PASS: TestLifecycleXMLRoundTrip_FilterWithAnd (0.00s) === RUN TestLifecycleXMLRoundTrip_FilterWithSizeOnly --- PASS: TestLifecycleXMLRoundTrip_FilterWithSizeOnly (0.00s) === RUN TestLifecycleXML_TransitionSetFlag --- PASS: TestLifecycleXML_TransitionSetFlag (0.00s) === RUN TestLifecycleXML_NoncurrentVersionTransitionSetFlag --- PASS: TestLifecycleXML_NoncurrentVersionTransitionSetFlag (0.00s) === RUN TestLifecycleXMLRoundTrip_CompleteRule --- PASS: TestLifecycleXMLRoundTrip_CompleteRule (0.00s) === RUN TestCanonicalRoundTrip_WholeBucket --- PASS: TestCanonicalRoundTrip_WholeBucket (0.00s) === RUN TestCanonicalRoundTrip_PrefixOnly --- PASS: TestCanonicalRoundTrip_PrefixOnly (0.00s) === RUN TestCanonicalRoundTrip_TagOnly --- PASS: TestCanonicalRoundTrip_TagOnly (0.00s) === RUN TestCanonicalRoundTrip_PrefixAndTags --- PASS: TestCanonicalRoundTrip_PrefixAndTags (0.00s) === RUN TestCanonicalRoundTrip_SizeBounds --- PASS: TestCanonicalRoundTrip_SizeBounds (0.00s) === RUN TestCanonicalRoundTrip_PrefixAndTagsWithSizeBounds --- PASS: TestCanonicalRoundTrip_PrefixAndTagsWithSizeBounds (0.01s) === RUN TestCanonicalRoundTrip_ExpirationDate --- PASS: TestCanonicalRoundTrip_ExpirationDate (0.00s) === RUN TestCanonicalRoundTrip_ExpiredObjectDeleteMarker --- PASS: TestCanonicalRoundTrip_ExpiredObjectDeleteMarker (0.00s) === RUN TestCanonicalRoundTrip_NoncurrentVersionExpiration --- PASS: TestCanonicalRoundTrip_NoncurrentVersionExpiration (0.00s) === RUN TestCanonicalRoundTrip_AbortMultipartUpload --- PASS: TestCanonicalRoundTrip_AbortMultipartUpload (0.00s) === RUN TestCanonicalRoundTrip_Disabled --- PASS: TestCanonicalRoundTrip_Disabled (0.00s) === RUN TestCanonicalRoundTrip_TagOrderIsStable --- PASS: TestCanonicalRoundTrip_TagOrderIsStable (0.00s) === RUN TestCanonicalRoundTrip_PrefixWithSizeBoundsUsesAnd --- PASS: TestCanonicalRoundTrip_PrefixWithSizeBoundsUsesAnd (0.00s) === RUN TestCanonicalRoundTrip_TagWithSizeBoundsUsesAnd --- PASS: TestCanonicalRoundTrip_TagWithSizeBoundsUsesAnd (0.00s) === RUN TestCanonicalRoundTrip_SizeOnlyOmitsPrefixElement --- PASS: TestCanonicalRoundTrip_SizeOnlyOmitsPrefixElement (0.00s) === RUN TestCanonicalRoundTrip_SizeRangeUsesAnd --- PASS: TestCanonicalRoundTrip_SizeRangeUsesAnd (0.00s) === RUN TestMarshalCanonical_CarriesTheS3Namespace --- PASS: TestMarshalCanonical_CarriesTheS3Namespace (0.00s) PASS ok github.com/seaweedfs/seaweedfs/weed/s3api/lifecycle_xml 13.412s === RUN TestCheckPostPolicy_RejectsUnknownConditionKey --- PASS: TestCheckPostPolicy_RejectsUnknownConditionKey (0.13s) === RUN TestCheckPostPolicy_RejectsExtraXAmzFormField --- PASS: TestCheckPostPolicy_RejectsExtraXAmzFormField (0.00s) === RUN TestCheckPostPolicy_AllowsXAmzAuthFields --- PASS: TestCheckPostPolicy_AllowsXAmzAuthFields (0.00s) === RUN TestCheckPostPolicy_AllowsMatchingXAmzField --- PASS: TestCheckPostPolicy_AllowsMatchingXAmzField (0.00s) === RUN TestCheckPostPolicy_ExistingXAmzMetaCheckStillWorks --- PASS: TestCheckPostPolicy_ExistingXAmzMetaCheckStillWorks (0.00s) === RUN TestCheckPostPolicy_AllowsStartsWithPrefixStem --- PASS: TestCheckPostPolicy_AllowsStartsWithPrefixStem (0.00s) === RUN TestCheckPostPolicy_PrefixStemDoesNotCoverOtherPrefixes --- PASS: TestCheckPostPolicy_PrefixStemDoesNotCoverOtherPrefixes (0.00s) === RUN TestCheckPostPolicy_PrefixStemEnforcesValuePrefix --- PASS: TestCheckPostPolicy_PrefixStemEnforcesValuePrefix (0.00s) === RUN TestCheckPostPolicy_ExactAndPrefixBothEnforced --- PASS: TestCheckPostPolicy_ExactAndPrefixBothEnforced (0.00s) === RUN TestCheckPostPolicy_MultiplePrefixStemsAllEnforced --- PASS: TestCheckPostPolicy_MultiplePrefixStemsAllEnforced (0.00s) === RUN TestCheckPostPolicy_UnknownKeyErrorIncludesPolicyValue --- PASS: TestCheckPostPolicy_UnknownKeyErrorIncludesPolicyValue (0.00s) PASS ok github.com/seaweedfs/seaweedfs/weed/s3api/policy 4.910s === RUN TestBucketsAllowedForAction === RUN TestBucketsAllowedForAction/multipart_inherits_put === RUN TestBucketsAllowedForAction/multipart_inherits_put_in_any_case === RUN TestBucketsAllowedForAction/unrelated_action_names_nothing === RUN TestBucketsAllowedForAction/mixed_case_grant_is_read === RUN TestBucketsAllowedForAction/wildcard_bucket_is_incomplete === RUN TestBucketsAllowedForAction/unknown_policy_is_incomplete --- PASS: TestBucketsAllowedForAction (0.02s) --- PASS: TestBucketsAllowedForAction/multipart_inherits_put (0.00s) --- PASS: TestBucketsAllowedForAction/multipart_inherits_put_in_any_case (0.00s) --- PASS: TestBucketsAllowedForAction/unrelated_action_names_nothing (0.00s) --- PASS: TestBucketsAllowedForAction/mixed_case_grant_is_read (0.00s) --- PASS: TestBucketsAllowedForAction/wildcard_bucket_is_incomplete (0.00s) --- PASS: TestBucketsAllowedForAction/unknown_policy_is_incomplete (0.00s) === RUN TestExtractPrincipalVariables === RUN TestExtractPrincipalVariables/IAM_User_ARN === RUN TestExtractPrincipalVariables/Assumed_Role_ARN === RUN TestExtractPrincipalVariables/IAM_Role_ARN === RUN TestExtractPrincipalVariables/Non-ARN_principal === RUN TestExtractPrincipalVariables/Wildcard_principal --- PASS: TestExtractPrincipalVariables (0.00s) --- PASS: TestExtractPrincipalVariables/IAM_User_ARN (0.00s) --- PASS: TestExtractPrincipalVariables/Assumed_Role_ARN (0.00s) --- PASS: TestExtractPrincipalVariables/IAM_Role_ARN (0.00s) --- PASS: TestExtractPrincipalVariables/Non-ARN_principal (0.00s) --- PASS: TestExtractPrincipalVariables/Wildcard_principal (0.00s) === RUN TestSubstituteVariablesWithClaims === RUN TestSubstituteVariablesWithClaims/Standard_context_variable === RUN TestSubstituteVariablesWithClaims/JWT_claim_substitution === RUN TestSubstituteVariablesWithClaims/Mixed_variables === RUN TestSubstituteVariablesWithClaims/Variable_not_found --- PASS: TestSubstituteVariablesWithClaims (0.00s) --- PASS: TestSubstituteVariablesWithClaims/Standard_context_variable (0.00s) --- PASS: TestSubstituteVariablesWithClaims/JWT_claim_substitution (0.00s) --- PASS: TestSubstituteVariablesWithClaims/Mixed_variables (0.00s) --- PASS: TestSubstituteVariablesWithClaims/Variable_not_found (0.00s) === RUN TestPolicyVariablesWithPrincipalType --- PASS: TestPolicyVariablesWithPrincipalType (0.00s) === RUN TestPolicyVariablesWithJWTClaims --- PASS: TestPolicyVariablesWithJWTClaims (0.00s) === RUN TestExtractPrincipalVariablesWithAccount --- PASS: TestExtractPrincipalVariablesWithAccount (0.00s) === RUN TestSubstituteVariablesWithLDAP --- PASS: TestSubstituteVariablesWithLDAP (0.00s) === RUN TestSubstituteVariablesSpecialChars === RUN TestSubstituteVariablesSpecialChars/Comparison_operators_in_claims/vars === RUN TestSubstituteVariablesSpecialChars/Path_traversal_attempt_(should_just_substitute_text) --- PASS: TestSubstituteVariablesSpecialChars (0.00s) --- PASS: TestSubstituteVariablesSpecialChars/Comparison_operators_in_claims/vars (0.00s) --- PASS: TestSubstituteVariablesSpecialChars/Path_traversal_attempt_(should_just_substitute_text) (0.00s) === RUN TestIsolationPolicy --- PASS: TestIsolationPolicy (0.01s) === RUN TestNotResourceWithVariables --- PASS: TestNotResourceWithVariables (0.00s) === RUN TestExtractPrincipalVariablesWithPaths === RUN TestExtractPrincipalVariablesWithPaths/IAM_User_with_path === RUN TestExtractPrincipalVariablesWithPaths/IAM_Role_with_path === RUN TestExtractPrincipalVariablesWithPaths/Assumed_Role_with_path --- PASS: TestExtractPrincipalVariablesWithPaths (0.00s) --- PASS: TestExtractPrincipalVariablesWithPaths/IAM_User_with_path (0.00s) --- PASS: TestExtractPrincipalVariablesWithPaths/IAM_Role_with_path (0.00s) --- PASS: TestExtractPrincipalVariablesWithPaths/Assumed_Role_with_path (0.00s) === RUN TestPolicyEngine --- PASS: TestPolicyEngine (0.00s) === RUN TestConditionEvaluators === RUN TestConditionEvaluators/StringEquals_-_match === RUN TestConditionEvaluators/StringEquals_-_no_match === RUN TestConditionEvaluators/StringLike_-_wildcard_match === RUN TestConditionEvaluators/StringLike_-_wildcard_no_match === RUN TestConditionEvaluators/NumericEquals_-_match === RUN TestConditionEvaluators/NumericLessThan_-_match === RUN TestConditionEvaluators/NumericLessThan_-_no_match === RUN TestConditionEvaluators/IpAddress_-_CIDR_match === RUN TestConditionEvaluators/IpAddress_-_CIDR_no_match === RUN TestConditionEvaluators/Bool_-_true_match === RUN TestConditionEvaluators/Bool_-_false_match === RUN TestConditionEvaluators/Bool_-_no_match --- PASS: TestConditionEvaluators (0.00s) --- PASS: TestConditionEvaluators/StringEquals_-_match (0.00s) --- PASS: TestConditionEvaluators/StringEquals_-_no_match (0.00s) --- PASS: TestConditionEvaluators/StringLike_-_wildcard_match (0.00s) --- PASS: TestConditionEvaluators/StringLike_-_wildcard_no_match (0.00s) --- PASS: TestConditionEvaluators/NumericEquals_-_match (0.00s) --- PASS: TestConditionEvaluators/NumericLessThan_-_match (0.00s) --- PASS: TestConditionEvaluators/NumericLessThan_-_no_match (0.00s) --- PASS: TestConditionEvaluators/IpAddress_-_CIDR_match (0.00s) --- PASS: TestConditionEvaluators/IpAddress_-_CIDR_no_match (0.00s) --- PASS: TestConditionEvaluators/Bool_-_true_match (0.00s) --- PASS: TestConditionEvaluators/Bool_-_false_match (0.00s) --- PASS: TestConditionEvaluators/Bool_-_no_match (0.00s) === RUN TestPolicyValidation === RUN TestPolicyValidation/Valid_policy === RUN TestPolicyValidation/Valid_policy_with_single_statement_object === RUN TestPolicyValidation/Invalid_version === RUN TestPolicyValidation/Missing_action === RUN TestPolicyValidation/Invalid_JSON --- PASS: TestPolicyValidation (0.01s) --- PASS: TestPolicyValidation/Valid_policy (0.00s) --- PASS: TestPolicyValidation/Valid_policy_with_single_statement_object (0.00s) --- PASS: TestPolicyValidation/Invalid_version (0.00s) --- PASS: TestPolicyValidation/Missing_action (0.00s) --- PASS: TestPolicyValidation/Invalid_JSON (0.00s) === RUN TestPatternMatching === RUN TestPatternMatching/Exact_match === RUN TestPatternMatching/Wildcard_match === RUN TestPatternMatching/Wildcard_no_match === RUN TestPatternMatching/Full_wildcard === RUN TestPatternMatching/Question_mark_wildcard --- PASS: TestPatternMatching (0.00s) --- PASS: TestPatternMatching/Exact_match (0.00s) --- PASS: TestPatternMatching/Wildcard_match (0.00s) --- PASS: TestPatternMatching/Wildcard_no_match (0.00s) --- PASS: TestPatternMatching/Full_wildcard (0.00s) --- PASS: TestPatternMatching/Question_mark_wildcard (0.00s) === RUN TestExtractConditionValuesFromRequest --- PASS: TestExtractConditionValuesFromRequest (0.00s) === RUN TestExtractConditionValuesFromRequestSourceIPPrecedence === RUN TestExtractConditionValuesFromRequestSourceIPPrecedence/uses_right-most_public_X-Forwarded-For_entry === RUN TestExtractConditionValuesFromRequestSourceIPPrecedence/falls_back_to_X-Real-Ip_when_X-Forwarded-For_has_no_valid_ip === RUN TestExtractConditionValuesFromRequestSourceIPPrecedence/uses_RemoteAddr_ip_when_no_forwarding_headers === RUN TestExtractConditionValuesFromRequestSourceIPPrecedence/keeps_unix_socket_marker_when_RemoteAddr_is_not_an_ip === RUN TestExtractConditionValuesFromRequestSourceIPPrecedence/uses_IPv6_X-Forwarded-For_entry === RUN TestExtractConditionValuesFromRequestSourceIPPrecedence/ignores_spoofed_IP_when_real_client_is_public === RUN TestExtractConditionValuesFromRequestSourceIPPrecedence/handles_bracketed_IPv6_remote_address === RUN TestExtractConditionValuesFromRequestSourceIPPrecedence/avoids_returning_DNS_host_names --- PASS: TestExtractConditionValuesFromRequestSourceIPPrecedence (0.01s) --- PASS: TestExtractConditionValuesFromRequestSourceIPPrecedence/uses_right-most_public_X-Forwarded-For_entry (0.00s) --- PASS: TestExtractConditionValuesFromRequestSourceIPPrecedence/falls_back_to_X-Real-Ip_when_X-Forwarded-For_has_no_valid_ip (0.00s) --- PASS: TestExtractConditionValuesFromRequestSourceIPPrecedence/uses_RemoteAddr_ip_when_no_forwarding_headers (0.00s) --- PASS: TestExtractConditionValuesFromRequestSourceIPPrecedence/keeps_unix_socket_marker_when_RemoteAddr_is_not_an_ip (0.00s) --- PASS: TestExtractConditionValuesFromRequestSourceIPPrecedence/uses_IPv6_X-Forwarded-For_entry (0.00s) --- PASS: TestExtractConditionValuesFromRequestSourceIPPrecedence/ignores_spoofed_IP_when_real_client_is_public (0.00s) --- PASS: TestExtractConditionValuesFromRequestSourceIPPrecedence/handles_bracketed_IPv6_remote_address (0.00s) --- PASS: TestExtractConditionValuesFromRequestSourceIPPrecedence/avoids_returning_DNS_host_names (0.00s) === RUN TestPolicyEvaluationWithConditions --- PASS: TestPolicyEvaluationWithConditions (0.00s) === RUN TestResourceArn === RUN TestResourceArn/Bucket_only === RUN TestResourceArn/Bucket_and_object === RUN TestResourceArn/Bucket_and_nested_object --- PASS: TestResourceArn (0.00s) --- PASS: TestResourceArn/Bucket_only (0.00s) --- PASS: TestResourceArn/Bucket_and_object (0.00s) --- PASS: TestResourceArn/Bucket_and_nested_object (0.00s) === RUN TestActionConversion === RUN TestActionConversion/Already_has_s3_prefix === RUN TestActionConversion/Add_s3_prefix --- PASS: TestActionConversion (0.00s) --- PASS: TestActionConversion/Already_has_s3_prefix (0.00s) --- PASS: TestActionConversion/Add_s3_prefix (0.00s) === RUN TestPolicyEngineForRequest --- PASS: TestPolicyEngineForRequest (0.00s) === RUN TestWildcardMatching === RUN TestWildcardMatching/Exact_match === RUN TestWildcardMatching/Single_wildcard === RUN TestWildcardMatching/Prefix_wildcard === RUN TestWildcardMatching/Suffix_wildcard === RUN TestWildcardMatching/Middle_wildcard === RUN TestWildcardMatching/No_match === RUN TestWildcardMatching/Multiple_wildcards --- PASS: TestWildcardMatching (0.00s) --- PASS: TestWildcardMatching/Exact_match (0.00s) --- PASS: TestWildcardMatching/Single_wildcard (0.00s) --- PASS: TestWildcardMatching/Prefix_wildcard (0.00s) --- PASS: TestWildcardMatching/Suffix_wildcard (0.00s) --- PASS: TestWildcardMatching/Middle_wildcard (0.00s) --- PASS: TestWildcardMatching/No_match (0.00s) --- PASS: TestWildcardMatching/Multiple_wildcards (0.00s) === RUN TestCompilePolicy --- PASS: TestCompilePolicy (0.00s) === RUN TestExistingObjectTagCondition === RUN TestExistingObjectTagCondition/Matching_tag_value_-_should_allow === RUN TestExistingObjectTagCondition/Non-matching_tag_value_-_should_be_indeterminate === RUN TestExistingObjectTagCondition/Missing_tag_-_should_be_indeterminate === RUN TestExistingObjectTagCondition/No_tags_-_should_be_indeterminate === RUN TestExistingObjectTagCondition/Empty_tags_-_should_be_indeterminate === RUN TestExistingObjectTagCondition/Multiple_tags_with_matching_one_-_should_allow --- PASS: TestExistingObjectTagCondition (0.00s) --- PASS: TestExistingObjectTagCondition/Matching_tag_value_-_should_allow (0.00s) --- PASS: TestExistingObjectTagCondition/Non-matching_tag_value_-_should_be_indeterminate (0.00s) --- PASS: TestExistingObjectTagCondition/Missing_tag_-_should_be_indeterminate (0.00s) --- PASS: TestExistingObjectTagCondition/No_tags_-_should_be_indeterminate (0.00s) --- PASS: TestExistingObjectTagCondition/Empty_tags_-_should_be_indeterminate (0.00s) --- PASS: TestExistingObjectTagCondition/Multiple_tags_with_matching_one_-_should_allow (0.00s) === RUN TestExistingObjectTagConditionMultipleTags === RUN TestExistingObjectTagConditionMultipleTags/Both_tags_match_-_should_allow === RUN TestExistingObjectTagConditionMultipleTags/Both_tags_match_(premium_tier)_-_should_allow === RUN TestExistingObjectTagConditionMultipleTags/Only_status_matches_-_should_be_indeterminate === RUN TestExistingObjectTagConditionMultipleTags/Only_tier_matches_-_should_be_indeterminate === RUN TestExistingObjectTagConditionMultipleTags/Neither_tag_matches_-_should_be_indeterminate --- PASS: TestExistingObjectTagConditionMultipleTags (0.00s) --- PASS: TestExistingObjectTagConditionMultipleTags/Both_tags_match_-_should_allow (0.00s) --- PASS: TestExistingObjectTagConditionMultipleTags/Both_tags_match_(premium_tier)_-_should_allow (0.00s) --- PASS: TestExistingObjectTagConditionMultipleTags/Only_status_matches_-_should_be_indeterminate (0.00s) --- PASS: TestExistingObjectTagConditionMultipleTags/Only_tier_matches_-_should_be_indeterminate (0.00s) --- PASS: TestExistingObjectTagConditionMultipleTags/Neither_tag_matches_-_should_be_indeterminate (0.00s) === RUN TestExistingObjectTagDenyPolicy === RUN TestExistingObjectTagDenyPolicy/No_tags_-_allow_by_default_statement === RUN TestExistingObjectTagDenyPolicy/Non-confidential_tag_-_allow === RUN TestExistingObjectTagDenyPolicy/Confidential_tag_-_deny --- PASS: TestExistingObjectTagDenyPolicy (0.01s) --- PASS: TestExistingObjectTagDenyPolicy/No_tags_-_allow_by_default_statement (0.00s) --- PASS: TestExistingObjectTagDenyPolicy/Non-confidential_tag_-_allow (0.00s) --- PASS: TestExistingObjectTagDenyPolicy/Confidential_tag_-_deny (0.00s) === RUN TestMultipartUploadInheritsPutObjectPermission === RUN TestMultipartUploadInheritsPutObjectPermission/s3:CreateMultipartUpload === RUN TestMultipartUploadInheritsPutObjectPermission/s3:UploadPart === RUN TestMultipartUploadInheritsPutObjectPermission/s3:UploadPartCopy === RUN TestMultipartUploadInheritsPutObjectPermission/s3:CompleteMultipartUpload === RUN TestMultipartUploadInheritsPutObjectPermission/s3:AbortMultipartUpload === RUN TestMultipartUploadInheritsPutObjectPermission/s3:ListMultipartUploadParts === RUN TestMultipartUploadInheritsPutObjectPermission/s3:ListBucketMultipartUploads === RUN TestMultipartUploadInheritsPutObjectPermission/s3:ListBucketMultipartUploads_bucket_ARN === RUN TestMultipartUploadInheritsPutObjectPermission/s3:DeleteObject_not_inherited --- PASS: TestMultipartUploadInheritsPutObjectPermission (0.00s) --- PASS: TestMultipartUploadInheritsPutObjectPermission/s3:CreateMultipartUpload (0.00s) --- PASS: TestMultipartUploadInheritsPutObjectPermission/s3:UploadPart (0.00s) --- PASS: TestMultipartUploadInheritsPutObjectPermission/s3:UploadPartCopy (0.00s) --- PASS: TestMultipartUploadInheritsPutObjectPermission/s3:CompleteMultipartUpload (0.00s) --- PASS: TestMultipartUploadInheritsPutObjectPermission/s3:AbortMultipartUpload (0.00s) --- PASS: TestMultipartUploadInheritsPutObjectPermission/s3:ListMultipartUploadParts (0.00s) --- PASS: TestMultipartUploadInheritsPutObjectPermission/s3:ListBucketMultipartUploads (0.00s) --- PASS: TestMultipartUploadInheritsPutObjectPermission/s3:ListBucketMultipartUploads_bucket_ARN (0.00s) --- PASS: TestMultipartUploadInheritsPutObjectPermission/s3:DeleteObject_not_inherited (0.00s) === RUN TestPolicyVariables --- PASS: TestPolicyVariables (0.00s) === RUN TestEvaluatePolicyForRequestVariables --- PASS: TestEvaluatePolicyForRequestVariables (0.00s) === RUN TestPolicyPrincipalUnmarshal === RUN TestPolicyPrincipalUnmarshal/bare_wildcard === RUN TestPolicyPrincipalUnmarshal/bare_arn === RUN TestPolicyPrincipalUnmarshal/bare_array === RUN TestPolicyPrincipalUnmarshal/AWS_single === RUN TestPolicyPrincipalUnmarshal/AWS_array === RUN TestPolicyPrincipalUnmarshal/AWS_wildcard === RUN TestPolicyPrincipalUnmarshal/AWS_array_wildcard_(public_read) === RUN TestPolicyPrincipalUnmarshal/service === RUN TestPolicyPrincipalUnmarshal/canonical_user === RUN TestPolicyPrincipalUnmarshal/mixed_keys --- PASS: TestPolicyPrincipalUnmarshal (0.01s) --- PASS: TestPolicyPrincipalUnmarshal/bare_wildcard (0.00s) --- PASS: TestPolicyPrincipalUnmarshal/bare_arn (0.00s) --- PASS: TestPolicyPrincipalUnmarshal/bare_array (0.00s) --- PASS: TestPolicyPrincipalUnmarshal/AWS_single (0.00s) --- PASS: TestPolicyPrincipalUnmarshal/AWS_array (0.00s) --- PASS: TestPolicyPrincipalUnmarshal/AWS_wildcard (0.00s) --- PASS: TestPolicyPrincipalUnmarshal/AWS_array_wildcard_(public_read) (0.00s) --- PASS: TestPolicyPrincipalUnmarshal/service (0.00s) --- PASS: TestPolicyPrincipalUnmarshal/canonical_user (0.00s) --- PASS: TestPolicyPrincipalUnmarshal/mixed_keys (0.00s) === RUN TestPolicyPrincipalUnmarshalInvalid --- PASS: TestPolicyPrincipalUnmarshalInvalid (0.00s) === RUN TestNotPrincipalDynamicCompiledPath --- PASS: TestNotPrincipalDynamicCompiledPath (0.01s) === RUN TestPolicyPrincipalRoundTrip --- PASS: TestPolicyPrincipalRoundTrip (0.01s) === RUN TestBucketPolicyAWSObjectPrincipalEndToEnd --- PASS: TestBucketPolicyAWSObjectPrincipalEndToEnd (0.00s) === RUN TestBucketPolicySpecificAWSPrincipal --- PASS: TestBucketPolicySpecificAWSPrincipal (0.00s) === RUN TestBucketPolicyNotPrincipalDeny --- PASS: TestBucketPolicyNotPrincipalDeny (0.00s) === RUN TestBucketPolicyNotPrincipalCompiledPath --- PASS: TestBucketPolicyNotPrincipalCompiledPath (0.00s) === RUN TestPolicyBothPrincipalAndNotPrincipalRejected --- PASS: TestPolicyBothPrincipalAndNotPrincipalRejected (0.00s) === RUN TestSSEStringEqualsPresent --- PASS: TestSSEStringEqualsPresent (0.00s) === RUN TestSSEStringEqualsWrongValue --- PASS: TestSSEStringEqualsWrongValue (0.00s) === RUN TestSSENullConditionAbsent --- PASS: TestSSENullConditionAbsent (0.00s) === RUN TestSSENullConditionPresent --- PASS: TestSSENullConditionPresent (0.00s) === RUN TestSSECaseInsensitiveNormalizationAES256 --- PASS: TestSSECaseInsensitiveNormalizationAES256 (0.00s) === RUN TestSSECaseInsensitiveNormalizationKMS --- PASS: TestSSECaseInsensitiveNormalizationKMS (0.00s) === RUN TestSSEMultipartAES256Exempt --- PASS: TestSSEMultipartAES256Exempt (0.00s) === RUN TestSSEMultipartKMSExempt --- PASS: TestSSEMultipartKMSExempt (0.00s) === RUN TestSSEMultipartNoSSEDenied --- PASS: TestSSEMultipartNoSSEDenied (0.00s) === RUN TestSSEUploadPartCopyKMSExempt --- PASS: TestSSEUploadPartCopyKMSExempt (0.00s) === RUN TestSSEPutObjectStillBlockedWithoutHeader --- PASS: TestSSEPutObjectStillBlockedWithoutHeader (0.00s) === RUN TestCloneStringOrStringSliceCopiesBackingSlice --- PASS: TestCloneStringOrStringSliceCopiesBackingSlice (0.00s) PASS ok github.com/seaweedfs/seaweedfs/weed/s3api/policy_engine 7.103s === RUN TestEffectiveOwnership === RUN TestEffectiveOwnership/unset === RUN TestEffectiveOwnership/invalid === RUN TestEffectiveOwnership/object_writer === RUN TestEffectiveOwnership/bucket_owner_preferred === RUN TestEffectiveOwnership/bucket_owner_enforced --- PASS: TestEffectiveOwnership (0.05s) --- PASS: TestEffectiveOwnership/unset (0.00s) --- PASS: TestEffectiveOwnership/invalid (0.00s) --- PASS: TestEffectiveOwnership/object_writer (0.00s) --- PASS: TestEffectiveOwnership/bucket_owner_preferred (0.00s) --- PASS: TestEffectiveOwnership/bucket_owner_enforced (0.00s) === RUN TestNormalizeObjectKey === RUN TestNormalizeObjectKey/simple_key === RUN TestNormalizeObjectKey/key_with_leading_slash === RUN TestNormalizeObjectKey/key_with_directory === RUN TestNormalizeObjectKey/key_with_leading_slash_and_directory === RUN TestNormalizeObjectKey/key_with_duplicate_slashes === RUN TestNormalizeObjectKey/Windows_backslash_-_simple === RUN TestNormalizeObjectKey/Windows_backslash_-_nested === RUN TestNormalizeObjectKey/Windows_backslash_-_with_leading_slash === RUN TestNormalizeObjectKey/mixed_slashes === RUN TestNormalizeObjectKey/Windows_full_path_style_(edge_case) === RUN TestNormalizeObjectKey/empty_string === RUN TestNormalizeObjectKey/just_a_slash === RUN TestNormalizeObjectKey/just_a_backslash --- PASS: TestNormalizeObjectKey (0.01s) --- PASS: TestNormalizeObjectKey/simple_key (0.00s) --- PASS: TestNormalizeObjectKey/key_with_leading_slash (0.00s) --- PASS: TestNormalizeObjectKey/key_with_directory (0.00s) --- PASS: TestNormalizeObjectKey/key_with_leading_slash_and_directory (0.00s) --- PASS: TestNormalizeObjectKey/key_with_duplicate_slashes (0.00s) --- PASS: TestNormalizeObjectKey/Windows_backslash_-_simple (0.00s) --- PASS: TestNormalizeObjectKey/Windows_backslash_-_nested (0.00s) --- PASS: TestNormalizeObjectKey/Windows_backslash_-_with_leading_slash (0.00s) --- PASS: TestNormalizeObjectKey/mixed_slashes (0.00s) --- PASS: TestNormalizeObjectKey/Windows_full_path_style_(edge_case) (0.00s) --- PASS: TestNormalizeObjectKey/empty_string (0.00s) --- PASS: TestNormalizeObjectKey/just_a_slash (0.00s) --- PASS: TestNormalizeObjectKey/just_a_backslash (0.00s) === RUN TestIsValidObjectKey === RUN TestIsValidObjectKey/empty === RUN TestIsValidObjectKey/plain === RUN TestIsValidObjectKey/leading_slash === RUN TestIsValidObjectKey/trailing_slash === RUN TestIsValidObjectKey/hidden_file_ok === RUN TestIsValidObjectKey/dotdot_in_name_ok === RUN TestIsValidObjectKey/double_dots_inside_name === RUN TestIsValidObjectKey/bare_dotdot === RUN TestIsValidObjectKey/bare_dot === RUN TestIsValidObjectKey/leading_dotdot_segment === RUN TestIsValidObjectKey/leading_dot-slash === RUN TestIsValidObjectKey/nested_dotdot_segment === RUN TestIsValidObjectKey/trailing_dotdot_segment === RUN TestIsValidObjectKey/backslash_dotdot === RUN TestIsValidObjectKey/mixed-slash_dotdot === RUN TestIsValidObjectKey/dotdot_after_duplicate_slash === RUN TestIsValidObjectKey/nul_byte --- PASS: TestIsValidObjectKey (0.00s) --- PASS: TestIsValidObjectKey/empty (0.00s) --- PASS: TestIsValidObjectKey/plain (0.00s) --- PASS: TestIsValidObjectKey/leading_slash (0.00s) --- PASS: TestIsValidObjectKey/trailing_slash (0.00s) --- PASS: TestIsValidObjectKey/hidden_file_ok (0.00s) --- PASS: TestIsValidObjectKey/dotdot_in_name_ok (0.00s) --- PASS: TestIsValidObjectKey/double_dots_inside_name (0.00s) --- PASS: TestIsValidObjectKey/bare_dotdot (0.00s) --- PASS: TestIsValidObjectKey/bare_dot (0.00s) --- PASS: TestIsValidObjectKey/leading_dotdot_segment (0.00s) --- PASS: TestIsValidObjectKey/leading_dot-slash (0.00s) --- PASS: TestIsValidObjectKey/nested_dotdot_segment (0.00s) --- PASS: TestIsValidObjectKey/trailing_dotdot_segment (0.00s) --- PASS: TestIsValidObjectKey/backslash_dotdot (0.00s) --- PASS: TestIsValidObjectKey/mixed-slash_dotdot (0.00s) --- PASS: TestIsValidObjectKey/dotdot_after_duplicate_slash (0.00s) --- PASS: TestIsValidObjectKey/nul_byte (0.00s) === RUN TestIsValidBucketName === RUN TestIsValidBucketName/empty_ok === RUN TestIsValidBucketName/plain === RUN TestIsValidBucketName/name_containing_dots === RUN TestIsValidBucketName/bare_dot === RUN TestIsValidBucketName/bare_dotdot === RUN TestIsValidBucketName/with_slash === RUN TestIsValidBucketName/with_backslash === RUN TestIsValidBucketName/with_nul --- PASS: TestIsValidBucketName (0.01s) --- PASS: TestIsValidBucketName/empty_ok (0.00s) --- PASS: TestIsValidBucketName/plain (0.00s) --- PASS: TestIsValidBucketName/name_containing_dots (0.00s) --- PASS: TestIsValidBucketName/bare_dot (0.00s) --- PASS: TestIsValidBucketName/bare_dotdot (0.00s) --- PASS: TestIsValidBucketName/with_slash (0.00s) --- PASS: TestIsValidBucketName/with_backslash (0.00s) --- PASS: TestIsValidBucketName/with_nul (0.00s) === RUN TestIsValidPathSegment === RUN TestIsValidPathSegment/opaque-id_123 === RUN TestIsValidPathSegment/..hidden === RUN TestIsValidPathSegment/#00 === RUN TestIsValidPathSegment/. === RUN TestIsValidPathSegment/.. === RUN TestIsValidPathSegment/dir/value === RUN TestIsValidPathSegment/dir\value === RUN TestIsValidPathSegment/nul\x00value --- PASS: TestIsValidPathSegment (0.01s) --- PASS: TestIsValidPathSegment/opaque-id_123 (0.00s) --- PASS: TestIsValidPathSegment/..hidden (0.00s) --- PASS: TestIsValidPathSegment/#00 (0.00s) --- PASS: TestIsValidPathSegment/. (0.00s) --- PASS: TestIsValidPathSegment/.. (0.00s) --- PASS: TestIsValidPathSegment/dir/value (0.00s) --- PASS: TestIsValidPathSegment/dir\value (0.00s) --- PASS: TestIsValidPathSegment/nul\x00value (0.00s) === RUN TestRemoveDuplicateSlashes === RUN TestRemoveDuplicateSlashes/no_duplicates === RUN TestRemoveDuplicateSlashes/double_slash === RUN TestRemoveDuplicateSlashes/triple_slash === RUN TestRemoveDuplicateSlashes/multiple_duplicate_locations === RUN TestRemoveDuplicateSlashes/empty_string --- PASS: TestRemoveDuplicateSlashes (0.00s) --- PASS: TestRemoveDuplicateSlashes/no_duplicates (0.00s) --- PASS: TestRemoveDuplicateSlashes/double_slash (0.00s) --- PASS: TestRemoveDuplicateSlashes/triple_slash (0.00s) --- PASS: TestRemoveDuplicateSlashes/multiple_duplicate_locations (0.00s) --- PASS: TestRemoveDuplicateSlashes/empty_string (0.00s) === RUN TestIdentityHolderPropagation --- PASS: TestIdentityHolderPropagation (0.00s) === RUN TestEnsureIdentityHolderIdempotent --- PASS: TestEnsureIdentityHolderIdempotent (0.00s) === RUN TestGetIdentityNameFromContextWithoutHolder --- PASS: TestGetIdentityNameFromContextWithoutHolder (0.00s) PASS ok github.com/seaweedfs/seaweedfs/weed/s3api/s3_constants 7.615s ? github.com/seaweedfs/seaweedfs/weed/s3api/s3_objectlock [no test files] === RUN Test_verifyBucketName --- PASS: Test_verifyBucketName (0.00s) PASS ok github.com/seaweedfs/seaweedfs/weed/s3api/s3bucket 2.782s === RUN TestGetAccessLogUsesAmzRequestID --- PASS: TestGetAccessLogUsesAmzRequestID (0.08s) === RUN TestGetAccessLogRemoteIP === RUN TestGetAccessLogRemoteIP/falls_back_to_RemoteAddr_(port_stripped)_when_no_headers_set === RUN TestGetAccessLogRemoteIP/preserves_IPv6_host_from_RemoteAddr === RUN TestGetAccessLogRemoteIP/returns_RemoteAddr_unchanged_when_no_port_present === RUN TestGetAccessLogRemoteIP/uses_X-Real-IP_when_X-Forwarded-For_is_absent === RUN TestGetAccessLogRemoteIP/prefers_X-Forwarded-For_over_X-Real-IP === RUN TestGetAccessLogRemoteIP/uses_first_hop_in_X-Forwarded-For_chain === RUN TestGetAccessLogRemoteIP/skips_empty_leading_entries_in_X-Forwarded-For --- PASS: TestGetAccessLogRemoteIP (0.00s) --- PASS: TestGetAccessLogRemoteIP/falls_back_to_RemoteAddr_(port_stripped)_when_no_headers_set (0.00s) --- PASS: TestGetAccessLogRemoteIP/preserves_IPv6_host_from_RemoteAddr (0.00s) --- PASS: TestGetAccessLogRemoteIP/returns_RemoteAddr_unchanged_when_no_port_present (0.00s) --- PASS: TestGetAccessLogRemoteIP/uses_X-Real-IP_when_X-Forwarded-For_is_absent (0.00s) --- PASS: TestGetAccessLogRemoteIP/prefers_X-Forwarded-For_over_X-Real-IP (0.00s) --- PASS: TestGetAccessLogRemoteIP/uses_first_hop_in_X-Forwarded-For_chain (0.00s) --- PASS: TestGetAccessLogRemoteIP/skips_empty_leading_entries_in_X-Forwarded-For (0.00s) === RUN TestGetAccessLogRequesterFromFallback --- PASS: TestGetAccessLogRequesterFromFallback (0.00s) === RUN TestGetAccessLogRequesterAnonymous --- PASS: TestGetAccessLogRequesterAnonymous (0.00s) === RUN TestGetAccessLogRequesterArnForAssumedRole --- PASS: TestGetAccessLogRequesterArnForAssumedRole (0.00s) === RUN TestAuditTrackingFlag --- PASS: TestAuditTrackingFlag (0.00s) === RUN TestWriteErrorResponseReusesRequestID --- PASS: TestWriteErrorResponseReusesRequestID (0.09s) PASS ok github.com/seaweedfs/seaweedfs/weed/s3api/s3err 8.110s === RUN TestRuleActionKinds_SingleAction === RUN TestRuleActionKinds_SingleAction/expiration_days === RUN TestRuleActionKinds_SingleAction/expiration_date === RUN TestRuleActionKinds_SingleAction/expired_delete_marker === RUN TestRuleActionKinds_SingleAction/noncurrent_days === RUN TestRuleActionKinds_SingleAction/newer_noncurrent_alone === RUN TestRuleActionKinds_SingleAction/abort_mpu --- PASS: TestRuleActionKinds_SingleAction (0.09s) --- PASS: TestRuleActionKinds_SingleAction/expiration_days (0.06s) --- PASS: TestRuleActionKinds_SingleAction/expiration_date (0.00s) --- PASS: TestRuleActionKinds_SingleAction/expired_delete_marker (0.00s) --- PASS: TestRuleActionKinds_SingleAction/noncurrent_days (0.00s) --- PASS: TestRuleActionKinds_SingleAction/newer_noncurrent_alone (0.00s) --- PASS: TestRuleActionKinds_SingleAction/abort_mpu (0.00s) === RUN TestRuleActionKinds_MultiAction --- PASS: TestRuleActionKinds_MultiAction (0.00s) === RUN TestRuleActionKinds_NoncurrentDaysSubsumesNewerNoncurrent --- PASS: TestRuleActionKinds_NoncurrentDaysSubsumesNewerNoncurrent (0.00s) === RUN TestRuleActionKinds_Empty --- PASS: TestRuleActionKinds_Empty (0.00s) === RUN TestActionKind_StringIsStable --- PASS: TestActionKind_StringIsStable (0.00s) === RUN TestComputeDueAt_ExpirationDays --- PASS: TestComputeDueAt_ExpirationDays (0.00s) === RUN TestComputeDueAt_ExpirationDate --- PASS: TestComputeDueAt_ExpirationDate (0.00s) === RUN TestComputeDueAt_DeleteMarker_NotSoleSurvivor --- PASS: TestComputeDueAt_DeleteMarker_NotSoleSurvivor (0.00s) === RUN TestComputeDueAt_KindNotDeclared --- PASS: TestComputeDueAt_KindNotDeclared (0.00s) === RUN TestComputeDueAt_WrongShapeForKind --- PASS: TestComputeDueAt_WrongShapeForKind (0.00s) === RUN TestComputeDueAt_NoncurrentDeleteMarkerHonorsNoncurrentDays --- PASS: TestComputeDueAt_NoncurrentDeleteMarkerHonorsNoncurrentDays (0.00s) === RUN TestComputeDueAt_NoncurrentSuccessorMtime --- PASS: TestComputeDueAt_NoncurrentSuccessorMtime (0.00s) === RUN TestComputeDueAt_FilterRejects --- PASS: TestComputeDueAt_FilterRejects (0.00s) === RUN TestComputeDueAt_DisabledRule --- PASS: TestComputeDueAt_DisabledRule (0.00s) === RUN TestComputeDueAt_MPUInit --- PASS: TestComputeDueAt_MPUInit (0.00s) === RUN TestEvaluateAction_DisabledRuleNeverFires --- PASS: TestEvaluateAction_DisabledRuleNeverFires (0.00s) === RUN TestEvaluateAction_NilInputs --- PASS: TestEvaluateAction_NilInputs (0.00s) === RUN TestEvaluateAction_ExpirationDaysBoundary --- PASS: TestEvaluateAction_ExpirationDaysBoundary (0.00s) === RUN TestEvaluateAction_KindFiltersByDeclaredAction --- PASS: TestEvaluateAction_KindFiltersByDeclaredAction (0.00s) === RUN TestEvaluateAction_MultiActionRule_SiblingsIndependent --- PASS: TestEvaluateAction_MultiActionRule_SiblingsIndependent (0.00s) === RUN TestEvaluateAction_ExpirationDate --- PASS: TestEvaluateAction_ExpirationDate (0.00s) === RUN TestEvaluateAction_ExpiredObjectDeleteMarker --- PASS: TestEvaluateAction_ExpiredObjectDeleteMarker (0.00s) === RUN TestEvaluateAction_NoncurrentDeleteMarkerExpiresViaNoncurrentDays --- PASS: TestEvaluateAction_NoncurrentDeleteMarkerExpiresViaNoncurrentDays (0.00s) === RUN TestEvaluateAction_NoncurrentVersionDays --- PASS: TestEvaluateAction_NoncurrentVersionDays (0.00s) === RUN TestEvaluateAction_NoncurrentDaysFallsBackToModTime --- PASS: TestEvaluateAction_NoncurrentDaysFallsBackToModTime (0.00s) === RUN TestEvaluateAction_NoncurrentDays_WithKeepN_NilIndexIsNoOp --- PASS: TestEvaluateAction_NoncurrentDays_WithKeepN_NilIndexIsNoOp (0.00s) === RUN TestEvaluateAction_NewerNoncurrent_NilIndexIsNoOp --- PASS: TestEvaluateAction_NewerNoncurrent_NilIndexIsNoOp (0.00s) === RUN TestEvaluateAction_NewerNoncurrentCountOnly --- PASS: TestEvaluateAction_NewerNoncurrentCountOnly (0.00s) === RUN TestEvaluateAction_NoncurrentDaysAndCount --- PASS: TestEvaluateAction_NoncurrentDaysAndCount (0.00s) === RUN TestEvaluateAction_AbortMultipartUpload --- PASS: TestEvaluateAction_AbortMultipartUpload (0.00s) === RUN TestEvaluateAction_PrefixFilter --- PASS: TestEvaluateAction_PrefixFilter (0.00s) === RUN TestEvaluateAction_TagFilter --- PASS: TestEvaluateAction_TagFilter (0.00s) === RUN TestEvaluateAction_SizeFilter --- PASS: TestEvaluateAction_SizeFilter (0.00s) === RUN TestEvaluateAction_EmptyPrefixMatchesAll --- PASS: TestEvaluateAction_EmptyPrefixMatchesAll (0.00s) === RUN TestEvaluateAction_MPUInitDoesNotFireNoncurrent --- PASS: TestEvaluateAction_MPUInitDoesNotFireNoncurrent (0.00s) === RUN TestEventLogHorizon_PerActionIsIndependent === RUN TestEventLogHorizon_PerActionIsIndependent/expiration_days === RUN TestEventLogHorizon_PerActionIsIndependent/abort_mpu === RUN TestEventLogHorizon_PerActionIsIndependent/noncurrent_days === RUN TestEventLogHorizon_PerActionIsIndependent/expiration_date --- PASS: TestEventLogHorizon_PerActionIsIndependent (0.00s) --- PASS: TestEventLogHorizon_PerActionIsIndependent/expiration_days (0.00s) --- PASS: TestEventLogHorizon_PerActionIsIndependent/abort_mpu (0.00s) --- PASS: TestEventLogHorizon_PerActionIsIndependent/noncurrent_days (0.00s) --- PASS: TestEventLogHorizon_PerActionIsIndependent/expiration_date (0.00s) === RUN TestEventLogHorizon_NewerNoncurrentCountOnlyIsSmallDelay --- PASS: TestEventLogHorizon_NewerNoncurrentCountOnlyIsSmallDelay (0.00s) === RUN TestEventLogHorizon_NewerNoncurrentWithDaysIsZero --- PASS: TestEventLogHorizon_NewerNoncurrentWithDaysIsZero (0.00s) === RUN TestEventLogHorizon_ExpiredDeleteMarkerIsSmallDelay --- PASS: TestEventLogHorizon_ExpiredDeleteMarkerIsSmallDelay (0.00s) === RUN TestEventLogHorizon_DateKindReturnsZero --- PASS: TestEventLogHorizon_DateKindReturnsZero (0.00s) === RUN TestEventLogHorizon_NilRule --- PASS: TestEventLogHorizon_NilRule (0.00s) === RUN TestEventLogHorizon_KindNotDeclaredReturnsZero --- PASS: TestEventLogHorizon_KindNotDeclaredReturnsZero (0.00s) === RUN TestActionKind_StringUnspecifiedDefault --- PASS: TestActionKind_StringUnspecifiedDefault (0.00s) === RUN TestHashExtended_DirectFromLifecyclePackage --- PASS: TestHashExtended_DirectFromLifecyclePackage (0.00s) === RUN TestMinTriggerAge_PerKind === RUN TestMinTriggerAge_PerKind/expiration_days === RUN TestMinTriggerAge_PerKind/noncurrent_days === RUN TestMinTriggerAge_PerKind/abort_mpu === RUN TestMinTriggerAge_PerKind/expiration_date === RUN TestMinTriggerAge_PerKind/newer_noncurrent === RUN TestMinTriggerAge_PerKind/expired_delete_marker --- PASS: TestMinTriggerAge_PerKind (0.00s) --- PASS: TestMinTriggerAge_PerKind/expiration_days (0.00s) --- PASS: TestMinTriggerAge_PerKind/noncurrent_days (0.00s) --- PASS: TestMinTriggerAge_PerKind/abort_mpu (0.00s) --- PASS: TestMinTriggerAge_PerKind/expiration_date (0.00s) --- PASS: TestMinTriggerAge_PerKind/newer_noncurrent (0.00s) --- PASS: TestMinTriggerAge_PerKind/expired_delete_marker (0.00s) === RUN TestMinTriggerAge_KindNotSetReturnsZero --- PASS: TestMinTriggerAge_KindNotSetReturnsZero (0.00s) === RUN TestMinTriggerAge_DateOnlyReturnsZero --- PASS: TestMinTriggerAge_DateOnlyReturnsZero (0.00s) === RUN TestMinTriggerAge_NilRule --- PASS: TestMinTriggerAge_NilRule (0.00s) === RUN TestSuccessorFromEntryStamp_NilOrMissingReturnsZero --- PASS: TestSuccessorFromEntryStamp_NilOrMissingReturnsZero (0.00s) === RUN TestSuccessorFromEntryStamp_EmptyOrInvalidReturnsZero --- PASS: TestSuccessorFromEntryStamp_EmptyOrInvalidReturnsZero (0.00s) === RUN TestSuccessorFromEntryStamp_NonPositiveReturnsZero --- PASS: TestSuccessorFromEntryStamp_NonPositiveReturnsZero (0.00s) === RUN TestSuccessorFromEntryStamp_PositiveNanosRoundTrip --- PASS: TestSuccessorFromEntryStamp_PositiveNanosRoundTrip (0.00s) === RUN TestSuccessorFromEntryStamp_OrderedNanosStayOrdered --- PASS: TestSuccessorFromEntryStamp_OrderedNanosStayOrdered (0.00s) === RUN TestRuleHash_Stable --- PASS: TestRuleHash_Stable (0.00s) === RUN TestRuleHash_TagOrderInvariant --- PASS: TestRuleHash_TagOrderInvariant (0.00s) === RUN TestRuleHash_PrefixTrailingSlashMattersToHash --- PASS: TestRuleHash_PrefixTrailingSlashMattersToHash (0.00s) === RUN TestRuleHash_IDIgnored --- PASS: TestRuleHash_IDIgnored (0.00s) === RUN TestRuleHash_StatusIgnored --- PASS: TestRuleHash_StatusIgnored (0.00s) === RUN TestRuleHash_DifferentDaysHashDifferent --- PASS: TestRuleHash_DifferentDaysHashDifferent (0.00s) === RUN TestRuleHash_DifferentActionTypesHashDifferent --- PASS: TestRuleHash_DifferentActionTypesHashDifferent (0.00s) === RUN TestRuleHash_FilterMatters --- PASS: TestRuleHash_FilterMatters (0.00s) === RUN TestRuleHash_NilSafe --- PASS: TestRuleHash_NilSafe (0.00s) === RUN TestRuleHash_TagDelimiterCollisionResistant --- PASS: TestRuleHash_TagDelimiterCollisionResistant (0.00s) === RUN TestRuleHash_TagNewlineCollisionResistant --- PASS: TestRuleHash_TagNewlineCollisionResistant (0.00s) === RUN TestRuleHash_PrefixSeparatorIsolation --- PASS: TestRuleHash_PrefixSeparatorIsolation (0.00s) === RUN TestShardIDInRange --- PASS: TestShardIDInRange (0.00s) === RUN TestShardIDDeterministic --- PASS: TestShardIDDeterministic (0.00s) === RUN TestShardIDDistinctFromBucket --- PASS: TestShardIDDistinctFromBucket (0.00s) === RUN TestCompareVersionIds_EqualReturnsZero --- PASS: TestCompareVersionIds_EqualReturnsZero (0.00s) === RUN TestCompareVersionIds_NullSortsLast --- PASS: TestCompareVersionIds_NullSortsLast (0.00s) === RUN TestCompareVersionIds_BothNewFormatSmallerIsNewer --- PASS: TestCompareVersionIds_BothNewFormatSmallerIsNewer (0.00s) === RUN TestCompareVersionIds_BothOldFormatLargerIsNewer --- PASS: TestCompareVersionIds_BothOldFormatLargerIsNewer (0.00s) === RUN TestCompareVersionIds_MixedFormatComparesByTimestamp --- PASS: TestCompareVersionIds_MixedFormatComparesByTimestamp (0.00s) === RUN TestCompareVersionIds_MixedFormatEqualTimestampReturnsZero --- PASS: TestCompareVersionIds_MixedFormatEqualTimestampReturnsZero (0.00s) === RUN TestIsNewFormatVersionId_TooShortRejected --- PASS: TestIsNewFormatVersionId_TooShortRejected (0.00s) === RUN TestIsNewFormatVersionId_NullRejected --- PASS: TestIsNewFormatVersionId_NullRejected (0.00s) === RUN TestIsNewFormatVersionId_NonHexRejected --- PASS: TestIsNewFormatVersionId_NonHexRejected (0.00s) === RUN TestIsNewFormatVersionId_ThresholdBoundary --- PASS: TestIsNewFormatVersionId_ThresholdBoundary (0.00s) === RUN TestGetVersionTimestamp_ReturnsZeroOnMalformed --- PASS: TestGetVersionTimestamp_ReturnsZeroOnMalformed (0.00s) === RUN TestGetVersionTimestamp_OldFormatReturnsRawValue --- PASS: TestGetVersionTimestamp_OldFormatReturnsRawValue (0.00s) === RUN TestGetVersionTimestamp_NewFormatReturnsInvertedValue --- PASS: TestGetVersionTimestamp_NewFormatReturnsInvertedValue (0.00s) PASS ok github.com/seaweedfs/seaweedfs/weed/s3api/s3lifecycle 14.472s === RUN TestHasPrefix === RUN TestHasPrefix/matching_prefix === RUN TestHasPrefix/exact_match === RUN TestHasPrefix/non-matching_prefix === RUN TestHasPrefix/shorter_than_prefix === RUN TestHasPrefix/empty_prefix_matches_all === RUN TestHasPrefix/both_empty === RUN TestHasPrefix/empty_input_non-empty_prefix --- PASS: TestHasPrefix (0.04s) --- PASS: TestHasPrefix/matching_prefix (0.00s) --- PASS: TestHasPrefix/exact_match (0.00s) --- PASS: TestHasPrefix/non-matching_prefix (0.00s) --- PASS: TestHasPrefix/shorter_than_prefix (0.00s) --- PASS: TestHasPrefix/empty_prefix_matches_all (0.00s) --- PASS: TestHasPrefix/both_empty (0.00s) --- PASS: TestHasPrefix/empty_input_non-empty_prefix (0.00s) === RUN TestWalk_DispatchesDueActions --- PASS: TestWalk_DispatchesDueActions (0.00s) === RUN TestWalk_MultiActionRule_AllDueDispatched --- PASS: TestWalk_MultiActionRule_AllDueDispatched (0.00s) === RUN TestWalk_NotYetDueSkipped --- PASS: TestWalk_NotYetDueSkipped (0.00s) === RUN TestWalk_DateActionFiresAfterDate --- PASS: TestWalk_DateActionFiresAfterDate (0.00s) === RUN TestWalk_DateActionSkippedBeforeDate --- PASS: TestWalk_DateActionSkippedBeforeDate (0.00s) === RUN TestWalk_DirectoryEntriesSkipped --- PASS: TestWalk_DirectoryEntriesSkipped (0.00s) === RUN TestWalk_DisabledModeSkipped --- PASS: TestWalk_DisabledModeSkipped (0.01s) === RUN TestWalk_PendingBootstrapNotDispatched --- PASS: TestWalk_PendingBootstrapNotDispatched (0.00s) === RUN TestWalk_DispatchErrorHaltsAtCheckpoint W0824 22:18:22.373010 walker.go:173 lifecycle bootstrap: dispatch bk/a kind=expiration_days: dispatch boom --- PASS: TestWalk_DispatchErrorHaltsAtCheckpoint (0.01s) === RUN TestWalk_ResumeFromCheckpoint --- PASS: TestWalk_ResumeFromCheckpoint (0.00s) === RUN TestWalk_MPUInitDirMatchesByDestKey --- PASS: TestWalk_MPUInitDirMatchesByDestKey (0.00s) === RUN TestWalk_NonMPUDirectorySkipped --- PASS: TestWalk_NonMPUDirectorySkipped (0.00s) === RUN TestWalk_MPUInitDoesNotFireNoncurrent --- PASS: TestWalk_MPUInitDoesNotFireNoncurrent (0.00s) PASS ok github.com/seaweedfs/seaweedfs/weed/s3api/s3lifecycle/bootstrap 13.161s === RUN TestSummarizeShardCursorLag_AllColdStart --- PASS: TestSummarizeShardCursorLag_AllColdStart (0.07s) === RUN TestSummarizeShardCursorLag_PicksMaxAcrossShards --- PASS: TestSummarizeShardCursorLag_PicksMaxAcrossShards (0.00s) === RUN TestSummarizeShardCursorLag_PartialFill --- PASS: TestSummarizeShardCursorLag_PartialFill (0.00s) === RUN TestFilerCursorPersister_EmptyLoadReturnsNotFound --- PASS: TestFilerCursorPersister_EmptyLoadReturnsNotFound (0.03s) === RUN TestFilerCursorPersister_SaveLoadRoundTrip --- PASS: TestFilerCursorPersister_SaveLoadRoundTrip (0.01s) === RUN TestFilerCursorPersister_IsolatesShards --- PASS: TestFilerCursorPersister_IsolatesShards (0.00s) === RUN TestFilerCursorPersister_CorruptDataReturnsError --- PASS: TestFilerCursorPersister_CorruptDataReturnsError (0.00s) === RUN TestFilerCursorPersister_EmptyFileReturnsError --- PASS: TestFilerCursorPersister_EmptyFileReturnsError (0.00s) === RUN TestFilerCursorPersister_WrongVersionReturnsError --- PASS: TestFilerCursorPersister_WrongVersionReturnsError (0.00s) === RUN TestFilerCursorPersister_ShardIDMismatchReturnsError --- PASS: TestFilerCursorPersister_ShardIDMismatchReturnsError (0.03s) === RUN TestFilerCursorPersister_HashLengthMismatchReturnsError --- PASS: TestFilerCursorPersister_HashLengthMismatchReturnsError (0.00s) === RUN TestFilerCursorPersister_NilStoreErrors --- PASS: TestFilerCursorPersister_NilStoreErrors (0.00s) === RUN TestDispatch_FirstAttemptSucceeds --- PASS: TestDispatch_FirstAttemptSucceeds (0.00s) === RUN TestDispatch_TransportRetryThenSucceed --- PASS: TestDispatch_TransportRetryThenSucceed (0.53s) === RUN TestDispatch_ExhaustsRetriesReturnsError --- PASS: TestDispatch_ExhaustsRetriesReturnsError (0.56s) === RUN TestDispatch_ServerOutcomeRetryLaterIsNotRetried --- PASS: TestDispatch_ServerOutcomeRetryLaterIsNotRetried (0.00s) === RUN TestDispatch_ServerOutcomeBlockedNotRetried --- PASS: TestDispatch_ServerOutcomeBlockedNotRetried (0.00s) === RUN TestDispatch_ContextCancelledShortCircuits --- PASS: TestDispatch_ContextCancelledShortCircuits (0.00s) === RUN TestBuildDeleteRequest_RuleHashAndIdentity --- PASS: TestBuildDeleteRequest_RuleHashAndIdentity (0.00s) === RUN TestFilerListFunc_EmitsFlatFiles --- PASS: TestFilerListFunc_EmitsFlatFiles (0.07s) === RUN TestFilerListFunc_PropagatesTagsOnFlatFiles --- PASS: TestFilerListFunc_PropagatesTagsOnFlatFiles (0.00s) === RUN TestFilerListFunc_RecursesIntoSubdirs --- PASS: TestFilerListFunc_RecursesIntoSubdirs (0.00s) === RUN TestFilerListFunc_MPUInitEmitsWithDestKey --- PASS: TestFilerListFunc_MPUInitEmitsWithDestKey (0.00s) === RUN TestFilerListFunc_MPUInitWithoutDestKeyIsSkipped --- PASS: TestFilerListFunc_MPUInitWithoutDestKeyIsSkipped (0.00s) === RUN TestFilerListFunc_VersionedExpansionMarksLatestByPointer --- PASS: TestFilerListFunc_VersionedExpansionMarksLatestByPointer (0.00s) === RUN TestFilerListFunc_VersionedExpansionStalePointerFallsBackToNewestSibling --- PASS: TestFilerListFunc_VersionedExpansionStalePointerFallsBackToNewestSibling (0.00s) === RUN TestFilerListFunc_VersionedExpansionNoPointerNewestSiblingWins --- PASS: TestFilerListFunc_VersionedExpansionNoPointerNewestSiblingWins (0.00s) === RUN TestFilerListFunc_VersionedExpansionExplicitNullIsLatestWhenPointerMissing --- PASS: TestFilerListFunc_VersionedExpansionExplicitNullIsLatestWhenPointerMissing (0.00s) === RUN TestFilerListFunc_VersionedExpansionExplicitNullBeatsNewerSiblingWhenPointerMissing --- PASS: TestFilerListFunc_VersionedExpansionExplicitNullBeatsNewerSiblingWhenPointerMissing (0.00s) === RUN TestFilerListFunc_VersionsDirWithoutMarkersRecursesAsRegular --- PASS: TestFilerListFunc_VersionsDirWithoutMarkersRecursesAsRegular (0.00s) === RUN TestFilerListFunc_VersionedDeleteMarkerPropagates --- PASS: TestFilerListFunc_VersionedDeleteMarkerPropagates (0.00s) === RUN TestFilerListFunc_HonorsStart --- PASS: TestFilerListFunc_HonorsStart (0.00s) === RUN TestFilerListFunc_NilClient --- PASS: TestFilerListFunc_NilClient (0.00s) === RUN TestFilerListFunc_AttributesPropagate --- PASS: TestFilerListFunc_AttributesPropagate (0.00s) === RUN TestJitterBounds --- PASS: TestJitterBounds (0.00s) === RUN TestJitterZeroAndNegative --- PASS: TestJitterZeroAndNegative (0.00s) === RUN TestJitterTinyDuration --- PASS: TestJitterTinyDuration (0.00s) === RUN TestProcessMatches_NotYetDueDoesNotSuppressDueOnSameRule --- PASS: TestProcessMatches_NotYetDueDoesNotSuppressDueOnSameRule (0.00s) === RUN TestProcessMatches_OrderingDoesNotMatter --- PASS: TestProcessMatches_OrderingDoesNotMatter (0.00s) === RUN TestProcessMatches_HaltOnServerOutcomeStopsRemaining --- PASS: TestProcessMatches_HaltOnServerOutcomeStopsRemaining (0.00s) === RUN TestProcessMatches_EmptyMatchesIsNoop --- PASS: TestProcessMatches_EmptyMatchesIsNoop (0.00s) === RUN TestProcessMatches_AllDueNoSkippedFlag --- PASS: TestProcessMatches_AllDueNoSkippedFlag (0.00s) === RUN TestProcessMatches_DispatchCounterIncrements --- PASS: TestProcessMatches_DispatchCounterIncrements (0.00s) === RUN TestRun_EndsOnIdleSubscription I0824 22:18:24.568050 run.go:226 daily_run: status=ok shards=3 errors=0 duration=47ms cursor_lag_max=0s walked_max_age=cold --- PASS: TestRun_EndsOnIdleSubscription (0.07s) === RUN TestRun_ReaderFailureFailsThePass W0824 22:18:24.609421 run.go:197 daily_run: shared meta-log subscription: subscribe: rpc error: code = Unavailable desc = filer down I0824 22:18:24.612984 run.go:226 daily_run: status=error shards=2 errors=1 duration=12ms cursor_lag_max=0s walked_max_age=cold --- PASS: TestRun_ReaderFailureFailsThePass (0.01s) === RUN TestRun_CappedPassIsNotAFailure I0824 22:18:24.824716 run.go:226 daily_run: status=ok shards=2 errors=0 duration=210ms cursor_lag_max=0s walked_max_age=cold --- PASS: TestRun_CappedPassIsNotAFailure (0.21s) === RUN TestRun_StalledSubscriptionTimesOutThePass W0824 22:18:24.892751 run.go:197 daily_run: shared meta-log subscription: reader: metadata receive timeout after 50ms I0824 22:18:24.892906 run.go:226 daily_run: status=error shards=2 errors=1 duration=67ms cursor_lag_max=0s walked_max_age=cold --- PASS: TestRun_StalledSubscriptionTimesOutThePass (0.07s) === RUN TestValidate_RejectsNegativeSubscriptionReceiveTimeout --- PASS: TestValidate_RejectsNegativeSubscriptionReceiveTimeout (0.00s) === RUN TestRun_LateDeadlineDoesNotMaskReaderFailure W0824 22:18:25.206051 run.go:197 daily_run: shared meta-log subscription: subscribe: rpc error: code = Unavailable desc = filer down I0824 22:18:25.206331 run.go:226 daily_run: status=error shards=2 errors=1 duration=308ms cursor_lag_max=0s walked_max_age=cold --- PASS: TestRun_LateDeadlineDoesNotMaskReaderFailure (0.31s) === RUN TestRun_ServerSideCancelFailsThePass W0824 22:18:25.209600 run.go:197 daily_run: shared meta-log subscription: subscribe: rpc error: code = Canceled desc = context canceled I0824 22:18:25.209725 run.go:226 daily_run: status=error shards=2 errors=1 duration=1ms cursor_lag_max=0s walked_max_age=cold --- PASS: TestRun_ServerSideCancelFailsThePass (0.00s) === RUN TestRun_HaltedShardDoesNotStarveOthers I0824 22:18:25.226323 run.go:226 daily_run: status=ok shards=2 errors=0 duration=10ms cursor_lag_max=72h0m0s walked_max_age=cold --- PASS: TestRun_HaltedShardDoesNotStarveOthers (0.02s) === RUN TestWalkBuckets_DispatchesOnlyShardMatchingEntries --- PASS: TestWalkBuckets_DispatchesOnlyShardMatchingEntries (0.00s) === RUN TestWalkBuckets_NilGuards --- PASS: TestWalkBuckets_NilGuards (0.00s) === RUN TestWalkBuckets_OneBucketErrorDoesNotStopOthers --- PASS: TestWalkBuckets_OneBucketErrorDoesNotStopOthers (0.00s) === RUN TestWalkBuckets_HonorsContextCancellationBetweenBuckets --- PASS: TestWalkBuckets_HonorsContextCancellationBetweenBuckets (0.00s) === RUN TestEntryShardID_MPUUsesDestKey --- PASS: TestEntryShardID_MPUUsesDestKey (0.00s) === RUN TestWalkerDispatcher_NonVersionedSendsExpectedRequest --- PASS: TestWalkerDispatcher_NonVersionedSendsExpectedRequest (0.00s) === RUN TestWalkerDispatcher_VersionedPassesVersionID --- PASS: TestWalkerDispatcher_VersionedPassesVersionID (0.00s) === RUN TestWalkerDispatcher_MPUInitUsesUploadsPath --- PASS: TestWalkerDispatcher_MPUInitUsesUploadsPath (0.00s) === RUN TestWalkerDispatcher_MPUInitEmptyDestKeyErrors --- PASS: TestWalkerDispatcher_MPUInitEmptyDestKeyErrors (0.03s) === RUN TestWalkerDispatcher_AcceptsAllResolvedOutcomes --- PASS: TestWalkerDispatcher_AcceptsAllResolvedOutcomes (0.00s) === RUN TestWalkerDispatcher_UnresolvedOutcomeReturnsError --- PASS: TestWalkerDispatcher_UnresolvedOutcomeReturnsError (0.00s) === RUN TestWalkerDispatcher_TransportErrorReturnsWrappedError --- PASS: TestWalkerDispatcher_TransportErrorReturnsWrappedError (0.00s) === RUN TestWalkerDispatcher_NilResponseReturnsError --- PASS: TestWalkerDispatcher_NilResponseReturnsError (0.00s) === RUN TestWalkerDispatcher_LimiterWaitsBeforeDispatch --- PASS: TestWalkerDispatcher_LimiterWaitsBeforeDispatch (0.07s) === RUN TestWalkerDispatcher_LimiterContextCancelHaltsWalker --- PASS: TestWalkerDispatcher_LimiterContextCancelHaltsWalker (0.00s) === RUN TestWalkerDispatcher_NilGuardsReturnError --- PASS: TestWalkerDispatcher_NilGuardsReturnError (0.00s) === RUN TestWalkerDue === RUN TestWalkerDue/interval_zero_always_due === RUN TestWalkerDue/never_walked_is_due === RUN TestWalkerDue/interval_elapsed === RUN TestWalkerDue/interval_exactly_elapsed === RUN TestWalkerDue/interval_not_yet_elapsed === RUN TestWalkerDue/future_lastWalked --- PASS: TestWalkerDue (0.00s) --- PASS: TestWalkerDue/interval_zero_always_due (0.00s) --- PASS: TestWalkerDue/never_walked_is_due (0.00s) --- PASS: TestWalkerDue/interval_elapsed (0.00s) --- PASS: TestWalkerDue/interval_exactly_elapsed (0.00s) --- PASS: TestWalkerDue/interval_not_yet_elapsed (0.00s) --- PASS: TestWalkerDue/future_lastWalked (0.00s) === RUN TestRunShard_WalkerThrottle === RUN TestRunShard_WalkerThrottle/interval=0_fires_every_pass === RUN TestRunShard_WalkerThrottle/throttled:_second_pass_within_interval === RUN TestRunShard_WalkerThrottle/throttled:_second_pass_past_interval --- PASS: TestRunShard_WalkerThrottle (0.00s) --- PASS: TestRunShard_WalkerThrottle/interval=0_fires_every_pass (0.00s) --- PASS: TestRunShard_WalkerThrottle/throttled:_second_pass_within_interval (0.00s) --- PASS: TestRunShard_WalkerThrottle/throttled:_second_pass_past_interval (0.00s) === RUN TestValidate_RejectsNegativeWalkerInterval --- PASS: TestValidate_RejectsNegativeWalkerInterval (0.00s) === RUN TestRunShard_ColdStartDoesNotDoubleWalk --- PASS: TestRunShard_ColdStartDoesNotDoubleWalk (0.00s) === RUN TestRunShard_RecoveryWalkerSetsLastWalkedAnchor --- PASS: TestRunShard_RecoveryWalkerSetsLastWalkedAnchor (0.00s) === RUN TestRunShard_WalkerInvokedOnRecoveryBranch --- PASS: TestRunShard_WalkerInvokedOnRecoveryBranch (0.00s) === RUN TestRunShard_NilWalkerOnRecoveryIsNoop --- PASS: TestRunShard_NilWalkerOnRecoveryIsNoop (0.00s) === RUN TestRunShard_WalkerErrorPropagates --- PASS: TestRunShard_WalkerErrorPropagates (0.00s) PASS ok github.com/seaweedfs/seaweedfs/weed/s3api/s3lifecycle/dailyrun 16.100s === RUN TestSurvivors_NoVersionsNoNullReturnsZero --- PASS: TestSurvivors_NoVersionsNoNullReturnsZero (0.06s) === RUN TestSurvivors_OneVersionPopulatesLoneEntry --- PASS: TestSurvivors_OneVersionPopulatesLoneEntry (0.05s) === RUN TestSurvivors_MoreThanOneClearsLoneEntry --- PASS: TestSurvivors_MoreThanOneClearsLoneEntry (0.03s) === RUN TestSurvivors_NullVersionDetectedFromBareRegularFile --- PASS: TestSurvivors_NullVersionDetectedFromBareRegularFile (0.00s) === RUN TestSurvivors_DirectoryKeyMarkerCountsAsNull --- PASS: TestSurvivors_DirectoryKeyMarkerCountsAsNull (0.00s) === RUN TestSurvivors_PlainDirectoryDoesNotCount --- PASS: TestSurvivors_PlainDirectoryDoesNotCount (0.00s) === RUN TestSurvivors_VersionsListingErrorPropagates --- PASS: TestSurvivors_VersionsListingErrorPropagates (0.00s) === RUN TestSurvivors_NullLookupErrorPropagates --- PASS: TestSurvivors_NullLookupErrorPropagates (0.00s) === RUN TestListVersions_NotFoundReturnsNil --- PASS: TestListVersions_NotFoundReturnsNil (0.00s) === RUN TestListVersions_FiltersOutDirectoriesAndMissingVersionId --- PASS: TestListVersions_FiltersOutDirectoriesAndMissingVersionId (0.00s) === RUN TestListVersions_PaginatesAcrossPages --- PASS: TestListVersions_PaginatesAcrossPages (0.05s) === RUN TestListVersions_ErrorPropagates --- PASS: TestListVersions_ErrorPropagates (0.00s) === RUN TestLookupNullVersion_RegularFileExplicitFalse --- PASS: TestLookupNullVersion_RegularFileExplicitFalse (0.00s) === RUN TestLookupNullVersion_ExplicitNullFlag --- PASS: TestLookupNullVersion_ExplicitNullFlag (0.00s) === RUN TestLookupNullVersion_DirectoryKeyMarkerQualifies --- PASS: TestLookupNullVersion_DirectoryKeyMarkerQualifies (0.00s) === RUN TestLookupNullVersion_PlainDirectoryRejected --- PASS: TestLookupNullVersion_PlainDirectoryRejected (0.00s) === RUN TestLookupNullVersion_NotFoundCollapses --- PASS: TestLookupNullVersion_NotFoundCollapses (0.00s) === RUN TestLookupNullVersion_TransportErrorPropagates --- PASS: TestLookupNullVersion_TransportErrorPropagates (0.00s) === RUN TestLookupVersion_EmptyVersionIdReturnsNil --- PASS: TestLookupVersion_EmptyVersionIdReturnsNil (0.00s) === RUN TestLookupVersion_AppliesVPrefix --- PASS: TestLookupVersion_AppliesVPrefix (0.00s) === RUN TestLookupVersion_NotFoundCollapses --- PASS: TestLookupVersion_NotFoundCollapses (0.00s) === RUN TestLookupVersion_TransportErrorPropagates --- PASS: TestLookupVersion_TransportErrorPropagates (0.00s) PASS ok github.com/seaweedfs/seaweedfs/weed/s3api/s3lifecycle/dispatcher 15.814s === RUN TestRulePredicateSensitive_NilRuleReturnsFalse --- PASS: TestRulePredicateSensitive_NilRuleReturnsFalse (0.03s) === RUN TestRulePredicateSensitive_NoFilterTagsReturnsFalse --- PASS: TestRulePredicateSensitive_NoFilterTagsReturnsFalse (0.00s) === RUN TestRulePredicateSensitive_EmptyFilterTagsReturnsFalse --- PASS: TestRulePredicateSensitive_EmptyFilterTagsReturnsFalse (0.00s) === RUN TestRulePredicateSensitive_PopulatedFilterTagsReturnsTrue --- PASS: TestRulePredicateSensitive_PopulatedFilterTagsReturnsTrue (0.00s) === RUN TestCompile_SingleRuleSingleAction --- PASS: TestCompile_SingleRuleSingleAction (0.03s) === RUN TestCompile_MultiAction_SiblingsHaveOwnEntries --- PASS: TestCompile_MultiAction_SiblingsHaveOwnEntries (0.00s) === RUN TestCompile_BootstrapPendingIndexedButInactive --- PASS: TestCompile_BootstrapPendingIndexedButInactive (0.00s) === RUN TestCompile_RetentionGate --- PASS: TestCompile_RetentionGate (0.00s) === RUN TestCompile_RetentionUnboundedNeverGates --- PASS: TestCompile_RetentionUnboundedNeverGates (0.00s) === RUN TestCompile_SiblingsDegradeIndependently --- PASS: TestCompile_SiblingsDegradeIndependently (0.00s) === RUN TestCompile_PriorModePreservedOverDecideMode --- PASS: TestCompile_PriorModePreservedOverDecideMode (0.00s) === RUN TestCompile_ExpirationDateScansAtDate --- PASS: TestCompile_ExpirationDateScansAtDate (0.00s) === RUN TestCompile_DisabledRuleNeverActivates --- PASS: TestCompile_DisabledRuleNeverActivates (0.00s) === RUN TestSnapshot_MarkActiveFlipsRoutingFilter --- PASS: TestSnapshot_MarkActiveFlipsRoutingFilter (0.00s) === RUN TestCompile_CrossBucketIdenticalRulesDoNotCollide --- PASS: TestCompile_CrossBucketIdenticalRulesDoNotCollide (0.00s) === RUN TestEngine_SnapshotAtomicSwap --- PASS: TestEngine_SnapshotAtomicSwap (0.00s) === RUN TestCompile_DelayGroupsDedupeAcrossBuckets --- PASS: TestCompile_DelayGroupsDedupeAcrossBuckets (0.06s) === RUN TestReplayContentHash_EmptyWhenNoReplayActions --- PASS: TestReplayContentHash_EmptyWhenNoReplayActions (0.00s) === RUN TestReplayContentHash_PartitionIndependent --- PASS: TestReplayContentHash_PartitionIndependent (0.00s) === RUN TestReplayContentHash_ChangesOnRuleContentEdit --- PASS: TestReplayContentHash_ChangesOnRuleContentEdit (0.00s) === RUN TestReplayContentHash_StableAcrossRuleReorder --- PASS: TestReplayContentHash_StableAcrossRuleReorder (0.00s) === RUN TestReplayContentHash_ExcludesWalkerOnlyAndDisabled --- PASS: TestReplayContentHash_ExcludesWalkerOnlyAndDisabled (0.00s) === RUN TestPromotedHash_EmptyWhenNoneAreInWalk --- PASS: TestPromotedHash_EmptyWhenNoneAreInWalk (0.00s) === RUN TestPromotedHash_ChangesOnReplayToWalkPromotion --- PASS: TestPromotedHash_ChangesOnReplayToWalkPromotion (0.00s) === RUN TestPromotedHash_ChangesOnWalkToReplayDemotion --- PASS: TestPromotedHash_ChangesOnWalkToReplayDemotion (0.00s) === RUN TestPromotedHash_StableWhenContentUnchangedAndPartitionStays --- PASS: TestPromotedHash_StableWhenContentUnchangedAndPartitionStays (0.00s) === RUN TestPromotedHash_MatchesRulesForShardWalkMembership --- PASS: TestPromotedHash_MatchesRulesForShardWalkMembership (0.00s) === RUN TestMaxEffectiveTTL_NilAndEmpty --- PASS: TestMaxEffectiveTTL_NilAndEmpty (0.00s) === RUN TestMaxEffectiveTTL_ReturnsMaxAcrossActiveReplay --- PASS: TestMaxEffectiveTTL_ReturnsMaxAcrossActiveReplay (0.00s) === RUN TestMaxEffectiveTTL_OperatesOnReplayView --- PASS: TestMaxEffectiveTTL_OperatesOnReplayView (0.00s) === RUN TestMaxEffectiveTTL_IgnoresInactiveActions --- PASS: TestMaxEffectiveTTL_IgnoresInactiveActions (0.00s) === RUN TestPrefixMatches_EmptyPrefixAllowsEverything --- PASS: TestPrefixMatches_EmptyPrefixAllowsEverything (0.00s) === RUN TestPrefixMatches_HonorsExactPrefix --- PASS: TestPrefixMatches_HonorsExactPrefix (0.00s) === RUN TestPrefixMatches_RejectsNonMatchingPath --- PASS: TestPrefixMatches_RejectsNonMatchingPath (0.00s) === RUN TestPrefixMatches_PathShorterThanPrefixIsReject --- PASS: TestPrefixMatches_PathShorterThanPrefixIsReject (0.00s) === RUN TestFilterAllows_NoFiltersAcceptsAnything --- PASS: TestFilterAllows_NoFiltersAcceptsAnything (0.00s) === RUN TestFilterAllows_SizeGreaterThanGate --- PASS: TestFilterAllows_SizeGreaterThanGate (0.00s) === RUN TestFilterAllows_SizeLessThanGate --- PASS: TestFilterAllows_SizeLessThanGate (0.00s) === RUN TestFilterAllows_ZeroSizeFilterIsDisabled --- PASS: TestFilterAllows_ZeroSizeFilterIsDisabled (0.00s) === RUN TestFilterAllows_RequiredTagPresent --- PASS: TestFilterAllows_RequiredTagPresent (0.00s) === RUN TestFilterAllows_RequiredTagMissingOrWrong --- PASS: TestFilterAllows_RequiredTagMissingOrWrong (0.00s) === RUN TestFilterAllows_AllRequiredTagsMustMatch --- PASS: TestFilterAllows_AllRequiredTagsMustMatch (0.00s) === RUN TestFilterAllows_SizeAndTagAreANDed --- PASS: TestFilterAllows_SizeAndTagAreANDed (0.00s) === RUN TestMatchOriginalWrite_NilEventReturnsNil --- PASS: TestMatchOriginalWrite_NilEventReturnsNil (0.00s) === RUN TestMatchOriginalWrite_WrongShapeReturnsNil --- PASS: TestMatchOriginalWrite_WrongShapeReturnsNil (0.00s) === RUN TestMatchOriginalWrite_DelayMismatchReturnsNil --- PASS: TestMatchOriginalWrite_DelayMismatchReturnsNil (0.00s) === RUN TestMatchPredicateChange_NilEventReturnsNil --- PASS: TestMatchPredicateChange_NilEventReturnsNil (0.00s) === RUN TestMatchPredicateChange_WrongShapeReturnsNil --- PASS: TestMatchPredicateChange_WrongShapeReturnsNil (0.00s) === RUN TestMatchPredicateChange_NoTagSensitiveActionsReturnsNil --- PASS: TestMatchPredicateChange_NoTagSensitiveActionsReturnsNil (0.00s) === RUN TestMatchPath_UnknownBucketReturnsNil --- PASS: TestMatchPath_UnknownBucketReturnsNil (0.00s) === RUN TestMatchPath_NilEventStillAppliesPrefix --- PASS: TestMatchPath_NilEventStillAppliesPrefix (0.00s) === RUN TestMatchPath_FilterRespectedWhenEventProvided --- PASS: TestMatchPath_FilterRespectedWhenEventProvided (0.00s) === RUN TestFilterMatching_BucketScoping --- PASS: TestFilterMatching_BucketScoping (0.00s) === RUN TestFilterMatching_AbortMPURequiresMPUInit --- PASS: TestFilterMatching_AbortMPURequiresMPUInit (0.00s) === RUN TestFilterMatching_ExpiredDeleteMarkerRequiresMarkerLatest --- PASS: TestFilterMatching_ExpiredDeleteMarkerRequiresMarkerLatest (0.00s) === RUN TestMatchOriginalWrite_DelayGroupRoutes --- PASS: TestMatchOriginalWrite_DelayGroupRoutes (0.00s) === RUN TestMatchOriginalWrite_PrefixFilter --- PASS: TestMatchOriginalWrite_PrefixFilter (0.00s) === RUN TestMatchOriginalWrite_MarkActiveBecomesRoutable --- PASS: TestMatchOriginalWrite_MarkActiveBecomesRoutable (0.00s) === RUN TestMatchOriginalWrite_AbortMPUOnlyOnMPUInit --- PASS: TestMatchOriginalWrite_AbortMPUOnlyOnMPUInit (0.00s) === RUN TestMatchPredicateChange_OnlyTagSensitiveActions --- PASS: TestMatchPredicateChange_OnlyTagSensitiveActions (0.00s) === RUN TestMatchPath_BootstrapWalkerSeesAllActiveActions --- PASS: TestMatchPath_BootstrapWalkerSeesAllActiveActions (0.02s) === RUN TestMatchPath_PrefixMismatchExcludes --- PASS: TestMatchPath_PrefixMismatchExcludes (0.00s) === RUN TestMatchPath_FilterRejectsByTag --- PASS: TestMatchPath_FilterRejectsByTag (0.00s) === RUN TestDecideMode_NilRuleIsDisabled --- PASS: TestDecideMode_NilRuleIsDisabled (0.00s) === RUN TestDecideMode_DisabledStatusIsDisabled --- PASS: TestDecideMode_DisabledStatusIsDisabled (0.00s) === RUN TestDecideMode_ExpirationDateIsScanAtDate --- PASS: TestDecideMode_ExpirationDateIsScanAtDate (0.00s) === RUN TestDecideMode_UnboundedRetentionIsEventDriven --- PASS: TestDecideMode_UnboundedRetentionIsEventDriven (0.00s) === RUN TestDecideMode_HorizonWithinRetentionIsEventDriven --- PASS: TestDecideMode_HorizonWithinRetentionIsEventDriven (0.00s) === RUN TestDecideMode_HorizonExceedsRetentionIsScanOnly --- PASS: TestDecideMode_HorizonExceedsRetentionIsScanOnly (0.00s) === RUN TestDecideMode_LookbackPushesAcrossThreshold --- PASS: TestDecideMode_LookbackPushesAcrossThreshold (0.00s) === RUN TestDecideMode_ZeroHorizonNeverGates --- PASS: TestDecideMode_ZeroHorizonNeverGates (0.00s) === RUN TestRuleMode_StringRendersDocumentedNames === RUN TestRuleMode_StringRendersDocumentedNames/unspecified === RUN TestRuleMode_StringRendersDocumentedNames/event_driven === RUN TestRuleMode_StringRendersDocumentedNames/scan_at_date === RUN TestRuleMode_StringRendersDocumentedNames/scan_only === RUN TestRuleMode_StringRendersDocumentedNames/disabled === RUN TestRuleMode_StringRendersDocumentedNames/pending_bootstrap --- PASS: TestRuleMode_StringRendersDocumentedNames (0.00s) --- PASS: TestRuleMode_StringRendersDocumentedNames/unspecified (0.00s) --- PASS: TestRuleMode_StringRendersDocumentedNames/event_driven (0.00s) --- PASS: TestRuleMode_StringRendersDocumentedNames/scan_at_date (0.00s) --- PASS: TestRuleMode_StringRendersDocumentedNames/scan_only (0.00s) --- PASS: TestRuleMode_StringRendersDocumentedNames/disabled (0.00s) --- PASS: TestRuleMode_StringRendersDocumentedNames/pending_bootstrap (0.00s) === RUN TestRuleMode_StringFallsToUnspecifiedForUnknown --- PASS: TestRuleMode_StringFallsToUnspecifiedForUnknown (0.00s) === RUN TestSnapshot_OriginalDelayGroupsExposesCompiledGroups --- PASS: TestSnapshot_OriginalDelayGroupsExposesCompiledGroups (0.00s) === RUN TestSnapshot_OriginalDelayGroupsScanOnlyExcluded --- PASS: TestSnapshot_OriginalDelayGroupsScanOnlyExcluded (0.00s) === RUN TestSnapshot_PredicateActionsContainsTagSensitive --- PASS: TestSnapshot_PredicateActionsContainsTagSensitive (0.00s) === RUN TestSnapshot_PredicateActionsEmptyForNonTagSensitiveRule --- PASS: TestSnapshot_PredicateActionsEmptyForNonTagSensitiveRule (0.00s) === RUN TestSnapshot_DateActionsContainsExpirationDate --- PASS: TestSnapshot_DateActionsContainsExpirationDate (0.00s) === RUN TestSnapshot_DateActionsEmptyForNonDateRule --- PASS: TestSnapshot_DateActionsEmptyForNonDateRule (0.00s) === RUN TestSnapshot_MarkActiveUnknownKeyIsNoOp --- PASS: TestSnapshot_MarkActiveUnknownKeyIsNoOp (0.00s) === RUN TestSnapshot_MarkActiveFlipsCompiledActionToActive --- PASS: TestSnapshot_MarkActiveFlipsCompiledActionToActive (0.00s) === RUN TestSnapshot_BucketIndexedActionKeysCoverAllKinds --- PASS: TestSnapshot_BucketIndexedActionKeysCoverAllKinds (0.00s) === RUN TestCurrentSnapshot_NilWhenNoEngineRegistered --- PASS: TestCurrentSnapshot_NilWhenNoEngineRegistered (0.00s) === RUN TestCurrentSnapshot_ReturnsRegisteredEngineSnapshot --- PASS: TestCurrentSnapshot_ReturnsRegisteredEngineSnapshot (0.00s) === RUN TestRulesForShard_NilSnapshot --- PASS: TestRulesForShard_NilSnapshot (0.00s) === RUN TestRulesForShard_ReplayPartitionMembership --- PASS: TestRulesForShard_ReplayPartitionMembership (0.00s) === RUN TestRulesForShard_WalkOnlyKindsRouteToWalk --- PASS: TestRulesForShard_WalkOnlyKindsRouteToWalk (0.00s) === RUN TestRulesForShard_MultiActionXMLRuleSplits --- PASS: TestRulesForShard_MultiActionXMLRuleSplits (0.00s) === RUN TestRulesForShard_ScanOnlyPromotionRoutesToWalk --- PASS: TestRulesForShard_ScanOnlyPromotionRoutesToWalk (0.00s) === RUN TestRulesForShard_ReplayRewritesScanOnlyMode --- PASS: TestRulesForShard_ReplayRewritesScanOnlyMode (0.00s) === RUN TestRulesForShard_DisabledRuleExcludedFromBothViews --- PASS: TestRulesForShard_DisabledRuleExcludedFromBothViews (0.00s) === RUN TestRulesForShard_ClonesAreIndependentOfBase --- PASS: TestRulesForShard_ClonesAreIndependentOfBase (0.00s) === RUN TestRulesForShard_ZeroRetentionRoutesAllReplayKindsToWalk --- PASS: TestRulesForShard_ZeroRetentionRoutesAllReplayKindsToWalk (0.00s) === RUN TestRecoveryView_ActivatesEveryNonDisabledAction --- PASS: TestRecoveryView_ActivatesEveryNonDisabledAction (0.00s) === RUN TestRecoveryView_PreservesBaseModeForScanAtDate --- PASS: TestRecoveryView_PreservesBaseModeForScanAtDate (0.00s) === RUN TestRecoveryView_ExcludesDisabledRules --- PASS: TestRecoveryView_ExcludesDisabledRules (0.00s) === RUN TestRecoveryView_ClonesAreIndependentOfBase --- PASS: TestRecoveryView_ClonesAreIndependentOfBase (0.00s) === RUN TestRecoveryView_NilSnapshot --- PASS: TestRecoveryView_NilSnapshot (0.00s) PASS ok github.com/seaweedfs/seaweedfs/weed/s3api/s3lifecycle/engine 17.446s === RUN TestEventOption_WithModTime_AppliesToOldEntryOnDelete --- PASS: TestEventOption_WithModTime_AppliesToOldEntryOnDelete (0.08s) === RUN TestEventOption_WithTtlSec_AppliesToOldEntryOnDelete --- PASS: TestEventOption_WithTtlSec_AppliesToOldEntryOnDelete (0.03s) === RUN TestEventOption_WithVersionID_AppliesToOldEntryOnDelete --- PASS: TestEventOption_WithVersionID_AppliesToOldEntryOnDelete (0.00s) === RUN TestEventOption_WithExtended_AppliesToOldEntryOnDelete --- PASS: TestEventOption_WithExtended_AppliesToOldEntryOnDelete (0.00s) === RUN TestEventOption_WithChunks_AppliesToOldEntryOnDelete --- PASS: TestEventOption_WithChunks_AppliesToOldEntryOnDelete (0.00s) === RUN TestEventOption_WithBootstrapVersion --- PASS: TestEventOption_WithBootstrapVersion (0.00s) === RUN TestEventOption_NoPanicOnEmptyEvent --- PASS: TestEventOption_NoPanicOnEmptyEvent (0.00s) === RUN TestNewCreate_PopulatesNewEntryOnly --- PASS: TestNewCreate_PopulatesNewEntryOnly (0.00s) === RUN TestNewDelete_PopulatesOldEntryOnly --- PASS: TestNewDelete_PopulatesOldEntryOnly (0.00s) === RUN TestNewUpdate_PopulatesBothEntries --- PASS: TestNewUpdate_PopulatesBothEntries (0.00s) === RUN TestNewCreate_NestedKeyUsesLeafName --- PASS: TestNewCreate_NestedKeyUsesLeafName (0.00s) === RUN TestNewCreate_DirectoryKeyUsesSlashlessLeaf --- PASS: TestNewCreate_DirectoryKeyUsesSlashlessLeaf (0.00s) === RUN TestNewCreate_ShardIDDerivedFromKey --- PASS: TestNewCreate_ShardIDDerivedFromKey (0.00s) === RUN TestEventOption_WithSize --- PASS: TestEventOption_WithSize (0.00s) === RUN TestEventOption_WithSizeAppliesToOldEntryOnDelete --- PASS: TestEventOption_WithSizeAppliesToOldEntryOnDelete (0.00s) === RUN TestEventOption_WithModTimeOverridesDefault --- PASS: TestEventOption_WithModTimeOverridesDefault (0.00s) === RUN TestEventOption_WithTtlSec --- PASS: TestEventOption_WithTtlSec (0.00s) === RUN TestEventOption_WithVersionID --- PASS: TestEventOption_WithVersionID (0.00s) === RUN TestEventOption_WithExtendedKeyValue --- PASS: TestEventOption_WithExtendedKeyValue (0.00s) === RUN TestEventOption_WithChunks --- PASS: TestEventOption_WithChunks (0.00s) === RUN TestEventOption_WithShardIDOverrides --- PASS: TestEventOption_WithShardIDOverrides (0.00s) === RUN TestEventOption_WithOldSizeTargetsOldEntryOnUpdate --- PASS: TestEventOption_WithOldSizeTargetsOldEntryOnUpdate (0.00s) === RUN TestEventOption_WithOldChunksTargetsOldEntryOnUpdate --- PASS: TestEventOption_WithOldChunksTargetsOldEntryOnUpdate (0.00s) === RUN TestEventOption_WithOldModTimeTargetsOldEntryOnUpdate --- PASS: TestEventOption_WithOldModTimeTargetsOldEntryOnUpdate (0.00s) === RUN TestEventOption_WithOldOptionsAreNoOpsOnCreate --- PASS: TestEventOption_WithOldOptionsAreNoOpsOnCreate (0.00s) === RUN TestEventOption_LaterOverridesEarlier --- PASS: TestEventOption_LaterOverridesEarlier (0.00s) === RUN TestMetaLogClock_DefaultStepIsOneSecond --- PASS: TestMetaLogClock_DefaultStepIsOneSecond (0.03s) === RUN TestMetaLogClock_CustomStep --- PASS: TestMetaLogClock_CustomStep (0.00s) === RUN TestMetaLogClock_PeekDoesNotAdvance --- PASS: TestMetaLogClock_PeekDoesNotAdvance (0.00s) === RUN TestMetaLogClock_ConcurrentNextNoRace --- PASS: TestMetaLogClock_ConcurrentNextNoRace (0.00s) === RUN TestFake_DefaultIsDoneOutOfTheBox --- PASS: TestFake_DefaultIsDoneOutOfTheBox (0.04s) === RUN TestFake_QueuedOutcomesPopFIFO --- PASS: TestFake_QueuedOutcomesPopFIFO (0.00s) === RUN TestFake_QueuesIsolatedByKey --- PASS: TestFake_QueuesIsolatedByKey (0.03s) === RUN TestFake_VersionIDPartOfKey --- PASS: TestFake_VersionIDPartOfKey (0.00s) === RUN TestFake_KeyComponentsWithDelimitersDoNotCollide --- PASS: TestFake_KeyComponentsWithDelimitersDoNotCollide (0.00s) === RUN TestFake_ErrShortCircuitsBeforeRecording --- PASS: TestFake_ErrShortCircuitsBeforeRecording (0.00s) === RUN TestFake_RecordsRequestsInOrder --- PASS: TestFake_RecordsRequestsInOrder (0.00s) === RUN TestFake_RecordedIsSnapshot --- PASS: TestFake_RecordedIsSnapshot (0.00s) === RUN TestFake_RecordedRequestsAreDeepCopies --- PASS: TestFake_RecordedRequestsAreDeepCopies (0.00s) === RUN TestFake_NilRequestUsesDefault --- PASS: TestFake_NilRequestUsesDefault (0.00s) === RUN TestFake_ConcurrentCallsSerializeWithoutDeadlock --- PASS: TestFake_ConcurrentCallsSerializeWithoutDeadlock (0.01s) PASS ok github.com/seaweedfs/seaweedfs/weed/s3api/s3lifecycle/lifecycletest 17.782s === RUN TestCursorMinTsNsWithFrozenKeysIncluded --- PASS: TestCursorMinTsNsWithFrozenKeysIncluded (0.03s) === RUN TestCursorSnapshotIsDeepCopy --- PASS: TestCursorSnapshotIsDeepCopy (0.00s) === RUN TestCursorRestoreReplacesNotMerges --- PASS: TestCursorRestoreReplacesNotMerges (0.00s) === RUN TestReaderRunValidatesInputsBeforeSubscribing === RUN TestReaderRunValidatesInputsBeforeSubscribing/shard_id_below_zero === RUN TestReaderRunValidatesInputsBeforeSubscribing/shard_id_at_or_above_ShardCount === RUN TestReaderRunValidatesInputsBeforeSubscribing/nil_events_channel === RUN TestReaderRunValidatesInputsBeforeSubscribing/empty_buckets_path --- PASS: TestReaderRunValidatesInputsBeforeSubscribing (0.03s) --- PASS: TestReaderRunValidatesInputsBeforeSubscribing/shard_id_below_zero (0.03s) --- PASS: TestReaderRunValidatesInputsBeforeSubscribing/shard_id_at_or_above_ShardCount (0.00s) --- PASS: TestReaderRunValidatesInputsBeforeSubscribing/nil_events_channel (0.00s) --- PASS: TestReaderRunValidatesInputsBeforeSubscribing/empty_buckets_path (0.00s) === RUN TestCursorMinTsNsEmpty --- PASS: TestCursorMinTsNsEmpty (0.00s) === RUN TestCursorAdvanceMonotonic --- PASS: TestCursorAdvanceMonotonic (0.00s) === RUN TestCursorAdvanceIgnoresZero --- PASS: TestCursorAdvanceIgnoresZero (0.00s) === RUN TestCursorMinTsNsAcrossKeys --- PASS: TestCursorMinTsNsAcrossKeys (0.00s) === RUN TestCursorFreeze --- PASS: TestCursorFreeze (0.00s) === RUN TestCursorFreezeOnUnsetKeySeedsPosition --- PASS: TestCursorFreezeOnUnsetKeySeedsPosition (0.00s) === RUN TestCursorSnapshotRestore --- PASS: TestCursorSnapshotRestore (0.00s) === RUN TestEventIsCreate_PopulatedNewEntryNoOldEntry --- PASS: TestEventIsCreate_PopulatedNewEntryNoOldEntry (0.00s) === RUN TestEventIsDelete_PopulatedOldEntryNoNewEntry --- PASS: TestEventIsDelete_PopulatedOldEntryNoNewEntry (0.00s) === RUN TestEventBothEntries_NeitherCreateNorDelete --- PASS: TestEventBothEntries_NeitherCreateNorDelete (0.00s) === RUN TestEventNoEntries_NeitherCreateNorDelete --- PASS: TestEventNoEntries_NeitherCreateNorDelete (0.00s) === RUN TestLogStartup_ShardIDOnly --- PASS: TestLogStartup_ShardIDOnly (0.00s) === RUN TestLogStartup_ShardPredicate --- PASS: TestLogStartup_ShardPredicate (0.00s) === RUN TestLogStartup_StartTsNsOverridesCursor --- PASS: TestLogStartup_StartTsNsOverridesCursor (0.00s) === RUN TestLogStartup_CursorMinFallback --- PASS: TestLogStartup_CursorMinFallback (0.00s) === RUN TestInMemoryPersister_LoadOnUnknownShardReturnsEmptyMap --- PASS: TestInMemoryPersister_LoadOnUnknownShardReturnsEmptyMap (0.00s) === RUN TestInMemoryPersister_SaveLoadRoundTrip --- PASS: TestInMemoryPersister_SaveLoadRoundTrip (0.00s) === RUN TestInMemoryPersister_SaveCopiesInput --- PASS: TestInMemoryPersister_SaveCopiesInput (0.00s) === RUN TestInMemoryPersister_LoadReturnsACopy --- PASS: TestInMemoryPersister_LoadReturnsACopy (0.00s) === RUN TestInMemoryPersister_SaveReplacesNotMerges --- PASS: TestInMemoryPersister_SaveReplacesNotMerges (0.03s) === RUN TestInMemoryPersister_DifferentShardsIsolated --- PASS: TestInMemoryPersister_DifferentShardsIsolated (0.00s) === RUN TestInMemoryPersister_SaveEmptyMapClearsState --- PASS: TestInMemoryPersister_SaveEmptyMapClearsState (0.00s) === RUN TestInMemoryPersister_ConcurrentSaveLoadDoesNotRace --- PASS: TestInMemoryPersister_ConcurrentSaveLoadDoesNotRace (0.01s) === RUN TestExtractBucketKeyCreate --- PASS: TestExtractBucketKeyCreate (0.00s) === RUN TestExtractBucketKeyTopLevelObject --- PASS: TestExtractBucketKeyTopLevelObject (0.00s) === RUN TestExtractBucketKeyDeleteUsesOldEntry --- PASS: TestExtractBucketKeyDeleteUsesOldEntry (0.00s) === RUN TestExtractBucketKeyDeleteWithEmptyParent --- PASS: TestExtractBucketKeyDeleteWithEmptyParent (0.00s) === RUN TestExtractBucketKeyOutsideBucketsSkipped --- PASS: TestExtractBucketKeyOutsideBucketsSkipped (0.00s) === RUN TestExtractBucketKeyBucketCreateAtRoot --- PASS: TestExtractBucketKeyBucketCreateAtRoot (0.00s) === RUN TestExtractBucketKeyBucketCreateBareRoot --- PASS: TestExtractBucketKeyBucketCreateBareRoot (0.00s) === RUN TestDispatchOneFiltersByShard --- PASS: TestDispatchOneFiltersByShard (0.00s) === RUN TestDispatchOneRespectsCtxCancel --- PASS: TestDispatchOneRespectsCtxCancel (0.00s) === RUN TestRun_ReceiveTimeoutOnStalledStream --- PASS: TestRun_ReceiveTimeoutOnStalledStream (0.06s) === RUN TestRun_HeartbeatsHoldTheStreamOpen --- PASS: TestRun_HeartbeatsHoldTheStreamOpen (0.27s) === RUN TestRun_RejectsNegativeReceiveTimeout --- PASS: TestRun_RejectsNegativeReceiveTimeout (0.00s) PASS ok github.com/seaweedfs/seaweedfs/weed/s3api/s3lifecycle/reader 13.967s === RUN TestSuccessorModTimeFromContainer_MissingExtReturnsZero --- PASS: TestSuccessorModTimeFromContainer_MissingExtReturnsZero (0.03s) === RUN TestSuccessorModTimeFromContainer_EmptyValueReturnsZero --- PASS: TestSuccessorModTimeFromContainer_EmptyValueReturnsZero (0.00s) === RUN TestSuccessorModTimeFromContainer_NonNumericReturnsZero --- PASS: TestSuccessorModTimeFromContainer_NonNumericReturnsZero (0.00s) === RUN TestSuccessorModTimeFromContainer_NonPositiveReturnsZero === RUN TestSuccessorModTimeFromContainer_NonPositiveReturnsZero/0 === RUN TestSuccessorModTimeFromContainer_NonPositiveReturnsZero/-1 === RUN TestSuccessorModTimeFromContainer_NonPositiveReturnsZero/-1000 --- PASS: TestSuccessorModTimeFromContainer_NonPositiveReturnsZero (0.03s) --- PASS: TestSuccessorModTimeFromContainer_NonPositiveReturnsZero/0 (0.00s) --- PASS: TestSuccessorModTimeFromContainer_NonPositiveReturnsZero/-1 (0.00s) --- PASS: TestSuccessorModTimeFromContainer_NonPositiveReturnsZero/-1000 (0.00s) === RUN TestSuccessorModTimeFromContainer_PositiveSecondsRoundTrip --- PASS: TestSuccessorModTimeFromContainer_PositiveSecondsRoundTrip (0.00s) === RUN TestLogicalKeyFromVersionPath_ExtractsLogicalKey --- PASS: TestLogicalKeyFromVersionPath_ExtractsLogicalKey (0.00s) === RUN TestLogicalKeyFromVersionPath_RejectsPathWithoutVersionsParent --- PASS: TestLogicalKeyFromVersionPath_RejectsPathWithoutVersionsParent (0.00s) === RUN TestLogicalKeyFromVersionPath_RejectsRootLevelPath --- PASS: TestLogicalKeyFromVersionPath_RejectsRootLevelPath (0.00s) === RUN TestLogicalKeyFromVersionPath_RejectsNoSlashes --- PASS: TestLogicalKeyFromVersionPath_RejectsNoSlashes (0.00s) === RUN TestLogicalKeyFromVersionPath_RejectsBareVersionsContainer --- PASS: TestLogicalKeyFromVersionPath_RejectsBareVersionsContainer (0.00s) === RUN TestIsVersionsContainerKey === RUN TestIsVersionsContainerKey/obj.versions === RUN TestIsVersionsContainerKey/a/b/obj.versions === RUN TestIsVersionsContainerKey/.versions === RUN TestIsVersionsContainerKey/obj.versions/v_x === RUN TestIsVersionsContainerKey/obj === RUN TestIsVersionsContainerKey/#00 --- PASS: TestIsVersionsContainerKey (0.01s) --- PASS: TestIsVersionsContainerKey/obj.versions (0.00s) --- PASS: TestIsVersionsContainerKey/a/b/obj.versions (0.00s) --- PASS: TestIsVersionsContainerKey/.versions (0.00s) --- PASS: TestIsVersionsContainerKey/obj.versions/v_x (0.00s) --- PASS: TestIsVersionsContainerKey/obj (0.00s) --- PASS: TestIsVersionsContainerKey/#00 (0.00s) === RUN TestIsVersionFolderPath === RUN TestIsVersionFolderPath/obj.versions/v_aaa === RUN TestIsVersionFolderPath/a/b/obj.versions/v_aaa === RUN TestIsVersionFolderPath/obj/v_aaa === RUN TestIsVersionFolderPath/obj.versions === RUN TestIsVersionFolderPath/obj.versions/ === RUN TestIsVersionFolderPath/v_aaa === RUN TestIsVersionFolderPath/#00 --- PASS: TestIsVersionFolderPath (0.01s) --- PASS: TestIsVersionFolderPath/obj.versions/v_aaa (0.00s) --- PASS: TestIsVersionFolderPath/a/b/obj.versions/v_aaa (0.00s) --- PASS: TestIsVersionFolderPath/obj/v_aaa (0.00s) --- PASS: TestIsVersionFolderPath/obj.versions (0.00s) --- PASS: TestIsVersionFolderPath/obj.versions/ (0.00s) --- PASS: TestIsVersionFolderPath/v_aaa (0.00s) --- PASS: TestIsVersionFolderPath/#00 (0.00s) === RUN TestIsDeleteMarkerEntry_NilOrEmptyExtendedReturnsFalse --- PASS: TestIsDeleteMarkerEntry_NilOrEmptyExtendedReturnsFalse (0.00s) === RUN TestIsDeleteMarkerEntry_TrueOnlyForLiteralTrue --- PASS: TestIsDeleteMarkerEntry_TrueOnlyForLiteralTrue (0.00s) === RUN TestExtractTags_NilOrEmptyReturnsNil --- PASS: TestExtractTags_NilOrEmptyReturnsNil (0.00s) === RUN TestExtractTags_OnlyKeysWithObjectTaggingPrefix --- PASS: TestExtractTags_OnlyKeysWithObjectTaggingPrefix (0.00s) === RUN TestExtractTags_ReturnsNilWhenNoTaggingPrefixedKeys --- PASS: TestExtractTags_ReturnsNilWhenNoTaggingPrefixedKeys (0.00s) === RUN TestHasActiveEventDrivenAction --- PASS: TestHasActiveEventDrivenAction (0.00s) === RUN TestHasActiveEventDrivenAction_NilActionSkipped --- PASS: TestHasActiveEventDrivenAction_NilActionSkipped (0.03s) === RUN TestSnapshot_BucketVersionedReportsCompiledFlag --- PASS: TestSnapshot_BucketVersionedReportsCompiledFlag (0.03s) === RUN TestSnapshot_BucketActionKeysReturnsCompiledList --- PASS: TestSnapshot_BucketActionKeysReturnsCompiledList (0.00s) === RUN TestSnapshot_BucketActionKeysUnknownBucketReturnsNil --- PASS: TestSnapshot_BucketActionKeysUnknownBucketReturnsNil (0.00s) === RUN TestSnapshot_ActionUnknownKeyReturnsNil --- PASS: TestSnapshot_ActionUnknownKeyReturnsNil (0.00s) === RUN TestSnapshot_AllActionsCoversEveryCompiledKind --- PASS: TestSnapshot_AllActionsCoversEveryCompiledKind (0.00s) === RUN TestSnapshot_SnapshotIDIsMonotonicAcrossRecompiles --- PASS: TestSnapshot_SnapshotIDIsMonotonicAcrossRecompiles (0.00s) === RUN TestRouteNoSnapshotNoMatches --- PASS: TestRouteNoSnapshotNoMatches (0.00s) === RUN TestRouteMissingBucketNoMatches --- PASS: TestRouteMissingBucketNoMatches (0.00s) === RUN TestRouteInactiveSkipped --- PASS: TestRouteInactiveSkipped (0.00s) === RUN TestRouteExpirationDaysFires --- PASS: TestRouteExpirationDaysFires (0.00s) === RUN TestRouteFreshObjectSchedulesInFuture --- PASS: TestRouteFreshObjectSchedulesInFuture (0.00s) === RUN TestRouteRespectsPrefixFilter --- PASS: TestRouteRespectsPrefixFilter (0.00s) === RUN TestRouteSkipsHardDelete --- PASS: TestRouteSkipsHardDelete (0.00s) === RUN TestRouteSkipsMissingAttributes --- PASS: TestRouteSkipsMissingAttributes (0.00s) === RUN TestRouteIdentityCapturedForNewEntry --- PASS: TestRouteIdentityCapturedForNewEntry (0.00s) === RUN TestRouteIdentityHashesExtended --- PASS: TestRouteIdentityHashesExtended (0.00s) === RUN TestRouteMPUInitFiresAbortAfterDelay --- PASS: TestRouteMPUInitFiresAbortAfterDelay (0.00s) === RUN TestRouteMPUInitFilteredOutByPrefix --- PASS: TestRouteMPUInitFilteredOutByPrefix (0.00s) === RUN TestRouteMPUInitMissingDestKeySkipped --- PASS: TestRouteMPUInitMissingDestKeySkipped (0.00s) === RUN TestRouteMPUPartEventSkipped --- PASS: TestRouteMPUPartEventSkipped (0.00s) === RUN TestRouteMPUInitDoesNotFireNoncurrent --- PASS: TestRouteMPUInitDoesNotFireNoncurrent (0.00s) === RUN TestRouteRegularObjectUnderDualRuleSkipsAbortMPU --- PASS: TestRouteRegularObjectUnderDualRuleSkipsAbortMPU (0.00s) === RUN TestRouteVersionedNoncurrentEventDoesNotFireFromRouter --- PASS: TestRouteVersionedNoncurrentEventDoesNotFireFromRouter (0.00s) === RUN TestRouteVersionedCurrentEventStaysLatest --- PASS: TestRouteVersionedCurrentEventStaysLatest (0.00s) === RUN TestRouteNonVersionedBucketIgnoresVersionsSuffix --- PASS: TestRouteNonVersionedBucketIgnoresVersionsSuffix (0.00s) === RUN TestRouteVersionedExpiredDeleteMarkerNilListerSuppresses --- PASS: TestRouteVersionedExpiredDeleteMarkerNilListerSuppresses (0.00s) === RUN TestRouteVersionedExpiredDeleteMarkerSoleSurvivorFires --- PASS: TestRouteVersionedExpiredDeleteMarkerSoleSurvivorFires (0.00s) === RUN TestRouteVersionedExpiredDeleteMarkerSiblingsRemainSuppressed --- PASS: TestRouteVersionedExpiredDeleteMarkerSiblingsRemainSuppressed (0.00s) === RUN TestRouteVersionedExpiredDeleteMarkerNullVersionSuppresses --- PASS: TestRouteVersionedExpiredDeleteMarkerNullVersionSuppresses (0.00s) === RUN TestRouteVersionedExpiredDeleteMarkerHardDeleteLeavesLoneMarkerFires --- PASS: TestRouteVersionedExpiredDeleteMarkerHardDeleteLeavesLoneMarkerFires (0.00s) === RUN TestRouteVersionedExpiredDeleteMarkerHardDeleteLoneNonMarkerNoFire --- PASS: TestRouteVersionedExpiredDeleteMarkerHardDeleteLoneNonMarkerNoFire (0.00s) === RUN TestRouteVersionedExpiredDeleteMarkerListerErrorSuppressed --- PASS: TestRouteVersionedExpiredDeleteMarkerListerErrorSuppressed (0.00s) === RUN TestRouteVersionedExpiredDeleteMarkerInactiveActionSkipsListing --- PASS: TestRouteVersionedExpiredDeleteMarkerInactiveActionSkipsListing (0.00s) === RUN TestRouteVersionedRegularVersionCreateSkipsListing --- PASS: TestRouteVersionedRegularVersionCreateSkipsListing (0.00s) === RUN TestRouteVersionedDeleteMarkerNoExpDMRuleSkipsListing --- PASS: TestRouteVersionedDeleteMarkerNoExpDMRuleSkipsListing (0.00s) === RUN TestRouteVersionedAllVersionFolderPathsSkipped === RUN TestRouteVersionedAllVersionFolderPathsSkipped/tracked_version_file === RUN TestRouteVersionedAllVersionFolderPathsSkipped/literal-key_collision === RUN TestRouteVersionedAllVersionFolderPathsSkipped/bucket-root_.versions === RUN TestRouteVersionedAllVersionFolderPathsSkipped/versions_dir_itself --- PASS: TestRouteVersionedAllVersionFolderPathsSkipped (0.00s) --- PASS: TestRouteVersionedAllVersionFolderPathsSkipped/tracked_version_file (0.00s) --- PASS: TestRouteVersionedAllVersionFolderPathsSkipped/literal-key_collision (0.00s) --- PASS: TestRouteVersionedAllVersionFolderPathsSkipped/bucket-root_.versions (0.00s) --- PASS: TestRouteVersionedAllVersionFolderPathsSkipped/versions_dir_itself (0.00s) === RUN TestRouteBootstrapVersionLatestExpirationDaysFires --- PASS: TestRouteBootstrapVersionLatestExpirationDaysFires (0.00s) === RUN TestRouteBootstrapVersionNoncurrentDaysFires --- PASS: TestRouteBootstrapVersionNoncurrentDaysFires (0.00s) === RUN TestRouteBootstrapVersionNoncurrentRespectsNewerNoncurrentVersions --- PASS: TestRouteBootstrapVersionNoncurrentRespectsNewerNoncurrentVersions (0.00s) === RUN TestRouteBootstrapVersionAbortMPUNeverEmittedForVersion --- PASS: TestRouteBootstrapVersionAbortMPUNeverEmittedForVersion (0.00s) === RUN TestRoutePointerTransitionFiresNoncurrentDays --- PASS: TestRoutePointerTransitionFiresNoncurrentDays (0.00s) === RUN TestRoutePointerTransitionPointerUnchangedSkipped --- PASS: TestRoutePointerTransitionPointerUnchangedSkipped (0.00s) === RUN TestRoutePointerTransitionEmptyOldPointerNoNullSkipped --- PASS: TestRoutePointerTransitionEmptyOldPointerNoNullSkipped (0.00s) === RUN TestRoutePointerTransitionEmptyOldPointerWithNullSchedules --- PASS: TestRoutePointerTransitionEmptyOldPointerWithNullSchedules (0.00s) === RUN TestRoutePointerTransitionExpansionIncludesNullVersion --- PASS: TestRoutePointerTransitionExpansionIncludesNullVersion (0.00s) === RUN TestRoutePointerTransitionDisplacedVersionMissingSuppressed --- PASS: TestRoutePointerTransitionDisplacedVersionMissingSuppressed (0.00s) === RUN TestRoutePointerTransitionNoNoncurrentRuleSkipsLookup --- PASS: TestRoutePointerTransitionNoNoncurrentRuleSkipsLookup (0.00s) === RUN TestRoutePointerTransitionNewerNoncurrentNewestNoncurrentRetained --- PASS: TestRoutePointerTransitionNewerNoncurrentNewestNoncurrentRetained (0.00s) === RUN TestRoutePointerTransitionExpansionMissingNewIDSuppressed --- PASS: TestRoutePointerTransitionExpansionMissingNewIDSuppressed (0.00s) === RUN TestRoutePointerTransitionUnversionedBucketSkipped --- PASS: TestRoutePointerTransitionUnversionedBucketSkipped (0.00s) === RUN TestRoutePointerTransitionMissingCachedMtimeSuppressed --- PASS: TestRoutePointerTransitionMissingCachedMtimeSuppressed (0.00s) === RUN TestRoutePointerTransitionUsesCachedMtimeNotStaleDirMtime --- PASS: TestRoutePointerTransitionUsesCachedMtimeNotStaleDirMtime (0.00s) === RUN TestRoutePointerTransitionNewerNoncurrentExpansionFiresOnCrossingThreshold --- PASS: TestRoutePointerTransitionNewerNoncurrentExpansionFiresOnCrossingThreshold (0.00s) === RUN TestRoutePointerTransitionExpansionEmitsOnlyThresholdCrossing --- PASS: TestRoutePointerTransitionExpansionEmitsOnlyThresholdCrossing (0.00s) === RUN TestRoutePointerTransitionSuspendedClearsPointerUsesNullMtime --- PASS: TestRoutePointerTransitionSuspendedClearsPointerUsesNullMtime (0.00s) === RUN TestRoutePointerTransitionSuspendedClearsPointerExpansionLatestPosIsNull --- PASS: TestRoutePointerTransitionSuspendedClearsPointerExpansionLatestPosIsNull (0.00s) === RUN TestScheduleEmpty --- PASS: TestScheduleEmpty (0.00s) === RUN TestScheduleOrderedByDueTime --- PASS: TestScheduleOrderedByDueTime (0.00s) === RUN TestScheduleDrainBoundaryInclusive --- PASS: TestScheduleDrainBoundaryInclusive (0.00s) === RUN TestScheduleAllowsDuplicates --- PASS: TestScheduleAllowsDuplicates (0.00s) === RUN TestScheduleDrainBeforeAnyDueReturnsNothing --- PASS: TestScheduleDrainBeforeAnyDueReturnsNothing (0.00s) === RUN TestScheduleNextDueAfterPartialDrain --- PASS: TestScheduleNextDueAfterPartialDrain (0.00s) === RUN TestScheduleAddAfterDrainKeepsOrder --- PASS: TestScheduleAddAfterDrainKeepsOrder (0.00s) === RUN TestScheduleDrainOrderIsAscendingDueTime --- PASS: TestScheduleDrainOrderIsAscendingDueTime (0.00s) === RUN TestScheduleConcurrentAddDrainNoRace --- PASS: TestScheduleConcurrentAddDrainNoRace (0.01s) PASS ok github.com/seaweedfs/seaweedfs/weed/s3api/s3lifecycle/router 15.188s === RUN TestIsBucketVersioned_MissingExtendedReturnsFalse --- PASS: TestIsBucketVersioned_MissingExtendedReturnsFalse (0.00s) === RUN TestIsBucketVersioned_AcceptsEnabledAndSuspended === RUN TestIsBucketVersioned_AcceptsEnabledAndSuspended/Enabled === RUN TestIsBucketVersioned_AcceptsEnabledAndSuspended/ENABLED === RUN TestIsBucketVersioned_AcceptsEnabledAndSuspended/enabled === RUN TestIsBucketVersioned_AcceptsEnabledAndSuspended/__enabled__ === RUN TestIsBucketVersioned_AcceptsEnabledAndSuspended/Suspended === RUN TestIsBucketVersioned_AcceptsEnabledAndSuspended/suspended --- PASS: TestIsBucketVersioned_AcceptsEnabledAndSuspended (0.03s) --- PASS: TestIsBucketVersioned_AcceptsEnabledAndSuspended/Enabled (0.00s) --- PASS: TestIsBucketVersioned_AcceptsEnabledAndSuspended/ENABLED (0.00s) --- PASS: TestIsBucketVersioned_AcceptsEnabledAndSuspended/enabled (0.00s) --- PASS: TestIsBucketVersioned_AcceptsEnabledAndSuspended/__enabled__ (0.00s) --- PASS: TestIsBucketVersioned_AcceptsEnabledAndSuspended/Suspended (0.00s) --- PASS: TestIsBucketVersioned_AcceptsEnabledAndSuspended/suspended (0.00s) === RUN TestIsBucketVersioned_RejectsOtherValues === RUN TestIsBucketVersioned_RejectsOtherValues/#00 === RUN TestIsBucketVersioned_RejectsOtherValues/Disabled === RUN TestIsBucketVersioned_RejectsOtherValues/Off === RUN TestIsBucketVersioned_RejectsOtherValues/true === RUN TestIsBucketVersioned_RejectsOtherValues/1 === RUN TestIsBucketVersioned_RejectsOtherValues/garbage --- PASS: TestIsBucketVersioned_RejectsOtherValues (0.00s) --- PASS: TestIsBucketVersioned_RejectsOtherValues/#00 (0.00s) --- PASS: TestIsBucketVersioned_RejectsOtherValues/Disabled (0.00s) --- PASS: TestIsBucketVersioned_RejectsOtherValues/Off (0.00s) --- PASS: TestIsBucketVersioned_RejectsOtherValues/true (0.00s) --- PASS: TestIsBucketVersioned_RejectsOtherValues/1 (0.00s) --- PASS: TestIsBucketVersioned_RejectsOtherValues/garbage (0.00s) === RUN TestAllActivePriorStates_EmptyInputs --- PASS: TestAllActivePriorStates_EmptyInputs (0.00s) === RUN TestAllActivePriorStates_SeedsAllActionsAsActive --- PASS: TestAllActivePriorStates_SeedsAllActionsAsActive (0.03s) === RUN TestAllActivePriorStates_KeysSeparatedByBucket --- PASS: TestAllActivePriorStates_KeysSeparatedByBucket (0.00s) === RUN TestLoadCompileInputs_EmptyBucketDir --- PASS: TestLoadCompileInputs_EmptyBucketDir (0.06s) === RUN TestLoadCompileInputs_FilesAtBucketLevelSkipped --- PASS: TestLoadCompileInputs_FilesAtBucketLevelSkipped (0.00s) === RUN TestLoadCompileInputs_BucketWithoutLifecycleConfigSkipped --- PASS: TestLoadCompileInputs_BucketWithoutLifecycleConfigSkipped (0.00s) === RUN TestLoadCompileInputs_BucketWithEmptyLifecycleXMLSkipped --- PASS: TestLoadCompileInputs_BucketWithEmptyLifecycleXMLSkipped (0.00s) === RUN TestLoadCompileInputs_ValidConfigBecomesInput --- PASS: TestLoadCompileInputs_ValidConfigBecomesInput (0.03s) === RUN TestLoadCompileInputs_VersioningAttrPropagates --- PASS: TestLoadCompileInputs_VersioningAttrPropagates (0.00s) === RUN TestLoadCompileInputs_MalformedXMLBecomesParseError --- PASS: TestLoadCompileInputs_MalformedXMLBecomesParseError (0.00s) === RUN TestLoadCompileInputs_PaginatesAcrossPages --- PASS: TestLoadCompileInputs_PaginatesAcrossPages (0.39s) PASS ok github.com/seaweedfs/seaweedfs/weed/s3api/s3lifecycle/scheduler 14.705s === RUN TestCreateTableRefusesAForeignFormat E0824 22:18:25.809799 handler.go:218 S3Tables: error handling CreateTable: table bucket formats holds LANCE tables --- PASS: TestCreateTableRefusesAForeignFormat (0.62s) === RUN TestCreateTableRefusesLanceInAnIcebergBucket E0824 22:18:25.895801 handler.go:218 S3Tables: error handling CreateTable: table bucket formats holds ICEBERG tables --- PASS: TestCreateTableRefusesLanceInAnIcebergBucket (0.06s) === RUN TestUndeclaredBucketAcceptsEitherFormat --- PASS: TestUndeclaredBucketAcceptsEitherFormat (0.25s) === RUN TestCreateViewRefusedInALanceBucket E0824 22:18:26.227933 handler.go:218 S3Tables: error handling CreateView: table bucket formats holds LANCE tables and cannot hold views --- PASS: TestCreateViewRefusedInALanceBucket (0.07s) === RUN TestNormalizeFormat --- PASS: TestNormalizeFormat (0.00s) === RUN TestDeleteTableDecoupledKeepsReusedNamePath --- PASS: TestDeleteTableDecoupledKeepsReusedNamePath (0.08s) === RUN TestDeleteTableRefusesAncestorDataPath E0824 22:18:26.350178 handler.go:218 S3Tables: error handling DeleteTable: refusing to delete table badt: data path "/buckets/renamebkt/ns" is an ancestor of catalog path "/buckets/renamebkt/ns/badt" --- PASS: TestDeleteTableRefusesAncestorDataPath (0.04s) === RUN TestDeleteTableColocatedRemovesData --- PASS: TestDeleteTableColocatedRemovesData (0.05s) === RUN TestGetAccountIDPrefersClaimsOverAccountID --- PASS: TestGetAccountIDPrefersClaimsOverAccountID (0.03s) === RUN TestGetAccountIDUsesSubWhenPreferredUsernameMissing --- PASS: TestGetAccountIDUsesSubWhenPreferredUsernameMissing (0.00s) === RUN TestGetAccountIDFallsBackToHandlerDefaultAccount --- PASS: TestGetAccountIDFallsBackToHandlerDefaultAccount (0.00s) === RUN TestGetAccountIDIgnoresEmptyClaimValues --- PASS: TestGetAccountIDIgnoresEmptyClaimValues (0.00s) === RUN TestGetAccountIDFallsBackToIdentityName --- PASS: TestGetAccountIDFallsBackToIdentityName (0.00s) === RUN TestGetAccountIDFallsBackToARNColonSegment --- PASS: TestGetAccountIDFallsBackToARNColonSegment (0.00s) === RUN TestGetAccountIDFallsBackToAmzAccountIdHeader --- PASS: TestGetAccountIDFallsBackToAmzAccountIdHeader (0.00s) === RUN TestGetAccountIDFallsBackToAccountID --- PASS: TestGetAccountIDFallsBackToAccountID (0.00s) === RUN TestGetAccountIDNonAdminDoesNotInheritAdminAccount --- PASS: TestGetAccountIDNonAdminDoesNotInheritAdminAccount (0.00s) === RUN TestGetAccountIDAdminActionKeepsAdminAccount --- PASS: TestGetAccountIDAdminActionKeepsAdminAccount (0.00s) === RUN TestDefaultAllowForOnlyAppliesToUnauthenticatedOrAnonymous --- PASS: TestDefaultAllowForOnlyAppliesToUnauthenticatedOrAnonymous (0.00s) === RUN TestGetAccountIDNormalizesAccountIDARN --- PASS: TestGetAccountIDNormalizesAccountIDARN (0.00s) === RUN TestValidateMaintenanceValueAcceptsScopedTypes --- PASS: TestValidateMaintenanceValueAcceptsScopedTypes (0.00s) === RUN TestValidateMaintenanceValueRejectsWrongScope --- PASS: TestValidateMaintenanceValueRejectsWrongScope (0.00s) === RUN TestValidateMaintenanceValueRejectsBadInput === RUN TestValidateMaintenanceValueRejectsBadInput/empty_type === RUN TestValidateMaintenanceValueRejectsBadInput/unknown_type === RUN TestValidateMaintenanceValueRejectsBadInput/no_status === RUN TestValidateMaintenanceValueRejectsBadInput/bad_status === RUN TestValidateMaintenanceValueRejectsBadInput/settings_for_another_type === RUN TestValidateMaintenanceValueRejectsBadInput/nil_value --- PASS: TestValidateMaintenanceValueRejectsBadInput (0.00s) --- PASS: TestValidateMaintenanceValueRejectsBadInput/empty_type (0.00s) --- PASS: TestValidateMaintenanceValueRejectsBadInput/unknown_type (0.00s) --- PASS: TestValidateMaintenanceValueRejectsBadInput/no_status (0.00s) --- PASS: TestValidateMaintenanceValueRejectsBadInput/bad_status (0.00s) --- PASS: TestValidateMaintenanceValueRejectsBadInput/settings_for_another_type (0.00s) --- PASS: TestValidateMaintenanceValueRejectsBadInput/nil_value (0.00s) === RUN TestMaintenanceConfigurationRoundTrip --- PASS: TestMaintenanceConfigurationRoundTrip (0.00s) === RUN TestMaintenanceConfigurationMergeKeepsOtherTypes --- PASS: TestMaintenanceConfigurationMergeKeepsOtherTypes (0.00s) === RUN TestParseTableTarget === RUN TestParseTableTarget/missing_arn === RUN TestParseTableTarget/missing_namespace === RUN TestParseTableTarget/missing_name === RUN TestParseTableTarget/bad_arn --- PASS: TestParseTableTarget (0.00s) --- PASS: TestParseTableTarget/missing_arn (0.00s) --- PASS: TestParseTableTarget/missing_namespace (0.00s) --- PASS: TestParseTableTarget/missing_name (0.00s) --- PASS: TestParseTableTarget/bad_arn (0.00s) === RUN TestBuildJobStatusResponseReportsEveryType --- PASS: TestBuildJobStatusResponseReportsEveryType (0.00s) === RUN TestBuildJobStatusResponseDisabledBeatsRecorded --- PASS: TestBuildJobStatusResponseDisabledBeatsRecorded (0.00s) === RUN TestValidateMaintenanceValueCompactionStrategy --- PASS: TestValidateMaintenanceValueCompactionStrategy (0.00s) === RUN TestMaintenanceConfigurationPreservesStrategy --- PASS: TestMaintenanceConfigurationPreservesStrategy (0.00s) === RUN TestBuildJobStatusResponseHonoursBucketDisable --- PASS: TestBuildJobStatusResponseHonoursBucketDisable (0.00s) === RUN TestMergeMaintenanceConfiguration --- PASS: TestMergeMaintenanceConfiguration (0.00s) === RUN TestSnapshotExtendedCoversEveryAttribute --- PASS: TestSnapshotExtendedCoversEveryAttribute (0.00s) === RUN TestSnapshotExtendedAssertsAbsentMaintenanceKey --- PASS: TestSnapshotExtendedAssertsAbsentMaintenanceKey (0.00s) === RUN TestValidateMaintenanceSettingRange --- PASS: TestValidateMaintenanceSettingRange (0.00s) === RUN TestValidateMaintenanceSettingRangeCoversEveryField === RUN TestValidateMaintenanceSettingRangeCoversEveryField/minSnapshotsToKeep === RUN TestValidateMaintenanceSettingRangeCoversEveryField/maxSnapshotAgeHours === RUN TestValidateMaintenanceSettingRangeCoversEveryField/unreferencedDays === RUN TestValidateMaintenanceSettingRangeCoversEveryField/nonCurrentDays --- PASS: TestValidateMaintenanceSettingRangeCoversEveryField (0.00s) --- PASS: TestValidateMaintenanceSettingRangeCoversEveryField/minSnapshotsToKeep (0.00s) --- PASS: TestValidateMaintenanceSettingRangeCoversEveryField/maxSnapshotAgeHours (0.00s) --- PASS: TestValidateMaintenanceSettingRangeCoversEveryField/unreferencedDays (0.00s) --- PASS: TestValidateMaintenanceSettingRangeCoversEveryField/nonCurrentDays (0.00s) === RUN TestMaintenanceSettingsDistinguishZeroFromUnset --- PASS: TestMaintenanceSettingsDistinguishZeroFromUnset (0.00s) === RUN TestARNPartitionsBeyondCommercial --- PASS: TestARNPartitionsBeyondCommercial (0.00s) === RUN TestARNPartitionForRegion --- PASS: TestARNPartitionForRegion (0.00s) === RUN TestGeneratedARNsUseTheRegionPartition --- PASS: TestGeneratedARNsUseTheRegionPartition (0.00s) === RUN TestNameValidationAllowsHyphens --- PASS: TestNameValidationAllowsHyphens (0.00s) === RUN TestParseTableFromARNAllowsHyphens --- PASS: TestParseTableFromARNAllowsHyphens (0.00s) === RUN TestLoadNamespaceMetadataTreatsMissingAttributeAsAbsent --- PASS: TestLoadNamespaceMetadataTreatsMissingAttributeAsAbsent (0.04s) === RUN TestRenameTablePreservesData E0824 22:18:26.640313 handler.go:218 S3Tables: error handling GetTable: attribute not found: s3tables.metadata --- PASS: TestRenameTablePreservesData (0.19s) === RUN TestRenameTableSourceMissing E0824 22:18:26.728972 handler.go:218 S3Tables: error handling RenameTable: filer: no entry is found in filer store --- PASS: TestRenameTableSourceMissing (0.05s) === RUN TestRenameTableDestExists E0824 22:18:26.812246 handler.go:218 S3Tables: error handling RenameTable: table already exists --- PASS: TestRenameTableDestExists (0.09s) === RUN TestRenameTableDestNamespaceMissing E0824 22:18:26.904791 handler.go:218 S3Tables: error handling RenameTable: filer: no entry is found in filer store --- PASS: TestRenameTableDestNamespaceMissing (0.08s) === RUN TestRenameTableDestNamespaceUnauthorized E0824 22:18:26.998878 handler.go:218 S3Tables: error handling RenameTable: unauthorized: mover is not permitted to perform RenameTable: not authorized to create table in the destination namespace --- PASS: TestRenameTableDestNamespaceUnauthorized (0.10s) === RUN TestRenameTableInvalidName E0824 22:18:27.026050 handler.go:218 S3Tables: error handling RenameTable: invalid table name: cannot be '.', '..' or contain '/' --- PASS: TestRenameTableInvalidName (0.02s) === RUN TestRenameTableCarriesMaintenanceConfiguration --- PASS: TestRenameTableCarriesMaintenanceConfiguration (0.25s) === RUN TestRenameTableRejectsConcurrentMaintenanceWrite --- PASS: TestRenameTableRejectsConcurrentMaintenanceWrite (0.03s) === RUN TestMetadataVersionFromLocation --- PASS: TestMetadataVersionFromLocation (0.00s) === RUN TestUpdateTableRejectsLostUpdate E0824 22:18:27.383319 handler.go:218 S3Tables: error handling UpdateTable: version token mismatch --- PASS: TestUpdateTableRejectsLostUpdate (0.11s) === RUN TestUpdateTableRejectsWhenTheAuthorizingPolicyChanges E0824 22:18:27.508761 handler.go:218 S3Tables: error handling UpdateTable: version token mismatch --- PASS: TestUpdateTableRejectsWhenTheAuthorizingPolicyChanges (0.09s) === RUN TestUpdateTableAppliesWithoutContention --- PASS: TestUpdateTableAppliesWithoutContention (0.14s) === RUN TestRenameViewMovesCatalogPointer --- PASS: TestRenameViewMovesCatalogPointer (0.33s) === RUN TestRenameViewRejectsTable E0824 22:18:28.109079 handler.go:218 S3Tables: error handling RenameView: view t not found --- PASS: TestRenameViewRejectsTable (0.14s) === RUN TestRenameTableRejectsView E0824 22:18:28.224162 handler.go:218 S3Tables: error handling RenameTable: table v not found --- PASS: TestRenameTableRejectsView (0.10s) === RUN TestRenameViewAuthorizesAgainstTheViewARN --- PASS: TestRenameViewAuthorizesAgainstTheViewARN (0.15s) === RUN TestEntryType === RUN TestEntryType/nil_extended_is_table === RUN TestEntryType/absent_marker_is_table === RUN TestEntryType/empty_marker_is_table === RUN TestEntryType/table_marker === RUN TestEntryType/view_marker --- PASS: TestEntryType (0.00s) --- PASS: TestEntryType/nil_extended_is_table (0.00s) --- PASS: TestEntryType/absent_marker_is_table (0.00s) --- PASS: TestEntryType/empty_marker_is_table (0.00s) --- PASS: TestEntryType/table_marker (0.00s) --- PASS: TestEntryType/view_marker (0.00s) === RUN TestGenerateViewARN --- PASS: TestGenerateViewARN (0.00s) === RUN TestIcebergLayoutValidator_AcceptsRealWorldManifestNames === RUN TestIcebergLayoutValidator_AcceptsRealWorldManifestNames/flink-style_manifest_(job-id_+_checkpoint_+_operator-id_+_counter) === RUN TestIcebergLayoutValidator_AcceptsRealWorldManifestNames/spark-style_manifest_with_two_uuids_and_-m_suffix === RUN TestIcebergLayoutValidator_AcceptsRealWorldManifestNames/snapshot_manifest_list_with_iceberg-internal_naming === RUN TestIcebergLayoutValidator_AcceptsRealWorldManifestNames/versioned_table_metadata === RUN TestIcebergLayoutValidator_AcceptsRealWorldManifestNames/uuid-named_metadata_json_(newer_iceberg) === RUN TestIcebergLayoutValidator_AcceptsRealWorldManifestNames/flink-style_metadata_json_with_dashes_and_digits === RUN TestIcebergLayoutValidator_AcceptsRealWorldManifestNames/version_hint === RUN TestIcebergLayoutValidator_AcceptsRealWorldManifestNames/trino/iceberg_stats_file --- PASS: TestIcebergLayoutValidator_AcceptsRealWorldManifestNames (0.01s) --- PASS: TestIcebergLayoutValidator_AcceptsRealWorldManifestNames/flink-style_manifest_(job-id_+_checkpoint_+_operator-id_+_counter) (0.00s) --- PASS: TestIcebergLayoutValidator_AcceptsRealWorldManifestNames/spark-style_manifest_with_two_uuids_and_-m_suffix (0.00s) --- PASS: TestIcebergLayoutValidator_AcceptsRealWorldManifestNames/snapshot_manifest_list_with_iceberg-internal_naming (0.00s) --- PASS: TestIcebergLayoutValidator_AcceptsRealWorldManifestNames/versioned_table_metadata (0.00s) --- PASS: TestIcebergLayoutValidator_AcceptsRealWorldManifestNames/uuid-named_metadata_json_(newer_iceberg) (0.00s) --- PASS: TestIcebergLayoutValidator_AcceptsRealWorldManifestNames/flink-style_metadata_json_with_dashes_and_digits (0.00s) --- PASS: TestIcebergLayoutValidator_AcceptsRealWorldManifestNames/version_hint (0.00s) --- PASS: TestIcebergLayoutValidator_AcceptsRealWorldManifestNames/trino/iceberg_stats_file (0.00s) === RUN TestIcebergLayoutValidator_RejectsClearlyBadMetadataNames === RUN TestIcebergLayoutValidator_RejectsClearlyBadMetadataNames/random_extension === RUN TestIcebergLayoutValidator_RejectsClearlyBadMetadataNames/executable_masquerading_as_avro_path === RUN TestIcebergLayoutValidator_RejectsClearlyBadMetadataNames/subdirectory_under_metadata_is_not_allowed === RUN TestIcebergLayoutValidator_RejectsClearlyBadMetadataNames/top-level_dir_other_than_metadata_or_data === RUN TestIcebergLayoutValidator_RejectsClearlyBadMetadataNames/catch-all_anchors_must_reject_trailing_extension --- PASS: TestIcebergLayoutValidator_RejectsClearlyBadMetadataNames (0.00s) --- PASS: TestIcebergLayoutValidator_RejectsClearlyBadMetadataNames/random_extension (0.00s) --- PASS: TestIcebergLayoutValidator_RejectsClearlyBadMetadataNames/executable_masquerading_as_avro_path (0.00s) --- PASS: TestIcebergLayoutValidator_RejectsClearlyBadMetadataNames/subdirectory_under_metadata_is_not_allowed (0.00s) --- PASS: TestIcebergLayoutValidator_RejectsClearlyBadMetadataNames/top-level_dir_other_than_metadata_or_data (0.00s) --- PASS: TestIcebergLayoutValidator_RejectsClearlyBadMetadataNames/catch-all_anchors_must_reject_trailing_extension (0.00s) === RUN TestIcebergLayoutValidator_AcceptsRealWorldDataFiles === RUN TestIcebergLayoutValidator_AcceptsRealWorldDataFiles/data/00000-0-ede83b82-08e1-40cd-af8a-6d83680a5194-00001.parquet === RUN TestIcebergLayoutValidator_AcceptsRealWorldDataFiles/data/part-00000.parquet === RUN TestIcebergLayoutValidator_AcceptsRealWorldDataFiles/data/some-file.orc === RUN TestIcebergLayoutValidator_AcceptsRealWorldDataFiles/data/year=2026/month=05/00000-0-uuid.parquet --- PASS: TestIcebergLayoutValidator_AcceptsRealWorldDataFiles (0.00s) --- PASS: TestIcebergLayoutValidator_AcceptsRealWorldDataFiles/data/00000-0-ede83b82-08e1-40cd-af8a-6d83680a5194-00001.parquet (0.00s) --- PASS: TestIcebergLayoutValidator_AcceptsRealWorldDataFiles/data/part-00000.parquet (0.00s) --- PASS: TestIcebergLayoutValidator_AcceptsRealWorldDataFiles/data/some-file.orc (0.00s) --- PASS: TestIcebergLayoutValidator_AcceptsRealWorldDataFiles/data/year=2026/month=05/00000-0-uuid.parquet (0.00s) === RUN TestValidateFilePathAcceptsLanceLayout --- PASS: TestValidateFilePathAcceptsLanceLayout (0.00s) === RUN TestReadManifestListWithoutFormatVersion --- PASS: TestReadManifestListWithoutFormatVersion (0.05s) === RUN TestReadManifestListKeepsWrittenFormatVersion --- PASS: TestReadManifestListKeepsWrittenFormatVersion (0.06s) === RUN TestReadManifestListWithoutFormatVersionStaysV1 --- PASS: TestReadManifestListWithoutFormatVersionStaysV1 (0.01s) === RUN TestReadManifestListRejectsNonAvro --- PASS: TestReadManifestListRejectsNonAvro (0.00s) === RUN TestManifestListFormatVersion === RUN TestManifestListFormatVersion/v1_fields === RUN TestManifestListFormatVersion/v2_adds_content === RUN TestManifestListFormatVersion/v2_adds_sequence_numbers === RUN TestManifestListFormatVersion/v3_adds_first_row_id === RUN TestManifestListFormatVersion/v3_field_ordering === RUN TestManifestListFormatVersion/unparseable_schema === RUN TestManifestListFormatVersion/missing_schema --- PASS: TestManifestListFormatVersion (0.00s) --- PASS: TestManifestListFormatVersion/v1_fields (0.00s) --- PASS: TestManifestListFormatVersion/v2_adds_content (0.00s) --- PASS: TestManifestListFormatVersion/v2_adds_sequence_numbers (0.00s) --- PASS: TestManifestListFormatVersion/v3_adds_first_row_id (0.00s) --- PASS: TestManifestListFormatVersion/v3_field_ordering (0.00s) --- PASS: TestManifestListFormatVersion/unparseable_schema (0.00s) --- PASS: TestManifestListFormatVersion/missing_schema (0.00s) === RUN TestManagerCreateTableBucketAuthorization === RUN TestManagerCreateTableBucketAuthorization/authenticated_identity_struct_is_enforced E0824 22:18:28.530130 handler.go:218 S3Tables: error handling CreateTableBucket: unauthorized: alice is not permitted to perform CreateTableBucket: not authorized to create table buckets === RUN TestManagerCreateTableBucketAuthorization/secured_manager_denies_a_name_without_struct E0824 22:18:28.531176 handler.go:218 S3Tables: error handling CreateTableBucket: unauthorized: alice is not permitted to perform CreateTableBucket: not authorized to create table buckets === RUN TestManagerCreateTableBucketAuthorization/untrusted_name_without_struct_is_enforced E0824 22:18:28.531835 handler.go:218 S3Tables: error handling CreateTableBucket: unauthorized: alice is not permitted to perform CreateTableBucket: not authorized to create table buckets === RUN TestManagerCreateTableBucketAuthorization/trusted_manager_allows_a_name_without_struct E0824 22:18:28.532412 handler_bucket_create.go:111 S3Tables: failed to check bucket existence: filer reached E0824 22:18:28.532665 handler.go:218 S3Tables: error handling CreateTableBucket: filer reached === RUN TestManagerCreateTableBucketAuthorization/admin_principal_is_allowed E0824 22:18:28.533172 handler_bucket_create.go:111 S3Tables: failed to check bucket existence: filer reached E0824 22:18:28.533373 handler.go:218 S3Tables: error handling CreateTableBucket: filer reached --- PASS: TestManagerCreateTableBucketAuthorization (0.00s) --- PASS: TestManagerCreateTableBucketAuthorization/authenticated_identity_struct_is_enforced (0.00s) --- PASS: TestManagerCreateTableBucketAuthorization/secured_manager_denies_a_name_without_struct (0.00s) --- PASS: TestManagerCreateTableBucketAuthorization/untrusted_name_without_struct_is_enforced (0.00s) --- PASS: TestManagerCreateTableBucketAuthorization/trusted_manager_allows_a_name_without_struct (0.00s) --- PASS: TestManagerCreateTableBucketAuthorization/admin_principal_is_allowed (0.00s) === RUN TestMatchesActionPattern === RUN TestMatchesActionPattern/exact_match === RUN TestMatchesActionPattern/no_match === RUN TestMatchesActionPattern/universal_wildcard === RUN TestMatchesActionPattern/suffix_wildcard_match === RUN TestMatchesActionPattern/suffix_wildcard_no_match === RUN TestMatchesActionPattern/middle_wildcard_Get*Table === RUN TestMatchesActionPattern/middle_wildcard_Get*Table_no_match_GetTableBucket === RUN TestMatchesActionPattern/middle_wildcard_Get*Table_no_match_DeleteTable === RUN TestMatchesActionPattern/middle_wildcard_*Table* === RUN TestMatchesActionPattern/middle_wildcard_*Table*_match_CreateTable === RUN TestMatchesActionPattern/question_mark_single_char === RUN TestMatchesActionPattern/question_mark_no_match === RUN TestMatchesActionPattern/combined_*_and_?_singular === RUN TestMatchesActionPattern/combined_*_and_?_plural === RUN TestMatchesActionPattern/combined_no_match_-_?_needs_1_char --- PASS: TestMatchesActionPattern (0.00s) --- PASS: TestMatchesActionPattern/exact_match (0.00s) --- PASS: TestMatchesActionPattern/no_match (0.00s) --- PASS: TestMatchesActionPattern/universal_wildcard (0.00s) --- PASS: TestMatchesActionPattern/suffix_wildcard_match (0.00s) --- PASS: TestMatchesActionPattern/suffix_wildcard_no_match (0.00s) --- PASS: TestMatchesActionPattern/middle_wildcard_Get*Table (0.00s) --- PASS: TestMatchesActionPattern/middle_wildcard_Get*Table_no_match_GetTableBucket (0.00s) --- PASS: TestMatchesActionPattern/middle_wildcard_Get*Table_no_match_DeleteTable (0.00s) --- PASS: TestMatchesActionPattern/middle_wildcard_*Table* (0.00s) --- PASS: TestMatchesActionPattern/middle_wildcard_*Table*_match_CreateTable (0.00s) --- PASS: TestMatchesActionPattern/question_mark_single_char (0.00s) --- PASS: TestMatchesActionPattern/question_mark_no_match (0.00s) --- PASS: TestMatchesActionPattern/combined_*_and_?_singular (0.00s) --- PASS: TestMatchesActionPattern/combined_*_and_?_plural (0.00s) --- PASS: TestMatchesActionPattern/combined_no_match_-_?_needs_1_char (0.00s) === RUN TestMatchesPrincipal === RUN TestMatchesPrincipal/exact_match === RUN TestMatchesPrincipal/no_match === RUN TestMatchesPrincipal/universal_wildcard === RUN TestMatchesPrincipal/prefix_wildcard === RUN TestMatchesPrincipal/prefix_wildcard_no_match === RUN TestMatchesPrincipal/middle_wildcard === RUN TestMatchesPrincipal/array_match_first === RUN TestMatchesPrincipal/array_match_second === RUN TestMatchesPrincipal/array_no_match === RUN TestMatchesPrincipal/array_wildcard === RUN TestMatchesPrincipal/AWS_map_exact === RUN TestMatchesPrincipal/AWS_map_wildcard === RUN TestMatchesPrincipal/AWS_map_array === RUN TestMatchesPrincipal/nil_principal --- PASS: TestMatchesPrincipal (0.00s) --- PASS: TestMatchesPrincipal/exact_match (0.00s) --- PASS: TestMatchesPrincipal/no_match (0.00s) --- PASS: TestMatchesPrincipal/universal_wildcard (0.00s) --- PASS: TestMatchesPrincipal/prefix_wildcard (0.00s) --- PASS: TestMatchesPrincipal/prefix_wildcard_no_match (0.00s) --- PASS: TestMatchesPrincipal/middle_wildcard (0.00s) --- PASS: TestMatchesPrincipal/array_match_first (0.00s) --- PASS: TestMatchesPrincipal/array_match_second (0.00s) --- PASS: TestMatchesPrincipal/array_no_match (0.00s) --- PASS: TestMatchesPrincipal/array_wildcard (0.00s) --- PASS: TestMatchesPrincipal/AWS_map_exact (0.00s) --- PASS: TestMatchesPrincipal/AWS_map_wildcard (0.00s) --- PASS: TestMatchesPrincipal/AWS_map_array (0.00s) --- PASS: TestMatchesPrincipal/nil_principal (0.00s) === RUN TestEvaluatePolicyWithConditions === RUN TestEvaluatePolicyWithConditions/all_conditions_match === RUN TestEvaluatePolicyWithConditions/namespace_mismatch === RUN TestEvaluatePolicyWithConditions/table_name_mismatch === RUN TestEvaluatePolicyWithConditions/numeric_condition_failure === RUN TestEvaluatePolicyWithConditions/bool_condition_failure --- PASS: TestEvaluatePolicyWithConditions (0.00s) --- PASS: TestEvaluatePolicyWithConditions/all_conditions_match (0.00s) --- PASS: TestEvaluatePolicyWithConditions/namespace_mismatch (0.00s) --- PASS: TestEvaluatePolicyWithConditions/table_name_mismatch (0.00s) --- PASS: TestEvaluatePolicyWithConditions/numeric_condition_failure (0.00s) --- PASS: TestEvaluatePolicyWithConditions/bool_condition_failure (0.00s) === RUN TestCheckPermissionWithDefaultAllow === RUN TestCheckPermissionWithDefaultAllow/default_deny_(no_policy,_DefaultAllow=false) === RUN TestCheckPermissionWithDefaultAllow/default_allow_(no_policy,_DefaultAllow=true) === RUN TestCheckPermissionWithDefaultAllow/explicit_deny_overrides_DefaultAllow=true === RUN TestCheckPermissionWithDefaultAllow/explicit_allow_works_with_DefaultAllow=false --- PASS: TestCheckPermissionWithDefaultAllow (0.01s) --- PASS: TestCheckPermissionWithDefaultAllow/default_deny_(no_policy,_DefaultAllow=false) (0.00s) --- PASS: TestCheckPermissionWithDefaultAllow/default_allow_(no_policy,_DefaultAllow=true) (0.00s) --- PASS: TestCheckPermissionWithDefaultAllow/explicit_deny_overrides_DefaultAllow=true (0.00s) --- PASS: TestCheckPermissionWithDefaultAllow/explicit_allow_works_with_DefaultAllow=false (0.00s) === RUN TestTableDataDirFromMetadataLocation --- PASS: TestTableDataDirFromMetadataLocation (0.00s) === RUN TestMetadataVersionFromLocationUniqueSuffix --- PASS: TestMetadataVersionFromLocationUniqueSuffix (0.00s) === RUN TestValidateNamespaceSupportsMultiLevel --- PASS: TestValidateNamespaceSupportsMultiLevel (0.00s) === RUN TestValidateNamespaceSupportsDottedInput --- PASS: TestValidateNamespaceSupportsDottedInput (0.00s) === RUN TestValidateNamespaceRejectsEmptyDottedSegment --- PASS: TestValidateNamespaceRejectsEmptyDottedSegment (0.00s) === RUN TestParseNamespace === RUN TestParseNamespace/single_level === RUN TestParseNamespace/multi_level_dotted === RUN TestParseNamespace/invalid_reserved_prefix === RUN TestParseNamespace/invalid_empty_segment --- PASS: TestParseNamespace (0.00s) --- PASS: TestParseNamespace/single_level (0.00s) --- PASS: TestParseNamespace/multi_level_dotted (0.00s) --- PASS: TestParseNamespace/invalid_reserved_prefix (0.00s) --- PASS: TestParseNamespace/invalid_empty_segment (0.00s) === RUN TestParseTableFromARNWithMultiLevelNamespace --- PASS: TestParseTableFromARNWithMultiLevelNamespace (0.00s) === RUN TestBuildTableARNWithDottedNamespace --- PASS: TestBuildTableARNWithDottedNamespace (0.00s) === RUN TestExpandNamespace --- PASS: TestExpandNamespace (0.00s) === RUN TestNamespaceMetadataPropertiesRoundTrip === RUN TestNamespaceMetadataPropertiesRoundTrip/with_properties === RUN TestNamespaceMetadataPropertiesRoundTrip/nil_properties === RUN TestNamespaceMetadataPropertiesRoundTrip/empty_properties --- PASS: TestNamespaceMetadataPropertiesRoundTrip (0.00s) --- PASS: TestNamespaceMetadataPropertiesRoundTrip/with_properties (0.00s) --- PASS: TestNamespaceMetadataPropertiesRoundTrip/nil_properties (0.00s) --- PASS: TestNamespaceMetadataPropertiesRoundTrip/empty_properties (0.00s) PASS ok github.com/seaweedfs/seaweedfs/weed/s3api/s3tables 19.293s ? github.com/seaweedfs/seaweedfs/weed/s3api/s3tables/s3tablestest [no test files] === RUN TestUpdateSigningKeysRotates --- PASS: TestUpdateSigningKeysRotates (0.02s) === RUN TestUpdateSigningKeysConcurrent --- PASS: TestUpdateSigningKeysConcurrent (2.14s) === RUN TestUpdateSigningKeysTogglesWriteActive --- PASS: TestUpdateSigningKeysTogglesWriteActive (0.00s) === RUN TestIsWhiteListedEmptyConfigAllowsEverything --- PASS: TestIsWhiteListedEmptyConfigAllowsEverything (0.00s) === RUN TestIsWhiteListedWithListRejectsUnknown --- PASS: TestIsWhiteListedWithListRejectsUnknown (0.00s) === RUN TestIsWhiteListedWithSigningKeyButNoWhitelistAllowsAll --- PASS: TestIsWhiteListedWithSigningKeyButNoWhitelistAllowsAll (0.00s) === RUN TestLoadClientTLSPrefersClientCert --- PASS: TestLoadClientTLSPrefersClientCert (0.41s) === RUN TestLoadClientTLSFallsBackToServingCert --- PASS: TestLoadClientTLSFallsBackToServingCert (0.09s) === RUN TestLoadClientTLSServerOnlyEkuRejected --- PASS: TestLoadClientTLSServerOnlyEkuRejected (0.15s) === RUN TestSNI_HostnameStripsPort --- PASS: TestSNI_HostnameStripsPort (0.06s) === RUN TestSNI_IPAddressEmptySNI --- PASS: TestSNI_IPAddressEmptySNI (0.04s) === RUN TestSNI_IPv6AddressEmptySNI --- PASS: TestSNI_IPv6AddressEmptySNI (0.04s) PASS ok github.com/seaweedfs/seaweedfs/weed/security 13.893s === RUN TestServerGetCertificatePicksUpNewFile --- PASS: TestServerGetCertificatePicksUpNewFile (1.34s) PASS ok github.com/seaweedfs/seaweedfs/weed/security/certreload 8.074s === RUN TestSequencer I0824 22:18:19.958429 snowflake_sequencer.go:24 use snowflake seq id generator, nodeid:for_test hex_of_nodeid: 1 1d085357fec01000 1d085357fec01001 1d085357fec01002 1d085357fec01003 1d085357fec01004 1d085357fec01005 1d085357fec01006 1d085357fec01007 1d085357fec01008 1d085357fec01009 1d085357fec0100a 1d085357fec0100b 1d085357fec0100c 1d085357fec0100d 1d085357fec0100e 1d085357fec0100f 1d085357fec01010 1d085357fec01011 1d085357fec01012 1d085357fec01013 1d085357fec01014 1d085357fec01015 1d085357fec01016 1d085357fec01017 1d085357fec01018 1d085357fec01019 1d085357fec0101a 1d085357fec0101b 1d085357fec0101c 1d085357fec0101d 1d085357fec0101e 1d085357fec0101f 1d085357fec01020 1d085357fec01021 1d085357fec01022 1d085357fec01023 1d085357fec01024 1d085357fec01025 1d085357fec01026 1d085357fec01027 1d085357fec01028 1d085357fec01029 1d085357fec0102a 1d085357fec0102b 1d085357fec0102c 1d085357fec0102d 1d085357fec0102e 1d085357fec0102f 1d085357fec01030 1d085357fec01031 1d085357fec01032 1d085357fec01033 1d085357fec01034 1d085357fec01035 1d085357fec01036 1d085357fec01037 1d085357fec01038 1d085357fec01039 1d085357fec0103a 1d085357fec0103b 1d085357fec0103c 1d085357fec0103d 1d085357fec0103e 1d085357fec0103f 1d085357fec01040 1d085357fec01041 1d085357fec01042 1d085357fec01043 1d085357fec01044 1d085357fec01045 1d085357fec01046 1d085357fec01047 1d085357fec01048 1d085357fec01049 1d085357fec0104a 1d085357fec0104b 1d085357fec0104c 1d085357ff001000 1d085357ff001001 1d085357ff001002 1d085357ff001003 1d085357ff001004 1d085357ff001005 1d085357ff001006 1d085357ff001007 1d085357ff001008 1d085357ff001009 1d085357ff00100a 1d085357ff00100b 1d085357ff00100c 1d085357ff00100d 1d085357ff00100e 1d085357ff00100f 1d085357ff001010 1d085357ff001011 1d085357ff001012 1d085357ff001013 1d085357ff001014 1d085357ff001015 1d085357ff001016 --- PASS: TestSequencer (0.08s) PASS ok github.com/seaweedfs/seaweedfs/weed/sequence 10.694s === RUN TestParseURL --- PASS: TestParseURL (0.02s) === RUN TestWriteJsonNoJSONP === RUN TestWriteJsonNoJSONP/callback= === RUN TestWriteJsonNoJSONP/callback=myCb === RUN TestWriteJsonNoJSONP/callback= --- PASS: TestWriteJsonNoJSONP (0.13s) --- PASS: TestWriteJsonNoJSONP/callback= (0.07s) --- PASS: TestWriteJsonNoJSONP/callback=myCb (0.00s) --- PASS: TestWriteJsonNoJSONP/callback= (0.00s) === RUN TestWriteJsonPrettyDoesNotReadMultipartBody I0824 22:18:40.363379 common.go:109 response method:POST URL:/x with httpStatus:429 and JSON:{"error":"busy"} --- PASS: TestWriteJsonPrettyDoesNotReadMultipartBody (0.12s) === RUN TestChunkFidsEqualClause --- PASS: TestChunkFidsEqualClause (0.07s) === RUN TestUpdateEntryChunkConditionPreventsStrand === RUN TestUpdateEntryChunkConditionPreventsStrand/unconditional-resurrects === RUN TestUpdateEntryChunkConditionPreventsStrand/condition-rejects-stale === RUN TestUpdateEntryChunkConditionPreventsStrand/condition-passes-fresh === RUN TestUpdateEntryChunkConditionPreventsStrand/empty-expectation-rejects --- PASS: TestUpdateEntryChunkConditionPreventsStrand (0.98s) --- PASS: TestUpdateEntryChunkConditionPreventsStrand/unconditional-resurrects (0.21s) --- PASS: TestUpdateEntryChunkConditionPreventsStrand/condition-rejects-stale (0.31s) --- PASS: TestUpdateEntryChunkConditionPreventsStrand/condition-passes-fresh (0.01s) --- PASS: TestUpdateEntryChunkConditionPreventsStrand/empty-expectation-rejects (0.42s) === RUN TestDeleteEntryWaitsForPathLock --- PASS: TestDeleteEntryWaitsForPathLock (0.13s) === RUN TestWriteConditionSatisfied --- PASS: TestWriteConditionSatisfied (0.00s) === RUN TestWriteConditionClauses --- PASS: TestWriteConditionClauses (0.00s) === RUN TestWriteConditionObjectLockGuards --- PASS: TestWriteConditionObjectLockGuards (0.00s) === RUN TestWriteConditionUnknownKindFailsClosed --- PASS: TestWriteConditionUnknownKindFailsClosed (0.00s) === RUN TestStoredEntryETag --- PASS: TestStoredEntryETag (0.00s) === RUN TestCreateEntryConditionEnforced --- PASS: TestCreateEntryConditionEnforced (0.00s) === RUN TestCreateEntryReusesProvidedExisting --- PASS: TestCreateEntryReusesProvidedExisting (0.07s) === RUN TestUpdateEntryConditionEnforced --- PASS: TestUpdateEntryConditionEnforced (0.62s) === RUN TestCreateEntryOExclSerialized --- PASS: TestCreateEntryOExclSerialized (0.24s) === RUN TestFindLockOwnerExpiredLockReturnsNotFound I0824 22:18:42.522562 filer_grpc_server_dlm.go:122 find lock expired-lock moved to : lock not found --- PASS: TestFindLockOwnerExpiredLockReturnsNotFound (0.02s) === RUN TestMountRegister_EnabledStoresAndLists --- PASS: TestMountRegister_EnabledStoresAndLists (0.00s) === RUN TestMountRegister_RenewIsIdempotent --- PASS: TestMountRegister_RenewIsIdempotent (0.00s) === RUN TestObjectTransactionMultiEntry --- PASS: TestObjectTransactionMultiEntry (0.12s) === RUN TestObjectTransactionPatchNotifies --- PASS: TestObjectTransactionPatchNotifies (0.04s) === RUN TestObjectTransactionPatchContent --- PASS: TestObjectTransactionPatchContent (0.05s) === RUN TestObjectTransactionPreconditionAborts --- PASS: TestObjectTransactionPreconditionAborts (0.02s) === RUN TestObjectTransactionRecomputeLatest --- PASS: TestObjectTransactionRecomputeLatest (0.05s) === RUN TestObjectTransactionRecomputeAscending --- PASS: TestObjectTransactionRecomputeAscending (0.02s) === RUN TestObjectTransactionBatchIndependent --- PASS: TestObjectTransactionBatchIndependent (0.04s) === RUN TestObjectTransactionBatchNilTransaction --- PASS: TestObjectTransactionBatchNilTransaction (0.02s) === RUN TestObjectTransactionIdempotentNoops --- PASS: TestObjectTransactionIdempotentNoops (0.02s) === RUN TestObjectTransactionRecomputeDemoteAndAttrs --- PASS: TestObjectTransactionRecomputeDemoteAndAttrs (0.05s) === RUN TestObjectTransactionPutThenRecomputeLatest --- PASS: TestObjectTransactionPutThenRecomputeLatest (0.06s) === RUN TestObjectTransactionVersionDeleteWithWorm --- PASS: TestObjectTransactionVersionDeleteWithWorm (0.03s) === RUN TestObjectTransactionDeleteRemovesEmptyParent --- PASS: TestObjectTransactionDeleteRemovesEmptyParent (0.09s) === RUN TestObjectTransactionPatchTouchMtime --- PASS: TestObjectTransactionPatchTouchMtime (0.00s) === RUN TestObjectTransactionRouteKeyOwnerAppliesLocally --- PASS: TestObjectTransactionRouteKeyOwnerAppliesLocally (0.00s) === RUN TestObjectTransactionIsMovedSkipsForward --- PASS: TestObjectTransactionIsMovedSkipsForward (0.01s) === RUN TestObjectTransactionForwardsToOwner --- PASS: TestObjectTransactionForwardsToOwner (0.15s) === RUN TestObjectTransactionPutAppliesConfiguredTTLAndExpires --- PASS: TestObjectTransactionPutAppliesConfiguredTTLAndExpires (0.02s) === RUN TestObjectTransactionPutPreservesExplicitTTL --- PASS: TestObjectTransactionPutPreservesExplicitTTL (0.01s) === RUN TestObjectTransactionPutClearsTTLForRemoteEntry --- PASS: TestObjectTransactionPutClearsTTLForRemoteEntry (0.01s) === RUN TestObjectTransactionPutHonorsMaxFileNameLengthRule --- PASS: TestObjectTransactionPutHonorsMaxFileNameLengthRule (0.01s) === RUN TestObjectTransactionPutFailsOnReadOnlyStorageConfig --- PASS: TestObjectTransactionPutFailsOnReadOnlyStorageConfig (0.01s) === RUN TestPosixLockGrantAndConflict --- PASS: TestPosixLockGrantAndConflict (0.02s) === RUN TestPosixLockUnlockThenReacquire --- PASS: TestPosixLockUnlockThenReacquire (0.01s) === RUN TestPosixLockGetLk --- PASS: TestPosixLockGetLk (0.05s) === RUN TestPosixLockReleasePosixOwnerKeepsFlock --- PASS: TestPosixLockReleasePosixOwnerKeepsFlock (0.00s) === RUN TestPosixLockKeepAlive --- PASS: TestPosixLockKeepAlive (0.00s) === RUN TestPosixLockForwardsToOwner --- PASS: TestPosixLockForwardsToOwner (0.09s) === RUN TestPosixLockWarmupDefersGrants --- PASS: TestPosixLockWarmupDefersGrants (0.01s) === RUN TestRecomputeLatestEmitsPointerUpdateEvent --- PASS: TestRecomputeLatestEmitsPointerUpdateEvent (0.06s) === RUN TestRecomputeLatestDemoteEmitsEvent --- PASS: TestRecomputeLatestDemoteEmitsEvent (0.08s) === RUN TestAtomicRenameEntryEmitsLogicalRenameEvent --- PASS: TestAtomicRenameEntryEmitsLogicalRenameEvent (0.24s) === RUN TestAtomicRenameEntryOverwriteEmitsDeleteThenRename --- PASS: TestAtomicRenameEntryOverwriteEmitsDeleteThenRename (0.09s) === RUN TestAtomicRenameEntryDoesNotEmitEventOnDeleteFailure --- PASS: TestAtomicRenameEntryDoesNotEmitEventOnDeleteFailure (0.14s) === RUN TestAtomicRenameEntryDoesNotEmitEventOnCommitFailure --- PASS: TestAtomicRenameEntryDoesNotEmitEventOnCommitFailure (0.13s) === RUN TestAtomicRenameEntrySkipsDescendantTargetLookups --- PASS: TestAtomicRenameEntrySkipsDescendantTargetLookups (0.20s) === RUN TestStatisticsReplication --- PASS: TestStatisticsReplication (0.00s) === RUN TestReproStreamSerializationCeiling filer_grpc_server_stream_mutate_bench_test.go:550: service delay: 2ms filer_grpc_server_stream_mutate_bench_test.go:551: concurrency: 12 filer_grpc_server_stream_mutate_bench_test.go:552: ops per run: 1200 filer_grpc_server_stream_mutate_bench_test.go:553: unary : 1200 calls in 1.589321904s -> 755 QPS filer_grpc_server_stream_mutate_bench_test.go:554: stream : 1200 calls in 5.120179828s -> 234 QPS filer_grpc_server_stream_mutate_bench_test.go:555: unary / stream ratio: 3.22x --- PASS: TestReproStreamSerializationCeiling (6.75s) === RUN TestServerSerialVsConcurrentHandler filer_grpc_server_stream_mutate_bench_test.go:617: service delay: 2ms filer_grpc_server_stream_mutate_bench_test.go:618: server-side concurrency cap: 64 filer_grpc_server_stream_mutate_bench_test.go:619: theoretical serial ceiling: 500 QPS filer_grpc_server_stream_mutate_bench_test.go:620: theoretical concurrent cap: 32000 QPS filer_grpc_server_stream_mutate_bench_test.go:621: ops per run: 2400 filer_grpc_server_stream_mutate_bench_test.go:622: filer_grpc_server_stream_mutate_bench_test.go:623: client workers serial QPS concurrent QPS lift filer_grpc_server_stream_mutate_bench_test.go:637: 1 101 98 0.98x filer_grpc_server_stream_mutate_bench_test.go:637: 12 142 1108 7.78x filer_grpc_server_stream_mutate_bench_test.go:637: 64 133 2164 16.24x filer_grpc_server_stream_mutate_bench_test.go:637: 256 168 10061 59.88x --- PASS: TestServerSerialVsConcurrentHandler (101.16s) === RUN TestSchedulerOrderedParallelism filer_grpc_server_stream_mutate_bench_test.go:698: service delay: 2ms filer_grpc_server_stream_mutate_bench_test.go:699: server concurrency: 64 filer_grpc_server_stream_mutate_bench_test.go:700: client workers: 12 filer_grpc_server_stream_mutate_bench_test.go:701: ops per run: 2400 filer_grpc_server_stream_mutate_bench_test.go:702: serial ceiling: 500 QPS filer_grpc_server_stream_mutate_bench_test.go:703: concurrent ceiling: 32000 QPS (64 x 500) filer_grpc_server_stream_mutate_bench_test.go:704: filer_grpc_server_stream_mutate_bench_test.go:714: workload serial QPS sem QPS sched QPS sched peak-conc filer_grpc_server_stream_mutate_bench_test.go:732: distinct paths 143 1112 1044 12 filer_grpc_server_stream_mutate_bench_test.go:732: same path 122 1096 134 1 --- PASS: TestSchedulerOrderedParallelism (61.10s) === RUN TestStreamSyncVsAsyncClient filer_grpc_server_stream_mutate_bench_test.go:772: service delay: 2ms (theoretical serial ceiling = 500 QPS) filer_grpc_server_stream_mutate_bench_test.go:774: ops per run: 2400 filer_grpc_server_stream_mutate_bench_test.go:775: filer_grpc_server_stream_mutate_bench_test.go:776: concurrency sync QPS async QPS ratio filer_grpc_server_stream_mutate_bench_test.go:789: 1 100 110 1.11 x filer_grpc_server_stream_mutate_bench_test.go:789: 12 141 113 0.81 x filer_grpc_server_stream_mutate_bench_test.go:789: 64 153 102 0.67 x filer_grpc_server_stream_mutate_bench_test.go:789: 256 126 102 0.81 x --- PASS: TestStreamSyncVsAsyncClient (169.85s) === RUN TestMutatePathAncestors --- PASS: TestMutatePathAncestors (0.00s) === RUN TestClassifyMutation === RUN TestClassifyMutation/create_file === RUN TestClassifyMutation/create_directory_(barrier) === RUN TestClassifyMutation/update_file === RUN TestClassifyMutation/update_directory_(non-barrier_attr_bump) === RUN TestClassifyMutation/delete_non-recursive_(barrier:_target_type_unknown) === RUN TestClassifyMutation/delete_recursive_(barrier) === RUN TestClassifyMutation/malformed_create_(nil_Entry) === RUN TestClassifyMutation/malformed_update_(nil_Entry) === RUN TestClassifyMutation/empty_oneof === RUN TestClassifyMutation/rename --- PASS: TestClassifyMutation (0.00s) --- PASS: TestClassifyMutation/create_file (0.00s) --- PASS: TestClassifyMutation/create_directory_(barrier) (0.00s) --- PASS: TestClassifyMutation/update_file (0.00s) --- PASS: TestClassifyMutation/update_directory_(non-barrier_attr_bump) (0.00s) --- PASS: TestClassifyMutation/delete_non-recursive_(barrier:_target_type_unknown) (0.00s) --- PASS: TestClassifyMutation/delete_recursive_(barrier) (0.00s) --- PASS: TestClassifyMutation/malformed_create_(nil_Entry) (0.00s) --- PASS: TestClassifyMutation/malformed_update_(nil_Entry) (0.00s) --- PASS: TestClassifyMutation/empty_oneof (0.00s) --- PASS: TestClassifyMutation/rename (0.00s) === RUN TestPathConflictSamePathFile --- PASS: TestPathConflictSamePathFile (0.00s) === RUN TestPathConflictBarrierBlocksSamePath --- PASS: TestPathConflictBarrierBlocksSamePath (0.00s) === RUN TestPathConflictBarrierBlocksDescendants --- PASS: TestPathConflictBarrierBlocksDescendants (0.00s) === RUN TestPathConflictIncomingBarrierWaitsForDescendants --- PASS: TestPathConflictIncomingBarrierWaitsForDescendants (0.00s) === RUN TestPathConflictNonBarrierDirAllowsDescendants --- PASS: TestPathConflictNonBarrierDirAllowsDescendants (0.00s) === RUN TestPathConflictIncomingBarrierWaitsForSamePathNonBarrier --- PASS: TestPathConflictIncomingBarrierWaitsForSamePathNonBarrier (0.00s) === RUN TestAdmitDoneLifecycle --- PASS: TestAdmitDoneLifecycle (0.07s) === RUN TestAdmitParallelDistinctPaths --- PASS: TestAdmitParallelDistinctPaths (0.00s) === RUN TestAdmitConcurrencyCap --- PASS: TestAdmitConcurrencyCap (0.06s) === RUN TestAdmitRenameTwoPathConflict --- PASS: TestAdmitRenameTwoPathConflict (0.08s) === RUN TestAdmitSamePathFIFO --- PASS: TestAdmitSamePathFIFO (0.23s) === RUN TestAdmitSamePathNonBarrierSerializes --- PASS: TestAdmitSamePathNonBarrierSerializes (0.04s) === RUN TestAdmitPressureFromManyWaiters --- PASS: TestAdmitPressureFromManyWaiters (0.99s) === RUN TestResolveGapResume === RUN TestResolveGapResume/gap_the_ring_dropped_must_NOT_skip === RUN TestResolveGapResume/ancient_cursor_below_the_watermark_must_NOT_skip === RUN TestResolveGapResume/watermark_behind_earliest_must_NOT_skip_(may_be_unflushed) === RUN TestResolveGapResume/flush_watermark_at_earliest_proves_the_gap_and_skips === RUN TestResolveGapResume/flush_watermark_past_earliest_skips === RUN TestResolveGapResume/nothing_evicted_skips_despite_a_stale_flush_watermark === RUN TestResolveGapResume/cursor_on_the_eviction_watermark_skips_with_a_stalled_flush === RUN TestResolveGapResume/memory_not_ahead_of_current_must_NOT_skip === RUN TestResolveGapResume/no_in-memory_data_must_NOT_skip === RUN TestResolveGapResume/exclusive_cursor_adjacent_to_earliest_re-arms_to_a_sentinel === RUN TestResolveGapResume/sentinel_cursor_adjacent_to_earliest_must_NOT_re-arm --- PASS: TestResolveGapResume (0.00s) --- PASS: TestResolveGapResume/gap_the_ring_dropped_must_NOT_skip (0.00s) --- PASS: TestResolveGapResume/ancient_cursor_below_the_watermark_must_NOT_skip (0.00s) --- PASS: TestResolveGapResume/watermark_behind_earliest_must_NOT_skip_(may_be_unflushed) (0.00s) --- PASS: TestResolveGapResume/flush_watermark_at_earliest_proves_the_gap_and_skips (0.00s) --- PASS: TestResolveGapResume/flush_watermark_past_earliest_skips (0.00s) --- PASS: TestResolveGapResume/nothing_evicted_skips_despite_a_stale_flush_watermark (0.00s) --- PASS: TestResolveGapResume/cursor_on_the_eviction_watermark_skips_with_a_stalled_flush (0.00s) --- PASS: TestResolveGapResume/memory_not_ahead_of_current_must_NOT_skip (0.00s) --- PASS: TestResolveGapResume/no_in-memory_data_must_NOT_skip (0.00s) --- PASS: TestResolveGapResume/exclusive_cursor_adjacent_to_earliest_re-arms_to_a_sentinel (0.00s) --- PASS: TestResolveGapResume/sentinel_cursor_adjacent_to_earliest_must_NOT_re-arm (0.00s) === RUN TestGapPassProvenEmptyCrossing --- PASS: TestGapPassProvenEmptyCrossing (0.00s) === RUN TestChunkRefsStopTsNs --- PASS: TestChunkRefsStopTsNs (0.00s) === RUN TestInclusiveDiskCursorOnWatermarkStillAdvances --- PASS: TestInclusiveDiskCursorOnWatermarkStillAdvances (0.00s) === RUN TestParkOnGapExits === RUN TestParkOnGapExits/retries_on_the_timer_with_no_notification_channel === RUN TestParkOnGapExits/a_replaced_client_ends_the_stream_without_parking === RUN TestParkOnGapExits/a_bounded_subscription_past_its_window_ends_without_parking === RUN TestParkOnGapExits/a_bounded_subscription_exactly_at_its_window_ends_without_parking === RUN TestParkOnGapExits/a_cancelled_stream_ends_promptly === RUN TestParkOnGapExits/a_closed_notification_channel_does_not_spin === RUN TestParkOnGapExits/a_notification_wakes_it_early --- PASS: TestParkOnGapExits (4.02s) --- PASS: TestParkOnGapExits/retries_on_the_timer_with_no_notification_channel (2.01s) --- PASS: TestParkOnGapExits/a_replaced_client_ends_the_stream_without_parking (0.00s) --- PASS: TestParkOnGapExits/a_bounded_subscription_past_its_window_ends_without_parking (0.00s) --- PASS: TestParkOnGapExits/a_bounded_subscription_exactly_at_its_window_ends_without_parking (0.00s) --- PASS: TestParkOnGapExits/a_cancelled_stream_ends_promptly (0.00s) --- PASS: TestParkOnGapExits/a_closed_notification_channel_does_not_spin (2.01s) --- PASS: TestParkOnGapExits/a_notification_wakes_it_early (0.00s) === RUN TestGapResumeCursorOffsetIsSentinel --- PASS: TestGapResumeCursorOffsetIsSentinel (0.00s) === RUN TestDiskReadAdvancedRequiresForwardProgress --- PASS: TestDiskReadAdvancedRequiresForwardProgress (0.00s) === RUN TestReportUnprovenAggregatedCrossing W0824 22:24:29.237376 filer_grpc_server_sub_meta.go:231 aggregated subscriber c / crossed an evicted range (1970-01-01 00:00:00.00000001 +0000 UTC..1970-01-01 00:00:00.00000002 +0000 UTC] on peer disk reads; a peer that flushes into it later will not be re-read --- PASS: TestReportUnprovenAggregatedCrossing (0.01s) === RUN TestParkOnGapStallOutcomes === RUN TestParkOnGapStallOutcomes/a_stalled_park_below_the_watermark_skips_to_it W0824 22:24:29.454468 filer_grpc_server_sub_meta.go:347 aggregated subscriber c / parked 15m0s at 2026-08-24 21:23:29 +0000 UTC: test E0824 22:24:29.643055 filer_grpc_server_sub_meta.go:370 aggregated subscriber c / skipping the gap (2026-08-24 21:23:29 +0000 UTC..2026-08-24 21:24:29 +0000 UTC] after 15m0s parked: test; events a peer flushes into that range later will not be delivered === RUN TestParkOnGapStallOutcomes/a_stalled_park_with_nothing_to_skip_to_keeps_waiting W0824 22:24:29.650057 filer_grpc_server_sub_meta.go:347 aggregated subscriber c / parked 15m0s at 2026-08-24 21:24:29 +0000 UTC: test E0824 22:24:29.653322 filer_grpc_server_sub_meta.go:380 aggregated subscriber c / still parked after 15m0s at 2026-08-24 21:24:29 +0000 UTC with nothing to skip to: test W0824 22:24:31.654189 filer_grpc_server_sub_meta.go:391 aggregated subscriber c / disconnected after 2s parked, still behind --- PASS: TestParkOnGapStallOutcomes (2.41s) --- PASS: TestParkOnGapStallOutcomes/a_stalled_park_below_the_watermark_skips_to_it (0.19s) --- PASS: TestParkOnGapStallOutcomes/a_stalled_park_with_nothing_to_skip_to_keeps_waiting (2.00s) === RUN TestDeltaLogFileRefs --- PASS: TestDeltaLogFileRefs (0.00s) === RUN TestRefNeedsReship === RUN TestRefNeedsReship/file_above_the_answer_re-ships === RUN TestRefNeedsReship/complete_answering_file_stays_sent === RUN TestRefNeedsReship/prefix-limited_answering_file_re-ships_its_suffix === RUN TestRefNeedsReship/file_below_a_complete_answer_stays_sent === RUN TestRefNeedsReship/file_below_an_incomplete_answer_stays_sent_(client_moved_past) === RUN TestRefNeedsReship/everything_re-ships_when_nothing_answered --- PASS: TestRefNeedsReship (0.00s) --- PASS: TestRefNeedsReship/file_above_the_answer_re-ships (0.00s) --- PASS: TestRefNeedsReship/complete_answering_file_stays_sent (0.00s) --- PASS: TestRefNeedsReship/prefix-limited_answering_file_re-ships_its_suffix (0.00s) --- PASS: TestRefNeedsReship/file_below_a_complete_answer_stays_sent (0.00s) --- PASS: TestRefNeedsReship/file_below_an_incomplete_answer_stays_sent_(client_moved_past) (0.00s) --- PASS: TestRefNeedsReship/everything_re-ships_when_nothing_answered (0.00s) === RUN TestEachLogEntryFnPrefilterSkipsDecode --- PASS: TestEachLogEntryFnPrefilterSkipsDecode (0.02s) === RUN TestEachLogEntryFnNoFilterDecodesEverything --- PASS: TestEachLogEntryFnNoFilterDecodesEverything (0.00s) === RUN TestPipelinedSenderThroughput === RUN TestPipelinedSenderThroughput/direct_send filer_grpc_server_sub_meta_test.go:126: direct: 2100 events 2100 sends 5.388s 390 events/sec === RUN TestPipelinedSenderThroughput/pipelined_batched_send filer_grpc_server_sub_meta_test.go:151: pipelined+batch: 2100 events 12 sends 78ms 27017 events/sec === NAME TestPipelinedSenderThroughput filer_grpc_server_sub_meta_test.go:156: Speedup: 69.3x (pipelined+batched vs direct) --- PASS: TestPipelinedSenderThroughput (5.48s) --- PASS: TestPipelinedSenderThroughput/direct_send (5.39s) --- PASS: TestPipelinedSenderThroughput/pipelined_batched_send (0.08s) === RUN TestEachEventNotificationFnMatchesRenameTargetsForAllWatchTypes === RUN TestEachEventNotificationFnMatchesRenameTargetsForAllWatchTypes/additional_path_prefix === RUN TestEachEventNotificationFnMatchesRenameTargetsForAllWatchTypes/directory_watch --- PASS: TestEachEventNotificationFnMatchesRenameTargetsForAllWatchTypes (0.00s) --- PASS: TestEachEventNotificationFnMatchesRenameTargetsForAllWatchTypes/additional_path_prefix (0.00s) --- PASS: TestEachEventNotificationFnMatchesRenameTargetsForAllWatchTypes/directory_watch (0.00s) === RUN TestBatchingAdaptive === RUN TestBatchingAdaptive/old_events_are_batched filer_grpc_server_sub_meta_test.go:230: old events: 500 events in 2 sends (avg batch size: 250.0) === RUN TestBatchingAdaptive/recent_events_sent_individually filer_grpc_server_sub_meta_test.go:249: recent events: 500 events in 500 sends (avg batch size: 1.0) --- PASS: TestBatchingAdaptive (0.10s) --- PASS: TestBatchingAdaptive/old_events_are_batched (0.00s) --- PASS: TestBatchingAdaptive/recent_events_sent_individually (0.10s) === RUN TestPipelinedSenderErrorPropagation === RUN TestPipelinedSenderErrorPropagation/send_returns_error filer_grpc_server_sub_meta_test.go:298: Send returned error after stream failure: connection reset === RUN TestPipelinedSenderErrorPropagation/close_returns_error_if_not_consumed filer_grpc_server_sub_meta_test.go:315: Close returned error: connection reset --- PASS: TestPipelinedSenderErrorPropagation (0.00s) --- PASS: TestPipelinedSenderErrorPropagation/send_returns_error (0.00s) --- PASS: TestPipelinedSenderErrorPropagation/close_returns_error_if_not_consumed (0.00s) === RUN TestPipelinedSingleVsParallelStreams === RUN TestPipelinedSingleVsParallelStreams/1_pipelined_stream filer_grpc_server_sub_meta_test.go:383: 1 stream: 21000 events 110 sends 700ms 30019 events/sec === RUN TestPipelinedSingleVsParallelStreams/10_pipelined_streams filer_grpc_server_sub_meta_test.go:410: 10 streams: 21000 events 128 sends 99ms 211509 events/sec === NAME TestPipelinedSingleVsParallelStreams filer_grpc_server_sub_meta_test.go:415: Speedup: 7.0x (10 parallel pipelined streams vs 1) --- PASS: TestPipelinedSingleVsParallelStreams (0.85s) --- PASS: TestPipelinedSingleVsParallelStreams/1_pipelined_stream (0.70s) --- PASS: TestPipelinedSingleVsParallelStreams/10_pipelined_streams (0.10s) === RUN TestMaybeSendIdleHeartbeat === RUN TestMaybeSendIdleHeartbeat/not_opted_in === RUN TestMaybeSendIdleHeartbeat/behind_buffer_head === RUN TestMaybeSendIdleHeartbeat/caught_up_via_lastSeen === RUN TestMaybeSendIdleHeartbeat/caught_up_via_read_position_floor === RUN TestMaybeSendIdleHeartbeat/throttled_within_interval === RUN TestMaybeSendIdleHeartbeat/send_error_keeps_prior_heartbeat_time I0824 22:24:38.357628 filer_grpc_server_sub_meta.go:1201 => idle heartbeat to : broken stream --- PASS: TestMaybeSendIdleHeartbeat (0.24s) --- PASS: TestMaybeSendIdleHeartbeat/not_opted_in (0.00s) --- PASS: TestMaybeSendIdleHeartbeat/behind_buffer_head (0.00s) --- PASS: TestMaybeSendIdleHeartbeat/caught_up_via_lastSeen (0.00s) --- PASS: TestMaybeSendIdleHeartbeat/caught_up_via_read_position_floor (0.00s) --- PASS: TestMaybeSendIdleHeartbeat/throttled_within_interval (0.00s) --- PASS: TestMaybeSendIdleHeartbeat/send_error_keeps_prior_heartbeat_time (0.00s) === RUN TestFilteredEventsEmitMaxUnsyncedMarker filer_grpc_server_sub_meta_test.go:543: source emits marker{EventNotification:&{}, TsNs:1787610278358089933} after 1001 filtered events --- PASS: TestFilteredEventsEmitMaxUnsyncedMarker (0.01s) === RUN TestPipelinedSenderControlMessagesNeverNested --- PASS: TestPipelinedSenderControlMessagesNeverNested (0.00s) === RUN TestIdleHeartbeatCappedByInflightStamp --- PASS: TestIdleHeartbeatCappedByInflightStamp (0.00s) === RUN TestResolveAggReadHoldTsNs === RUN TestResolveAggReadHoldTsNs/healthy_watermark_ahead_of_horizon === RUN TestResolveAggReadHoldTsNs/stalled_watermark_falls_back_to_horizon === RUN TestResolveAggReadHoldTsNs/unknown_watermark_falls_back_to_horizon === RUN TestResolveAggReadHoldTsNs/watermark_exactly_at_horizon --- PASS: TestResolveAggReadHoldTsNs (0.00s) --- PASS: TestResolveAggReadHoldTsNs/healthy_watermark_ahead_of_horizon (0.00s) --- PASS: TestResolveAggReadHoldTsNs/stalled_watermark_falls_back_to_horizon (0.00s) --- PASS: TestResolveAggReadHoldTsNs/unknown_watermark_falls_back_to_horizon (0.00s) --- PASS: TestResolveAggReadHoldTsNs/watermark_exactly_at_horizon (0.00s) === RUN TestPreviousMinuteEndTsNs --- PASS: TestPreviousMinuteEndTsNs (0.00s) === RUN TestAggWatermarkHoldRewindRedelivers --- PASS: TestAggWatermarkHoldRewindRedelivers (0.00s) === RUN TestResolveAssignStorageOptionUsesBucketRuleBeforeFilerDiskDefault --- PASS: TestResolveAssignStorageOptionUsesBucketRuleBeforeFilerDiskDefault (0.00s) === RUN TestResolveAssignStorageOptionFallsBackToFilerDiskDefault --- PASS: TestResolveAssignStorageOptionFallsBackToFilerDiskDefault (0.00s) === RUN TestPtrie matched1 /topics/abc matched1 /topics/abc/d matched2 /topics/abc matched2 /topics/abc/d --- PASS: TestPtrie (0.00s) === RUN TestTraverseBfsMetadata I0824 22:24:38.375849 filer_grpc_server_traverse_meta.go:16 TraverseBfsMetadata directory:"/" excluded_prefixes:"/.system" --- PASS: TestTraverseBfsMetadata (0.02s) === RUN TestFilerServer_maybeCheckJwtAuthorization_Scoped === RUN TestFilerServer_maybeCheckJwtAuthorization_Scoped/no_restrictions === RUN TestFilerServer_maybeCheckJwtAuthorization_Scoped/allowed_prefix_match === RUN TestFilerServer_maybeCheckJwtAuthorization_Scoped/allowed_prefix_mismatch === RUN TestFilerServer_maybeCheckJwtAuthorization_Scoped/allowed_method_match === RUN TestFilerServer_maybeCheckJwtAuthorization_Scoped/allowed_method_mismatch === RUN TestFilerServer_maybeCheckJwtAuthorization_Scoped/both_match === RUN TestFilerServer_maybeCheckJwtAuthorization_Scoped/prefix_match,_method_mismatch === RUN TestFilerServer_maybeCheckJwtAuthorization_Scoped/multiple_prefixes_match === RUN TestFilerServer_maybeCheckJwtAuthorization_Scoped/write_operation_with_method_restriction === RUN TestFilerServer_maybeCheckJwtAuthorization_Scoped/root_path_with_prefix_restriction === RUN TestFilerServer_maybeCheckJwtAuthorization_Scoped/root_path_without_token === RUN TestFilerServer_maybeCheckJwtAuthorization_Scoped/exact_prefix_match === RUN TestFilerServer_maybeCheckJwtAuthorization_Scoped/tenant_prefix_does_not_match_sibling_tenant === RUN TestFilerServer_maybeCheckJwtAuthorization_Scoped/tenant_prefix_does_not_match_dashed_sibling === RUN TestFilerServer_maybeCheckJwtAuthorization_Scoped/tenant_prefix_matches_own_subtree === RUN TestFilerServer_maybeCheckJwtAuthorization_Scoped/tenant_prefix_with_trailing_slash_matches_own_subtree === RUN TestFilerServer_maybeCheckJwtAuthorization_Scoped/root_prefix_matches_anywhere === RUN TestFilerServer_maybeCheckJwtAuthorization_Scoped/dot-dot_cannot_escape_allowed_subtree === RUN TestFilerServer_maybeCheckJwtAuthorization_Scoped/copy_source_within_subtree_allowed === RUN TestFilerServer_maybeCheckJwtAuthorization_Scoped/copy_source_outside_subtree_denied === RUN TestFilerServer_maybeCheckJwtAuthorization_Scoped/copy_source_dot-dot_escape_denied === RUN TestFilerServer_maybeCheckJwtAuthorization_Scoped/move_source_outside_subtree_denied --- PASS: TestFilerServer_maybeCheckJwtAuthorization_Scoped (0.05s) --- PASS: TestFilerServer_maybeCheckJwtAuthorization_Scoped/no_restrictions (0.01s) --- PASS: TestFilerServer_maybeCheckJwtAuthorization_Scoped/allowed_prefix_match (0.00s) --- PASS: TestFilerServer_maybeCheckJwtAuthorization_Scoped/allowed_prefix_mismatch (0.00s) --- PASS: TestFilerServer_maybeCheckJwtAuthorization_Scoped/allowed_method_match (0.00s) --- PASS: TestFilerServer_maybeCheckJwtAuthorization_Scoped/allowed_method_mismatch (0.00s) --- PASS: TestFilerServer_maybeCheckJwtAuthorization_Scoped/both_match (0.00s) --- PASS: TestFilerServer_maybeCheckJwtAuthorization_Scoped/prefix_match,_method_mismatch (0.00s) --- PASS: TestFilerServer_maybeCheckJwtAuthorization_Scoped/multiple_prefixes_match (0.00s) --- PASS: TestFilerServer_maybeCheckJwtAuthorization_Scoped/write_operation_with_method_restriction (0.00s) --- PASS: TestFilerServer_maybeCheckJwtAuthorization_Scoped/root_path_with_prefix_restriction (0.00s) --- PASS: TestFilerServer_maybeCheckJwtAuthorization_Scoped/root_path_without_token (0.00s) --- PASS: TestFilerServer_maybeCheckJwtAuthorization_Scoped/exact_prefix_match (0.00s) --- PASS: TestFilerServer_maybeCheckJwtAuthorization_Scoped/tenant_prefix_does_not_match_sibling_tenant (0.00s) --- PASS: TestFilerServer_maybeCheckJwtAuthorization_Scoped/tenant_prefix_does_not_match_dashed_sibling (0.00s) --- PASS: TestFilerServer_maybeCheckJwtAuthorization_Scoped/tenant_prefix_matches_own_subtree (0.00s) --- PASS: TestFilerServer_maybeCheckJwtAuthorization_Scoped/tenant_prefix_with_trailing_slash_matches_own_subtree (0.00s) --- PASS: TestFilerServer_maybeCheckJwtAuthorization_Scoped/root_prefix_matches_anywhere (0.00s) --- PASS: TestFilerServer_maybeCheckJwtAuthorization_Scoped/dot-dot_cannot_escape_allowed_subtree (0.00s) --- PASS: TestFilerServer_maybeCheckJwtAuthorization_Scoped/copy_source_within_subtree_allowed (0.00s) --- PASS: TestFilerServer_maybeCheckJwtAuthorization_Scoped/copy_source_outside_subtree_denied (0.00s) --- PASS: TestFilerServer_maybeCheckJwtAuthorization_Scoped/copy_source_dot-dot_escape_denied (0.00s) --- PASS: TestFilerServer_maybeCheckJwtAuthorization_Scoped/move_source_outside_subtree_denied (0.00s) === RUN TestPathHasComponentPrefix === RUN TestPathHasComponentPrefix/exact_match === RUN TestPathHasComponentPrefix/subtree_match === RUN TestPathHasComponentPrefix/trailing_slash_prefix === RUN TestPathHasComponentPrefix/sibling_numeric_suffix === RUN TestPathHasComponentPrefix/sibling_dash_suffix === RUN TestPathHasComponentPrefix/sibling_dot_suffix === RUN TestPathHasComponentPrefix/unrelated_tree === RUN TestPathHasComponentPrefix/root_prefix_matches_root === RUN TestPathHasComponentPrefix/root_prefix_matches_any === RUN TestPathHasComponentPrefix/empty_prefix_denies === RUN TestPathHasComponentPrefix/dot-dot_does_not_escape === RUN TestPathHasComponentPrefix/dot-dot_stays_inside === RUN TestPathHasComponentPrefix/double_slashes_normalised --- PASS: TestPathHasComponentPrefix (0.00s) --- PASS: TestPathHasComponentPrefix/exact_match (0.00s) --- PASS: TestPathHasComponentPrefix/subtree_match (0.00s) --- PASS: TestPathHasComponentPrefix/trailing_slash_prefix (0.00s) --- PASS: TestPathHasComponentPrefix/sibling_numeric_suffix (0.00s) --- PASS: TestPathHasComponentPrefix/sibling_dash_suffix (0.00s) --- PASS: TestPathHasComponentPrefix/sibling_dot_suffix (0.00s) --- PASS: TestPathHasComponentPrefix/unrelated_tree (0.00s) --- PASS: TestPathHasComponentPrefix/root_prefix_matches_root (0.00s) --- PASS: TestPathHasComponentPrefix/root_prefix_matches_any (0.00s) --- PASS: TestPathHasComponentPrefix/empty_prefix_denies (0.00s) --- PASS: TestPathHasComponentPrefix/dot-dot_does_not_escape (0.00s) --- PASS: TestPathHasComponentPrefix/dot-dot_stays_inside (0.00s) --- PASS: TestPathHasComponentPrefix/double_slashes_normalised (0.00s) === RUN TestPipelinedSenderRefsNeverBatched --- PASS: TestPipelinedSenderRefsNeverBatched (0.08s) === RUN TestCopyEntryRefreshesDestinationTimestamps --- PASS: TestCopyEntryRefreshesDestinationTimestamps (0.00s) === RUN TestPreserveDestinationMetadataForDataCopy --- PASS: TestPreserveDestinationMetadataForDataCopy (0.00s) === RUN TestValidateCopySourcePreconditions --- PASS: TestValidateCopySourcePreconditions (0.00s) === RUN TestValidateCopyDestinationPreconditions --- PASS: TestValidateCopyDestinationPreconditions (0.00s) === RUN TestRecentCopyRequestDeduplicatesByRequestID --- PASS: TestRecentCopyRequestDeduplicatesByRequestID (0.00s) === RUN TestIamGrpc_NoMetadata_Unauthenticated --- PASS: TestIamGrpc_NoMetadata_Unauthenticated (0.00s) === RUN TestIamGrpc_MissingAuthorizationHeader_Unauthenticated --- PASS: TestIamGrpc_MissingAuthorizationHeader_Unauthenticated (0.00s) === RUN TestIamGrpc_NonBearerAuthorization_Unauthenticated --- PASS: TestIamGrpc_NonBearerAuthorization_Unauthenticated (0.00s) === RUN TestIamGrpc_InvalidToken_Unauthenticated --- PASS: TestIamGrpc_InvalidToken_Unauthenticated (0.00s) === RUN TestIamGrpc_GarbageToken_Unauthenticated --- PASS: TestIamGrpc_GarbageToken_Unauthenticated (0.00s) === RUN TestIamGrpc_ExpiredToken_Unauthenticated --- PASS: TestIamGrpc_ExpiredToken_Unauthenticated (0.00s) === RUN TestIamGrpc_ValidToken_ReachesHandler --- PASS: TestIamGrpc_ValidToken_ReachesHandler (0.00s) === RUN TestIamGrpc_NoSigningKey_Unauthenticated_Allowed --- PASS: TestIamGrpc_NoSigningKey_Unauthenticated_Allowed (0.00s) === RUN TestIamGrpc_CreateUser_RequiresAuth --- PASS: TestIamGrpc_CreateUser_RequiresAuth (0.00s) === RUN TestProxyReadDropsCallerJwtQueryParam === RUN TestProxyReadDropsCallerJwtQueryParam/GET === RUN TestProxyReadDropsCallerJwtQueryParam/HEAD --- PASS: TestProxyReadDropsCallerJwtQueryParam (0.02s) --- PASS: TestProxyReadDropsCallerJwtQueryParam/GET (0.01s) --- PASS: TestProxyReadDropsCallerJwtQueryParam/HEAD (0.00s) === RUN TestProxyWriteKeepsCallerJwtQueryParam --- PASS: TestProxyWriteKeepsCallerJwtQueryParam (0.00s) === RUN TestProxyMintsNothingWithoutReadKey --- PASS: TestProxyMintsNothingWithoutReadKey (0.00s) === RUN TestProxyReadTokenIsReadOnly --- PASS: TestProxyReadTokenIsReadOnly (0.00s) === RUN TestProxyWriteCarriesOnlyCallerCredential === RUN TestProxyWriteCarriesOnlyCallerCredential/anonymous_post === RUN TestProxyWriteCarriesOnlyCallerCredential/anonymous_post_with_read_key === RUN TestProxyWriteCarriesOnlyCallerCredential/anonymous_delete === RUN TestProxyWriteCarriesOnlyCallerCredential/anonymous_delete_with_read_key === RUN TestProxyWriteCarriesOnlyCallerCredential/anonymous_put_with_read_key === RUN TestProxyWriteCarriesOnlyCallerCredential/caller_token_forwarded_on_post === RUN TestProxyWriteCarriesOnlyCallerCredential/caller_token_forwarded_on_delete --- PASS: TestProxyWriteCarriesOnlyCallerCredential (0.03s) --- PASS: TestProxyWriteCarriesOnlyCallerCredential/anonymous_post (0.00s) --- PASS: TestProxyWriteCarriesOnlyCallerCredential/anonymous_post_with_read_key (0.00s) --- PASS: TestProxyWriteCarriesOnlyCallerCredential/anonymous_delete (0.00s) --- PASS: TestProxyWriteCarriesOnlyCallerCredential/anonymous_delete_with_read_key (0.00s) --- PASS: TestProxyWriteCarriesOnlyCallerCredential/anonymous_put_with_read_key (0.00s) --- PASS: TestProxyWriteCarriesOnlyCallerCredential/caller_token_forwarded_on_post (0.01s) --- PASS: TestProxyWriteCarriesOnlyCallerCredential/caller_token_forwarded_on_delete (0.01s) === RUN TestProxyReadReplacesCallerCredential --- PASS: TestProxyReadReplacesCallerCredential (0.01s) === RUN TestProxyReadDropsCallerCredentialWhenNothingMinted === RUN TestProxyReadDropsCallerCredentialWhenNothingMinted/GET === RUN TestProxyReadDropsCallerCredentialWhenNothingMinted/HEAD --- PASS: TestProxyReadDropsCallerCredentialWhenNothingMinted (0.01s) --- PASS: TestProxyReadDropsCallerCredentialWhenNothingMinted/GET (0.01s) --- PASS: TestProxyReadDropsCallerCredentialWhenNothingMinted/HEAD (0.00s) === RUN TestProxyWriteBypassesReadSemaphore --- PASS: TestProxyWriteBypassesReadSemaphore (0.00s) === RUN TestProxyReadTokenMatchesDeltaFid --- PASS: TestProxyReadTokenMatchesDeltaFid (0.00s) === RUN TestValidateProxyChunkId --- PASS: TestValidateProxyChunkId (0.01s) === RUN TestProxyRejectsTraversalBeforeLookup --- PASS: TestProxyRejectsTraversalBeforeLookup (0.00s) === RUN TestProxySemaphore_LimitsConcurrency --- PASS: TestProxySemaphore_LimitsConcurrency (0.03s) === RUN TestProxySemaphore_ContextCancellation --- PASS: TestProxySemaphore_ContextCancellation (0.00s) === RUN TestProxySemaphore_PerHostIsolation --- PASS: TestProxySemaphore_PerHostIsolation (0.00s) === RUN TestFilerServer_tusPatchHandler_PartialSkipsCompletion === RUN TestFilerServer_tusPatchHandler_PartialSkipsCompletion/plain_upload_completes === RUN TestFilerServer_tusPatchHandler_PartialSkipsCompletion/partial_upload_retained --- PASS: TestFilerServer_tusPatchHandler_PartialSkipsCompletion (0.08s) --- PASS: TestFilerServer_tusPatchHandler_PartialSkipsCompletion/plain_upload_completes (0.08s) --- PASS: TestFilerServer_tusPatchHandler_PartialSkipsCompletion/partial_upload_retained (0.00s) === RUN TestFilerServer_tusHeadHandler_EchoesUploadConcat --- PASS: TestFilerServer_tusHeadHandler_EchoesUploadConcat (0.00s) === RUN TestFilerServer_tusConcatFinal_AssemblesPartials --- PASS: TestFilerServer_tusConcatFinal_AssemblesPartials (0.02s) === RUN TestFilerServer_tusConcatFinal_Validation === RUN TestFilerServer_tusConcatFinal_Validation/upload_length_rejected === RUN TestFilerServer_tusConcatFinal_Validation/body_rejected === RUN TestFilerServer_tusConcatFinal_Validation/reference_outside_uploads_prefix === RUN TestFilerServer_tusConcatFinal_Validation/duplicate_reference === RUN TestFilerServer_tusConcatFinal_Validation/empty_reference_list === RUN TestFilerServer_tusConcatFinal_Validation/missing_partial === RUN TestFilerServer_tusConcatFinal_Validation/non-partial_session_rejected === RUN TestFilerServer_tusConcatFinal_Validation/unfinished_partial_rejected === RUN TestFilerServer_tusConcatFinal_Validation/cross-prefix_partial_denied I0824 22:24:38.878253 common.go:109 response method:POST URL:/.tus/buckets/data/final.bin with httpStatus:401 and JSON:{"error":"wrong jwt"} === RUN TestFilerServer_tusConcatFinal_Validation/combined_size_over_maximum --- PASS: TestFilerServer_tusConcatFinal_Validation (0.05s) --- PASS: TestFilerServer_tusConcatFinal_Validation/upload_length_rejected (0.00s) --- PASS: TestFilerServer_tusConcatFinal_Validation/body_rejected (0.00s) --- PASS: TestFilerServer_tusConcatFinal_Validation/reference_outside_uploads_prefix (0.00s) --- PASS: TestFilerServer_tusConcatFinal_Validation/duplicate_reference (0.00s) --- PASS: TestFilerServer_tusConcatFinal_Validation/empty_reference_list (0.00s) --- PASS: TestFilerServer_tusConcatFinal_Validation/missing_partial (0.00s) --- PASS: TestFilerServer_tusConcatFinal_Validation/non-partial_session_rejected (0.01s) --- PASS: TestFilerServer_tusConcatFinal_Validation/unfinished_partial_rejected (0.01s) --- PASS: TestFilerServer_tusConcatFinal_Validation/cross-prefix_partial_denied (0.00s) --- PASS: TestFilerServer_tusConcatFinal_Validation/combined_size_over_maximum (0.00s) === RUN TestFilerServer_tusConcatFinal_ClaimedPartialRejected --- PASS: TestFilerServer_tusConcatFinal_ClaimedPartialRejected (0.00s) === RUN TestFilerServer_tusDeleteHandler_ConsumedSessionKeepsChunks --- PASS: TestFilerServer_tusDeleteHandler_ConsumedSessionKeepsChunks (0.01s) === RUN TestFilerServer_tusPatchHandler_FinalRejected --- PASS: TestFilerServer_tusPatchHandler_FinalRejected (0.00s) === RUN TestFilerServer_tusHandler_CrossPrefixSessionHijack === RUN TestFilerServer_tusHandler_CrossPrefixSessionHijack/cross-prefix_HEAD_denied I0824 22:24:38.912553 common.go:109 response method:HEAD URL:/.tus/.uploads/9f6f0d4b-6556-48f6-b953-0d8fca1966f1 with httpStatus:401 and JSON:{"error":"wrong jwt"} === RUN TestFilerServer_tusHandler_CrossPrefixSessionHijack/matching-prefix_HEAD_allowed === RUN TestFilerServer_tusHandler_CrossPrefixSessionHijack/cross-prefix_PATCH_denied I0824 22:24:38.987419 common.go:109 response method:PATCH URL:/.tus/.uploads/9f6f0d4b-6556-48f6-b953-0d8fca1966f1 with httpStatus:401 and JSON:{"error":"wrong jwt"} === RUN TestFilerServer_tusHandler_CrossPrefixSessionHijack/matching-prefix_PATCH_allowed === RUN TestFilerServer_tusHandler_CrossPrefixSessionHijack/cross-prefix_DELETE_denied I0824 22:24:39.118426 common.go:109 response method:DELETE URL:/.tus/.uploads/9f6f0d4b-6556-48f6-b953-0d8fca1966f1 with httpStatus:401 and JSON:{"error":"wrong jwt"} === RUN TestFilerServer_tusHandler_CrossPrefixSessionHijack/matching-prefix_DELETE_allowed --- PASS: TestFilerServer_tusHandler_CrossPrefixSessionHijack (0.29s) --- PASS: TestFilerServer_tusHandler_CrossPrefixSessionHijack/cross-prefix_HEAD_denied (0.02s) --- PASS: TestFilerServer_tusHandler_CrossPrefixSessionHijack/matching-prefix_HEAD_allowed (0.01s) --- PASS: TestFilerServer_tusHandler_CrossPrefixSessionHijack/cross-prefix_PATCH_denied (0.06s) --- PASS: TestFilerServer_tusHandler_CrossPrefixSessionHijack/matching-prefix_PATCH_allowed (0.07s) --- PASS: TestFilerServer_tusHandler_CrossPrefixSessionHijack/cross-prefix_DELETE_denied (0.06s) --- PASS: TestFilerServer_tusHandler_CrossPrefixSessionHijack/matching-prefix_DELETE_allowed (0.06s) === RUN TestFilerServer_tusHandler_RejectsAliasesAndInvalidMetadata === RUN TestFilerServer_tusHandler_RejectsAliasesAndInvalidMetadata/trailing_path_alias === RUN TestFilerServer_tusHandler_RejectsAliasesAndInvalidMetadata/non-canonical_route_id === RUN TestFilerServer_tusHandler_RejectsAliasesAndInvalidMetadata/stored_id_mismatch === RUN TestFilerServer_tusHandler_RejectsAliasesAndInvalidMetadata/empty_stored_target === RUN TestFilerServer_tusHandler_RejectsAliasesAndInvalidMetadata/root_stored_target === RUN TestFilerServer_tusHandler_RejectsAliasesAndInvalidMetadata/relative_stored_target === RUN TestFilerServer_tusHandler_RejectsAliasesAndInvalidMetadata/oversize_stored_size --- PASS: TestFilerServer_tusHandler_RejectsAliasesAndInvalidMetadata (0.30s) --- PASS: TestFilerServer_tusHandler_RejectsAliasesAndInvalidMetadata/trailing_path_alias (0.07s) --- PASS: TestFilerServer_tusHandler_RejectsAliasesAndInvalidMetadata/non-canonical_route_id (0.07s) --- PASS: TestFilerServer_tusHandler_RejectsAliasesAndInvalidMetadata/stored_id_mismatch (0.02s) --- PASS: TestFilerServer_tusHandler_RejectsAliasesAndInvalidMetadata/empty_stored_target (0.00s) --- PASS: TestFilerServer_tusHandler_RejectsAliasesAndInvalidMetadata/root_stored_target (0.00s) --- PASS: TestFilerServer_tusHandler_RejectsAliasesAndInvalidMetadata/relative_stored_target (0.13s) --- PASS: TestFilerServer_tusHandler_RejectsAliasesAndInvalidMetadata/oversize_stored_size (0.01s) === RUN TestFilerServer_checkJwtAuthorization === RUN TestFilerServer_checkJwtAuthorization/post_without_token_denied === RUN TestFilerServer_checkJwtAuthorization/patch_without_token_denied === RUN TestFilerServer_checkJwtAuthorization/head_without_token_denied === RUN TestFilerServer_checkJwtAuthorization/post_write_token_allowed === RUN TestFilerServer_checkJwtAuthorization/head_read_token_allowed === RUN TestFilerServer_checkJwtAuthorization/head_write_token_denied === RUN TestFilerServer_checkJwtAuthorization/post_read_token_denied === RUN TestFilerServer_checkJwtAuthorization/within_allowed_prefix === RUN TestFilerServer_checkJwtAuthorization/outside_allowed_prefix_denied === RUN TestFilerServer_checkJwtAuthorization/restricted_token_without_resource_denied === RUN TestFilerServer_checkJwtAuthorization/matching_method_allowed === RUN TestFilerServer_checkJwtAuthorization/method_not_allowed_denied --- PASS: TestFilerServer_checkJwtAuthorization (0.01s) --- PASS: TestFilerServer_checkJwtAuthorization/post_without_token_denied (0.00s) --- PASS: TestFilerServer_checkJwtAuthorization/patch_without_token_denied (0.00s) --- PASS: TestFilerServer_checkJwtAuthorization/head_without_token_denied (0.00s) --- PASS: TestFilerServer_checkJwtAuthorization/post_write_token_allowed (0.00s) --- PASS: TestFilerServer_checkJwtAuthorization/head_read_token_allowed (0.00s) --- PASS: TestFilerServer_checkJwtAuthorization/head_write_token_denied (0.00s) --- PASS: TestFilerServer_checkJwtAuthorization/post_read_token_denied (0.00s) --- PASS: TestFilerServer_checkJwtAuthorization/within_allowed_prefix (0.00s) --- PASS: TestFilerServer_checkJwtAuthorization/outside_allowed_prefix_denied (0.00s) --- PASS: TestFilerServer_checkJwtAuthorization/restricted_token_without_resource_denied (0.00s) --- PASS: TestFilerServer_checkJwtAuthorization/matching_method_allowed (0.00s) --- PASS: TestFilerServer_checkJwtAuthorization/method_not_allowed_denied (0.00s) === RUN TestFilerServer_tusHandler_UnauthenticatedRejected I0824 22:24:39.501181 common.go:109 response method:POST URL:/.tus/buckets/secret/owned.txt with httpStatus:401 and JSON:{"error":"wrong jwt"} I0824 22:24:39.501415 common.go:109 response method:HEAD URL:/.tus/.uploads/9f6f0d4b-6556-48f6-b953-0d8fca1966f1 with httpStatus:401 and JSON:{"error":"wrong jwt"} --- PASS: TestFilerServer_tusHandler_UnauthenticatedRejected (0.00s) === RUN TestFilerServer_refreshTusSessionChunks_RevalidatesPinnedSession === RUN TestFilerServer_refreshTusSessionChunks_RevalidatesPinnedSession/deleted_before_completion === RUN TestFilerServer_refreshTusSessionChunks_RevalidatesPinnedSession/replaced_before_completion --- PASS: TestFilerServer_refreshTusSessionChunks_RevalidatesPinnedSession (0.01s) --- PASS: TestFilerServer_refreshTusSessionChunks_RevalidatesPinnedSession/deleted_before_completion (0.00s) --- PASS: TestFilerServer_refreshTusSessionChunks_RevalidatesPinnedSession/replaced_before_completion (0.01s) === RUN TestSubscribeLoop_AggregatedHoldIsQuiet I0824 22:24:39.571835 leveldb_store.go:48 filer store dir: /tmp/TestSubscribeLoop_AggregatedHoldIsQuiet612071634/001 I0824 22:24:39.571992 file_util.go:29 Folder /tmp/TestSubscribeLoop_AggregatedHoldIsQuiet612071634/001 Permission: -rwxr-xr-x I0824 22:24:39.606320 filer.go:201 create filer.store.id to 387817355 I0824 22:24:39.685491 filer_grpc_server_sub_meta.go:1443 + listener agg-loop-test@127.0.0.1:12345 clientId 11 clientEpoch 1 I0824 22:24:39.685598 filer_grpc_server_sub_meta.go:598 agg-loop-test@127.0.0.1:12345 starts to subscribe from {Time:2026-08-24 22:24:39.185288041 +0000 UTC Offset:-7 IsOffsetBased:false} filer_subscribe_agg_test.go:178: 121 writes over 1.009520432s produced 0 ERROR lines I0824 22:24:40.695259 filer_grpc_server_sub_meta.go:593 disconnect agg-loop-test@127.0.0.1:12345 subscriber clientId:11 I0824 22:24:40.695316 filer_grpc_server_sub_meta.go:1471 - listener agg-loop-test@127.0.0.1:12345 clientId 11 clientEpoch 1 I0824 22:24:40.695576 filer_deletion.go:515 retry processor shutting down, 0 items remaining in queue I0824 22:24:40.695648 filer_deletion.go:306 deletion processor shutting down --- PASS: TestSubscribeLoop_AggregatedHoldIsQuiet (1.19s) === RUN TestSubscribeLoop_AggregatedHoldIsPaced I0824 22:24:40.735783 leveldb_store.go:48 filer store dir: /tmp/TestSubscribeLoop_AggregatedHoldIsPaced1300283024/001 I0824 22:24:40.736060 file_util.go:29 Folder /tmp/TestSubscribeLoop_AggregatedHoldIsPaced1300283024/001 Permission: -rwxr-xr-x I0824 22:24:40.752519 filer.go:201 create filer.store.id to 1608008269 I0824 22:24:40.795924 filer_grpc_server_sub_meta.go:1443 + listener agg-loop-test@127.0.0.1:12345 clientId 11 clientEpoch 1 I0824 22:24:40.796062 filer_grpc_server_sub_meta.go:598 agg-loop-test@127.0.0.1:12345 starts to subscribe from {Time:2026-08-24 22:24:40.295338586 +0000 UTC Offset:-7 IsOffsetBased:false} filer_subscribe_agg_test.go:204: 140 writes over 1.009108652s produced 5 holds I0824 22:24:41.805361 filer_grpc_server_sub_meta.go:593 disconnect agg-loop-test@127.0.0.1:12345 subscriber clientId:11 I0824 22:24:41.805418 filer_grpc_server_sub_meta.go:1471 - listener agg-loop-test@127.0.0.1:12345 clientId 11 clientEpoch 1 I0824 22:24:41.805894 filer_deletion.go:306 deletion processor shutting down I0824 22:24:41.805908 filer_deletion.go:515 retry processor shutting down, 0 items remaining in queue --- PASS: TestSubscribeLoop_AggregatedHoldIsPaced (1.11s) === RUN TestSubscribeLoop_AggregatedHoldReleasesOnPeerProgress I0824 22:24:41.853918 leveldb_store.go:48 filer store dir: /tmp/TestSubscribeLoop_AggregatedHoldReleasesOnPeerProgress1885998675/001 I0824 22:24:41.854082 file_util.go:29 Folder /tmp/TestSubscribeLoop_AggregatedHoldReleasesOnPeerProgress1885998675/001 Permission: -rwxr-xr-x I0824 22:24:41.875911 filer.go:201 create filer.store.id to -641887713 I0824 22:24:42.104106 filer_grpc_server_sub_meta.go:1443 + listener agg-loop-test@127.0.0.1:12345 clientId 11 clientEpoch 1 I0824 22:24:42.107684 filer_grpc_server_sub_meta.go:598 agg-loop-test@127.0.0.1:12345 starts to subscribe from {Time:2026-08-24 22:24:41.602944459 +0000 UTC Offset:-7 IsOffsetBased:false} I0824 22:24:42.555888 filer_grpc_server_sub_meta.go:593 disconnect agg-loop-test@127.0.0.1:12345 subscriber clientId:11 I0824 22:24:42.556048 filer_grpc_server_sub_meta.go:1471 - listener agg-loop-test@127.0.0.1:12345 clientId 11 clientEpoch 1 I0824 22:24:42.556662 filer_deletion.go:515 retry processor shutting down, 0 items remaining in queue I0824 22:24:42.556769 filer_deletion.go:306 deletion processor shutting down --- PASS: TestSubscribeLoop_AggregatedHoldReleasesOnPeerProgress (0.75s) === RUN TestSubscribeLoop_AggregatedHoldCoalescesPeerProgress I0824 22:24:42.579769 leveldb_store.go:48 filer store dir: /tmp/TestSubscribeLoop_AggregatedHoldCoalescesPeerProgress2701370096/001 I0824 22:24:42.579933 file_util.go:29 Folder /tmp/TestSubscribeLoop_AggregatedHoldCoalescesPeerProgress2701370096/001 Permission: -rwxr-xr-x I0824 22:24:42.592267 filer.go:201 create filer.store.id to -2015151140 I0824 22:24:42.810131 filer_grpc_server_sub_meta.go:1443 + listener agg-loop-test@127.0.0.1:12345 clientId 11 clientEpoch 1 I0824 22:24:42.810634 filer_grpc_server_sub_meta.go:598 agg-loop-test@127.0.0.1:12345 starts to subscribe from {Time:2026-08-24 22:24:42.309134431 +0000 UTC Offset:-7 IsOffsetBased:false} filer_subscribe_agg_test.go:243: 101 writes over 1.004745233s produced 36 holds I0824 22:24:44.094529 filer_grpc_server_sub_meta.go:593 disconnect agg-loop-test@127.0.0.1:12345 subscriber clientId:11 I0824 22:24:44.094651 filer_grpc_server_sub_meta.go:1471 - listener agg-loop-test@127.0.0.1:12345 clientId 11 clientEpoch 1 I0824 22:24:44.094994 filer_deletion.go:306 deletion processor shutting down I0824 22:24:44.095122 filer_deletion.go:515 retry processor shutting down, 0 items remaining in queue --- PASS: TestSubscribeLoop_AggregatedHoldCoalescesPeerProgress (1.54s) === RUN TestPeerDeliveryClaimPacing I0824 22:24:44.104052 leveldb_store.go:48 filer store dir: /tmp/TestPeerDeliveryClaimPacing2695102983/001 I0824 22:24:44.104242 file_util.go:29 Folder /tmp/TestPeerDeliveryClaimPacing2695102983/001 Permission: -rwxr-xr-x I0824 22:24:44.110290 filer.go:201 create filer.store.id to 796329758 I0824 22:24:44.170144 filer_deletion.go:515 retry processor shutting down, 0 items remaining in queue I0824 22:24:44.170243 filer_deletion.go:306 deletion processor shutting down --- PASS: TestPeerDeliveryClaimPacing (0.08s) === RUN TestSubscribeLoop_AggregatedDiskHoldIgnoresDeliveryProgress I0824 22:24:44.176282 leveldb_store.go:48 filer store dir: /tmp/TestSubscribeLoop_AggregatedDiskHoldIgnoresDeliveryProgress2359380947/001 I0824 22:24:44.176432 file_util.go:29 Folder /tmp/TestSubscribeLoop_AggregatedDiskHoldIgnoresDeliveryProgress2359380947/001 Permission: -rwxr-xr-x I0824 22:24:44.186681 filer.go:201 create filer.store.id to -1738407449 I0824 22:24:44.229287 filer_grpc_server_sub_meta.go:1443 + listener agg-loop-test@127.0.0.1:12345 clientId 11 clientEpoch 1 I0824 22:24:44.229441 filer_grpc_server_sub_meta.go:598 agg-loop-test@127.0.0.1:12345 starts to subscribe from {Time:2026-08-24 22:24:09.20678692 +0000 UTC Offset:-7 IsOffsetBased:false} filer_subscribe_agg_test.go:309: 1.010039932s of delivery-only progress produced 2 disk holds I0824 22:24:45.705631 filer_grpc_server_sub_meta.go:593 disconnect agg-loop-test@127.0.0.1:12345 subscriber clientId:11 I0824 22:24:45.706001 filer_grpc_server_sub_meta.go:1471 - listener agg-loop-test@127.0.0.1:12345 clientId 11 clientEpoch 1 I0824 22:24:45.706891 filer_deletion.go:306 deletion processor shutting down I0824 22:24:45.707213 filer_deletion.go:515 retry processor shutting down, 0 items remaining in queue --- PASS: TestSubscribeLoop_AggregatedDiskHoldIgnoresDeliveryProgress (1.54s) === RUN TestSubscribeLoop_EvictedUnflushedGapWaitsThenDelivers I0824 22:24:45.734007 leveldb_store.go:48 filer store dir: /tmp/TestSubscribeLoop_EvictedUnflushedGapWaitsThenDelivers3116702003/001 I0824 22:24:45.734659 file_util.go:29 Folder /tmp/TestSubscribeLoop_EvictedUnflushedGapWaitsThenDelivers3116702003/001 Permission: -rwxr-xr-x I0824 22:24:45.749988 filer.go:201 create filer.store.id to -1073960458 E0824 22:24:45.772226 master_grpc_server.go:600 failed to get peer from ctx I0824 22:24:45.772560 filer_grpc_server_sub_meta.go:1443 + local listener loop-test@ clientId -7 clientEpoch 1 I0824 22:24:45.772702 filer_grpc_server_sub_meta.go:935 + loop-test@ local subscribe from {Time:1970-01-01 00:00:00 +0000 UTC Offset:-7 IsOffsetBased:false} clientId:-7 W0824 22:24:45.995063 filer_grpc_server_sub_meta.go:347 local subscriber loop-test@ parked 0s at 1970-01-01 00:00:00 +0000 UTC: gap is not flushed yet (earliest memory 2026-08-24 21:28:45 +0000 UTC, flushed through 1970-01-01 00:00:00 +0000 UTC) W0824 22:24:46.077109 filer_grpc_server_sub_meta.go:358 local subscriber loop-test@ resumed after 0s parked I0824 22:24:46.335044 filer_grpc_server_sub_meta.go:930 disconnect loop-test@ local subscriber clientId:-7 I0824 22:24:46.335361 filer_grpc_server_sub_meta.go:1471 - local listener loop-test@ clientId -7 clientEpoch 1 I0824 22:24:46.337339 filer_deletion.go:515 retry processor shutting down, 0 items remaining in queue I0824 22:24:46.337601 filer_deletion.go:306 deletion processor shutting down --- PASS: TestSubscribeLoop_EvictedUnflushedGapWaitsThenDelivers (0.63s) === RUN TestSubscribeLoop_NothingEvictedServesFromMemory I0824 22:24:46.361989 leveldb_store.go:48 filer store dir: /tmp/TestSubscribeLoop_NothingEvictedServesFromMemory3141773037/001 I0824 22:24:46.362423 file_util.go:29 Folder /tmp/TestSubscribeLoop_NothingEvictedServesFromMemory3141773037/001 Permission: -rwxr-xr-x I0824 22:24:46.377461 filer.go:201 create filer.store.id to -1359505782 E0824 22:24:46.399632 master_grpc_server.go:600 failed to get peer from ctx I0824 22:24:46.399944 filer_grpc_server_sub_meta.go:1443 + local listener loop-test@ clientId -7 clientEpoch 1 I0824 22:24:46.400056 filer_grpc_server_sub_meta.go:935 + loop-test@ local subscribe from {Time:1970-01-01 00:00:00 +0000 UTC Offset:-7 IsOffsetBased:false} clientId:-7 I0824 22:24:46.815381 filer_grpc_server_sub_meta.go:930 disconnect loop-test@ local subscriber clientId:-7 I0824 22:24:46.815787 filer_grpc_server_sub_meta.go:1471 - local listener loop-test@ clientId -7 clientEpoch 1 I0824 22:24:46.816634 filer_deletion.go:515 retry processor shutting down, 0 items remaining in queue I0824 22:24:46.816980 filer_deletion.go:306 deletion processor shutting down --- PASS: TestSubscribeLoop_NothingEvictedServesFromMemory (0.48s) === RUN TestSubscribeLoop_BacklogThenLiveExactlyOnce I0824 22:24:46.827066 leveldb_store.go:48 filer store dir: /tmp/TestSubscribeLoop_BacklogThenLiveExactlyOnce642863768/001 I0824 22:24:46.827254 file_util.go:29 Folder /tmp/TestSubscribeLoop_BacklogThenLiveExactlyOnce642863768/001 Permission: -rwxr-xr-x I0824 22:24:46.832854 filer.go:201 create filer.store.id to -36107819 E0824 22:24:46.868466 master_grpc_server.go:600 failed to get peer from ctx I0824 22:24:46.868620 filer_grpc_server_sub_meta.go:1443 + local listener loop-test@ clientId -7 clientEpoch 1 I0824 22:24:46.868683 filer_grpc_server_sub_meta.go:935 + loop-test@ local subscribe from {Time:1970-01-01 00:00:00 +0000 UTC Offset:-7 IsOffsetBased:false} clientId:-7 I0824 22:24:47.134428 filer_grpc_server_sub_meta.go:930 disconnect loop-test@ local subscriber clientId:-7 I0824 22:24:47.134571 filer_grpc_server_sub_meta.go:1471 - local listener loop-test@ clientId -7 clientEpoch 1 I0824 22:24:47.135109 filer_deletion.go:306 deletion processor shutting down I0824 22:24:47.135250 filer_deletion.go:515 retry processor shutting down, 0 items remaining in queue --- PASS: TestSubscribeLoop_BacklogThenLiveExactlyOnce (0.32s) === RUN TestSubscribeLoop_BoundedSubscriptionTerminates I0824 22:24:47.146298 leveldb_store.go:48 filer store dir: /tmp/TestSubscribeLoop_BoundedSubscriptionTerminates4012674599/001 I0824 22:24:47.146472 file_util.go:29 Folder /tmp/TestSubscribeLoop_BoundedSubscriptionTerminates4012674599/001 Permission: -rwxr-xr-x I0824 22:24:47.152599 filer.go:201 create filer.store.id to -1870746139 E0824 22:24:47.524399 master_grpc_server.go:600 failed to get peer from ctx I0824 22:24:47.524574 filer_grpc_server_sub_meta.go:1443 + local listener loop-test@ clientId -7 clientEpoch 1 I0824 22:24:47.524664 filer_grpc_server_sub_meta.go:935 + loop-test@ local subscribe from {Time:1970-01-01 00:00:00 +0000 UTC Offset:-7 IsOffsetBased:false} clientId:-7 I0824 22:24:47.526160 filer_grpc_server_sub_meta.go:930 disconnect loop-test@ local subscriber clientId:-7 I0824 22:24:47.526267 filer_grpc_server_sub_meta.go:1471 - local listener loop-test@ clientId -7 clientEpoch 1 I0824 22:24:47.534030 filer_deletion.go:515 retry processor shutting down, 0 items remaining in queue I0824 22:24:47.537266 filer_deletion.go:306 deletion processor shutting down --- PASS: TestSubscribeLoop_BoundedSubscriptionTerminates (0.41s) === RUN TestSubscribeLoop_BoundedEmptyFilerTerminates I0824 22:24:47.556810 leveldb_store.go:48 filer store dir: /tmp/TestSubscribeLoop_BoundedEmptyFilerTerminates3803748790/001 I0824 22:24:47.557331 file_util.go:29 Folder /tmp/TestSubscribeLoop_BoundedEmptyFilerTerminates3803748790/001 Permission: -rwxr-xr-x I0824 22:24:47.591152 filer.go:201 create filer.store.id to -2059246328 E0824 22:24:47.612543 master_grpc_server.go:600 failed to get peer from ctx I0824 22:24:47.612877 filer_grpc_server_sub_meta.go:1443 + local listener loop-test@ clientId -7 clientEpoch 1 I0824 22:24:47.613002 filer_grpc_server_sub_meta.go:935 + loop-test@ local subscribe from {Time:2026-08-24 21:24:47 +0000 UTC Offset:-7 IsOffsetBased:false} clientId:-7 I0824 22:24:47.614658 filer_grpc_server_sub_meta.go:930 disconnect loop-test@ local subscriber clientId:-7 I0824 22:24:47.614961 filer_grpc_server_sub_meta.go:1471 - local listener loop-test@ clientId -7 clientEpoch 1 I0824 22:24:47.615758 filer_deletion.go:515 retry processor shutting down, 0 items remaining in queue I0824 22:24:47.615932 filer_deletion.go:306 deletion processor shutting down --- PASS: TestSubscribeLoop_BoundedEmptyFilerTerminates (0.07s) === RUN TestSubscribeLoop_FlushProvenGapSkipsToRetained I0824 22:24:47.638301 leveldb_store.go:48 filer store dir: /tmp/TestSubscribeLoop_FlushProvenGapSkipsToRetained2330755195/001 I0824 22:24:47.638755 file_util.go:29 Folder /tmp/TestSubscribeLoop_FlushProvenGapSkipsToRetained2330755195/001 Permission: -rwxr-xr-x I0824 22:24:47.653970 filer.go:201 create filer.store.id to -1959658224 E0824 22:24:47.694002 master_grpc_server.go:600 failed to get peer from ctx I0824 22:24:47.694392 filer_grpc_server_sub_meta.go:1443 + local listener loop-test@ clientId -7 clientEpoch 1 I0824 22:24:47.694788 filer_grpc_server_sub_meta.go:935 + loop-test@ local subscribe from {Time:1970-01-01 00:00:00 +0000 UTC Offset:-7 IsOffsetBased:false} clientId:-7 I0824 22:24:47.954414 filer_grpc_server_sub_meta.go:930 disconnect loop-test@ local subscriber clientId:-7 I0824 22:24:47.954824 filer_grpc_server_sub_meta.go:1471 - local listener loop-test@ clientId -7 clientEpoch 1 I0824 22:24:47.955369 filer_deletion.go:306 deletion processor shutting down I0824 22:24:47.955337 filer_deletion.go:515 retry processor shutting down, 0 items remaining in queue --- PASS: TestSubscribeLoop_FlushProvenGapSkipsToRetained (0.34s) === RUN TestSubscribeLoop_GiveUpSkipsAndKeepsStreaming I0824 22:24:47.979871 leveldb_store.go:48 filer store dir: /tmp/TestSubscribeLoop_GiveUpSkipsAndKeepsStreaming2303881213/001 I0824 22:24:47.980319 file_util.go:29 Folder /tmp/TestSubscribeLoop_GiveUpSkipsAndKeepsStreaming2303881213/001 Permission: -rwxr-xr-x I0824 22:24:47.998768 filer.go:201 create filer.store.id to 1368966110 E0824 22:24:48.026174 master_grpc_server.go:600 failed to get peer from ctx I0824 22:24:48.026596 filer_grpc_server_sub_meta.go:1443 + local listener loop-test@ clientId -7 clientEpoch 1 I0824 22:24:48.026750 filer_grpc_server_sub_meta.go:935 + loop-test@ local subscribe from {Time:1970-01-01 00:00:00 +0000 UTC Offset:-7 IsOffsetBased:false} clientId:-7 W0824 22:24:48.237514 filer_grpc_server_sub_meta.go:347 local subscriber loop-test@ parked 0s at 1970-01-01 00:00:00 +0000 UTC: gap is not flushed yet (earliest memory 2026-08-24 21:28:47 +0000 UTC, flushed through 1970-01-01 00:00:00 +0000 UTC) E0824 22:24:48.294801 filer_grpc_server_sub_meta.go:370 local subscriber loop-test@ skipping the gap (1970-01-01 00:00:00 +0000 UTC..2026-08-24 21:26:47 +0000 UTC] after 0s parked: gap is not flushed yet (earliest memory 2026-08-24 21:28:47 +0000 UTC, flushed through 1970-01-01 00:00:00 +0000 UTC); events a peer flushes into that range later will not be delivered I0824 22:24:48.548823 filer_grpc_server_sub_meta.go:930 disconnect loop-test@ local subscriber clientId:-7 I0824 22:24:48.549170 filer_grpc_server_sub_meta.go:1471 - local listener loop-test@ clientId -7 clientEpoch 1 I0824 22:24:48.550026 filer_deletion.go:515 retry processor shutting down, 0 items remaining in queue I0824 22:24:48.550220 filer_deletion.go:306 deletion processor shutting down --- PASS: TestSubscribeLoop_GiveUpSkipsAndKeepsStreaming (0.59s) === RUN TestSubscribeLoop_ChunkModeMarkerMatchesClientReplay I0824 22:24:48.574344 leveldb_store.go:48 filer store dir: /tmp/TestSubscribeLoop_ChunkModeMarkerMatchesClientReplay1545120273/001 I0824 22:24:48.574868 file_util.go:29 Folder /tmp/TestSubscribeLoop_ChunkModeMarkerMatchesClientReplay1545120273/001 Permission: -rwxr-xr-x I0824 22:24:48.590437 filer.go:201 create filer.store.id to -1739191768 E0824 22:24:48.606002 master_grpc_server.go:600 failed to get peer from ctx I0824 22:24:48.606391 filer_grpc_server_sub_meta.go:1443 + local listener loop-test@ clientId -7 clientEpoch 1 I0824 22:24:48.606558 filer_grpc_server_sub_meta.go:935 + loop-test@ local subscribe from {Time:1970-01-01 00:00:00 +0000 UTC Offset:-7 IsOffsetBased:false} clientId:-7 I0824 22:24:48.915443 filer_grpc_server_sub_meta.go:930 disconnect loop-test@ local subscriber clientId:-7 I0824 22:24:48.916158 filer_grpc_server_sub_meta.go:1471 - local listener loop-test@ clientId -7 clientEpoch 1 I0824 22:24:48.916668 filer_deletion.go:515 retry processor shutting down, 0 items remaining in queue I0824 22:24:48.916704 filer_deletion.go:306 deletion processor shutting down --- PASS: TestSubscribeLoop_ChunkModeMarkerMatchesClientReplay (0.36s) === RUN TestSubscribeLoop_ChunkModeDeadVolumeAgreesWithClient I0824 22:24:48.926671 leveldb_store.go:48 filer store dir: /tmp/TestSubscribeLoop_ChunkModeDeadVolumeAgreesWithClient3852331462/001 I0824 22:24:48.926868 file_util.go:29 Folder /tmp/TestSubscribeLoop_ChunkModeDeadVolumeAgreesWithClient3852331462/001 Permission: -rwxr-xr-x I0824 22:24:48.932618 filer.go:201 create filer.store.id to -148447687 E0824 22:24:48.959327 master_grpc_server.go:600 failed to get peer from ctx I0824 22:24:48.959538 filer_grpc_server_sub_meta.go:1443 + local listener loop-test@ clientId -7 clientEpoch 1 I0824 22:24:48.959642 filer_grpc_server_sub_meta.go:935 + loop-test@ local subscribe from {Time:1970-01-01 00:00:00 +0000 UTC Offset:-7 IsOffsetBased:false} clientId:-7 I0824 22:24:48.978147 filer_pb_direct_read.go:228 skip log file filer=0000abcd ts=1787606760000000000 after 0 entries: read t,2: volume 42 not found I0824 22:24:49.214323 filer_grpc_server_sub_meta.go:930 disconnect loop-test@ local subscriber clientId:-7 I0824 22:24:49.214543 filer_grpc_server_sub_meta.go:1471 - local listener loop-test@ clientId -7 clientEpoch 1 I0824 22:24:49.215278 filer_deletion.go:306 deletion processor shutting down I0824 22:24:49.215575 filer_deletion.go:515 retry processor shutting down, 0 items remaining in queue --- PASS: TestSubscribeLoop_ChunkModeDeadVolumeAgreesWithClient (0.30s) === RUN TestMasterIsKnownPingTarget I0824 22:24:49.234376 node.go:471 test adds child dc1 I0824 22:24:49.234505 node.go:471 test:dc1 adds child rack1 I0824 22:24:49.234578 node.go:471 test:dc1:rack1 adds child vol1 === RUN TestMasterIsKnownPingTarget/known_filer === RUN TestMasterIsKnownPingTarget/known_volume_server === RUN TestMasterIsKnownPingTarget/known_volume_server_http_addr === RUN TestMasterIsKnownPingTarget/known_self_master === RUN TestMasterIsKnownPingTarget/unknown_localhost_low_port === RUN TestMasterIsKnownPingTarget/unknown_localhost_high_port === RUN TestMasterIsKnownPingTarget/unrelated_host === RUN TestMasterIsKnownPingTarget/unknown_target_type === RUN TestMasterIsKnownPingTarget/filer_address_checked_as_volume_server --- PASS: TestMasterIsKnownPingTarget (0.02s) --- PASS: TestMasterIsKnownPingTarget/known_filer (0.00s) --- PASS: TestMasterIsKnownPingTarget/known_volume_server (0.00s) --- PASS: TestMasterIsKnownPingTarget/known_volume_server_http_addr (0.00s) --- PASS: TestMasterIsKnownPingTarget/known_self_master (0.00s) --- PASS: TestMasterIsKnownPingTarget/unknown_localhost_low_port (0.00s) --- PASS: TestMasterIsKnownPingTarget/unknown_localhost_high_port (0.00s) --- PASS: TestMasterIsKnownPingTarget/unrelated_host (0.00s) --- PASS: TestMasterIsKnownPingTarget/unknown_target_type (0.00s) --- PASS: TestMasterIsKnownPingTarget/filer_address_checked_as_volume_server (0.00s) === RUN TestMasterIsKnownPingTargetFollowerEmptyTopology --- PASS: TestMasterIsKnownPingTargetFollowerEmptyTopology (0.00s) === RUN TestVolumeServerIsKnownPingTarget === RUN TestVolumeServerIsKnownPingTarget/seed_master === RUN TestVolumeServerIsKnownPingTarget/current_master === RUN TestVolumeServerIsKnownPingTarget/other_volume_server === RUN TestVolumeServerIsKnownPingTarget/random_filer === RUN TestVolumeServerIsKnownPingTarget/unknown_low_port === RUN TestVolumeServerIsKnownPingTarget/unknown_high_port === RUN TestVolumeServerIsKnownPingTarget/unrelated_host --- PASS: TestVolumeServerIsKnownPingTarget (0.00s) --- PASS: TestVolumeServerIsKnownPingTarget/seed_master (0.00s) --- PASS: TestVolumeServerIsKnownPingTarget/current_master (0.00s) --- PASS: TestVolumeServerIsKnownPingTarget/other_volume_server (0.00s) --- PASS: TestVolumeServerIsKnownPingTarget/random_filer (0.00s) --- PASS: TestVolumeServerIsKnownPingTarget/unknown_low_port (0.00s) --- PASS: TestVolumeServerIsKnownPingTarget/unknown_high_port (0.00s) --- PASS: TestVolumeServerIsKnownPingTarget/unrelated_host (0.00s) === RUN TestAssignShedsLoadWhenGrowthInFlight I0824 22:24:49.236933 node.go:471 test adds child dc1 I0824 22:24:49.237005 node.go:471 test:dc1 adds child rack1 I0824 22:24:49.237056 node.go:471 test:dc1:rack1 adds child dn1 I0824 22:24:49.237123 node.go:471 test:dc1:rack1:dn1 adds child --- PASS: TestAssignShedsLoadWhenGrowthInFlight (0.00s) === RUN TestAssignInitiatorWaitsForItsOwnGrowth I0824 22:24:49.238169 node.go:471 test adds child dc1 I0824 22:24:49.238221 node.go:471 test:dc1 adds child rack1 I0824 22:24:49.238271 node.go:471 test:dc1:rack1 adds child dn1 I0824 22:24:49.238332 node.go:471 test:dc1:rack1:dn1 adds child I0824 22:24:49.444464 volume_layout.go:815 Volume 1 becomes unwritable I0824 22:24:49.444715 volume_layout.go:391 Volume 1 reaches full capacity (effective=1048576, limit=1048576). --- PASS: TestAssignInitiatorWaitsForItsOwnGrowth (0.21s) === RUN TestAssignInitiatorShedsWhenGrowthConcludesUnfulfilled I0824 22:24:49.445420 node.go:471 test adds child dc1 I0824 22:24:49.445502 node.go:471 test:dc1 adds child rack1 I0824 22:24:49.445575 node.go:471 test:dc1:rack1 adds child dn1 I0824 22:24:49.445637 node.go:471 test:dc1:rack1:dn1 adds child --- PASS: TestAssignInitiatorShedsWhenGrowthConcludesUnfulfilled (0.21s) === RUN TestAssignAbortsOnCancel I0824 22:24:49.656156 node.go:471 test adds child dc1 I0824 22:24:49.656414 node.go:471 test:dc1 adds child rack1 I0824 22:24:49.656543 node.go:471 test:dc1:rack1 adds child dn1 I0824 22:24:49.656664 node.go:471 test:dc1:rack1:dn1 adds child --- PASS: TestAssignAbortsOnCancel (0.31s) === RUN TestAssignFailsFastWhenOutOfSpace I0824 22:24:49.964850 master_grpc_server_assign.go:199 assign count:1 replication:"000" {"replication":{},"ttl":{"Count":0,"Unit":0},"version":3}: failed to find writable volumes for collection: replication:000 ttl: error: No writable volumes and no free volumes left for {"replication":{},"ttl":{"Count":0,"Unit":0},"version":3} --- PASS: TestAssignFailsFastWhenOutOfSpace (0.00s) === RUN TestChangedVolumesBringTheMasterCurrent I0824 22:24:49.967130 node.go:471 test adds child dc1 I0824 22:24:49.967269 node.go:471 test:dc1 adds child rack1 I0824 22:24:49.967346 node.go:471 test:dc1:rack1 adds child 127.0.0.1:8080 I0824 22:24:49.967406 node.go:471 test:dc1:rack1:127.0.0.1:8080 adds child I0824 22:24:49.967662 node.go:471 test adds child dc1 I0824 22:24:49.967732 node.go:471 test:dc1 adds child rack1 I0824 22:24:49.967790 node.go:471 test:dc1:rack1 adds child 127.0.0.1:8080 I0824 22:24:49.967843 node.go:471 test:dc1:rack1:127.0.0.1:8080 adds child --- PASS: TestChangedVolumesBringTheMasterCurrent (0.00s) === RUN TestChangedVolumesDoNotRemoveUnmentionedVolumes I0824 22:24:49.968160 node.go:471 test adds child dc1 I0824 22:24:49.968200 node.go:471 test:dc1 adds child rack1 I0824 22:24:49.968256 node.go:471 test:dc1:rack1 adds child 127.0.0.1:8080 I0824 22:24:49.968328 node.go:471 test:dc1:rack1:127.0.0.1:8080 adds child --- PASS: TestChangedVolumesDoNotRemoveUnmentionedVolumes (0.00s) === RUN TestChangedVolumesRegisterUnknownVolumes I0824 22:24:49.968673 node.go:471 test adds child dc1 I0824 22:24:49.968706 node.go:471 test:dc1 adds child rack1 I0824 22:24:49.968765 node.go:471 test:dc1:rack1 adds child 127.0.0.1:8080 I0824 22:24:49.968824 node.go:471 test:dc1:rack1:127.0.0.1:8080 adds child --- PASS: TestChangedVolumesRegisterUnknownVolumes (0.00s) === RUN TestChangedVolumesAnnounceOnlyArrivals I0824 22:24:49.969085 node.go:471 test adds child dc1 I0824 22:24:49.969142 node.go:471 test:dc1 adds child rack1 I0824 22:24:49.969190 node.go:471 test:dc1:rack1 adds child 127.0.0.1:8080 I0824 22:24:49.969236 node.go:471 test:dc1:rack1:127.0.0.1:8080 adds child --- PASS: TestChangedVolumesAnnounceOnlyArrivals (0.00s) === RUN TestRepairedLookupEntryIsAnnounced === RUN TestRepairedLookupEntryIsAnnounced/ViaChanges I0824 22:24:49.969644 node.go:471 test adds child dc1 I0824 22:24:49.969711 node.go:471 test:dc1 adds child rack1 I0824 22:24:49.969755 node.go:471 test:dc1:rack1 adds child 127.0.0.1:8080 I0824 22:24:49.969811 node.go:471 test:dc1:rack1:127.0.0.1:8080 adds child I0824 22:24:49.969893 volume_layout.go:866 Volume 1 has 0 replica, less than required 1 I0824 22:24:49.969931 volume_layout.go:815 Volume 1 becomes unwritable === RUN TestRepairedLookupEntryIsAnnounced/ViaFullList I0824 22:24:49.970150 node.go:471 test adds child dc1 I0824 22:24:49.970185 node.go:471 test:dc1 adds child rack1 I0824 22:24:49.970224 node.go:471 test:dc1:rack1 adds child 127.0.0.1:8080 I0824 22:24:49.970265 node.go:471 test:dc1:rack1:127.0.0.1:8080 adds child I0824 22:24:49.970339 volume_layout.go:866 Volume 1 has 0 replica, less than required 1 I0824 22:24:49.970362 volume_layout.go:815 Volume 1 becomes unwritable --- PASS: TestRepairedLookupEntryIsAnnounced (0.00s) --- PASS: TestRepairedLookupEntryIsAnnounced/ViaChanges (0.00s) --- PASS: TestRepairedLookupEntryIsAnnounced/ViaFullList (0.00s) === RUN TestDeletedCollectionReleasesTheLookupIndex I0824 22:24:49.970658 node.go:471 test adds child dc1 I0824 22:24:49.970690 node.go:471 test:dc1 adds child rack1 I0824 22:24:49.970730 node.go:471 test:dc1:rack1 adds child 127.0.0.1:8080 I0824 22:24:49.970773 node.go:471 test:dc1:rack1:127.0.0.1:8080 adds child I0824 22:24:49.970945 topology.go:534 removing volume info: Id:1, Size:0, ReplicaPlacement:000, Collection:c, Version:3, Ttl:, FileCount:0, DeleteCount:0, DeletedByteCount:0, ReadOnly:false, ModifiedAtSecond:0 from 127.0.0.1:8080 I0824 22:24:49.971119 topology.go:534 removing volume info: Id:2, Size:0, ReplicaPlacement:000, Collection:c, Version:3, Ttl:, FileCount:0, DeleteCount:0, DeletedByteCount:0, ReadOnly:false, ModifiedAtSecond:0 from 127.0.0.1:8080 --- PASS: TestDeletedCollectionReleasesTheLookupIndex (0.00s) === RUN TestStaleFullListDoesNotEraseAFreshGrow I0824 22:24:49.971351 node.go:471 test adds child dc1 I0824 22:24:49.971393 node.go:471 test:dc1 adds child rack1 I0824 22:24:49.971437 node.go:471 test:dc1:rack1 adds child 127.0.0.1:8080 I0824 22:24:49.971477 node.go:471 test:dc1:rack1:127.0.0.1:8080 adds child I0824 22:24:49.971659 data_node.go:114 Deleting volume id: 2 I0824 22:24:49.971735 topology.go:534 removing volume info: Id:2, Size:8, ReplicaPlacement:000, Collection:c, Version:3, Ttl:, FileCount:1, DeleteCount:0, DeletedByteCount:0, ReadOnly:false, ModifiedAtSecond:0 from 127.0.0.1:8080 I0824 22:24:49.971779 volume_layout.go:815 Volume 2 becomes unwritable I0824 22:24:49.971804 volume_layout.go:335 volume 2 does not have enough copies --- PASS: TestStaleFullListDoesNotEraseAFreshGrow (0.00s) === RUN TestRegistrationRacingCollectionDeleteDoesNotLeak I0824 22:24:49.971982 node.go:471 test adds child dc1 I0824 22:24:49.972022 node.go:471 test:dc1 adds child rack1 I0824 22:24:49.972062 node.go:471 test:dc1:rack1 adds child 127.0.0.1:8080 I0824 22:24:49.972121 node.go:471 test:dc1:rack1:127.0.0.1:8080 adds child I0824 22:24:49.972235 topology.go:534 removing volume info: Id:1, Size:0, ReplicaPlacement:000, Collection:c, Version:3, Ttl:, FileCount:0, DeleteCount:0, DeletedByteCount:0, ReadOnly:false, ModifiedAtSecond:0 from 127.0.0.1:8080 --- PASS: TestRegistrationRacingCollectionDeleteDoesNotLeak (0.00s) === RUN TestDigestCheckIgnoresServersThatReportNone I0824 22:24:49.972442 node.go:471 test adds child dc1 I0824 22:24:49.972477 node.go:471 test:dc1 adds child rack1 I0824 22:24:49.972516 node.go:471 test:dc1:rack1 adds child 127.0.0.1:8080 I0824 22:24:49.972592 node.go:471 test:dc1:rack1:127.0.0.1:8080 adds child --- PASS: TestDigestCheckIgnoresServersThatReportNone (0.00s) === RUN TestDigestCheckAcceptsAMatchingReport I0824 22:24:49.972801 node.go:471 test adds child dc1 I0824 22:24:49.972840 node.go:471 test:dc1 adds child rack1 I0824 22:24:49.972891 node.go:471 test:dc1:rack1 adds child 127.0.0.1:8080 I0824 22:24:49.972949 node.go:471 test:dc1:rack1:127.0.0.1:8080 adds child --- PASS: TestDigestCheckAcceptsAMatchingReport (0.00s) === RUN TestDigestCheckDoesNotReaskAfterAFullList I0824 22:24:49.973242 node.go:471 test adds child dc1 I0824 22:24:49.973279 node.go:471 test:dc1 adds child rack1 I0824 22:24:49.973321 node.go:471 test:dc1:rack1 adds child 127.0.0.1:8080 I0824 22:24:49.973382 node.go:471 test:dc1:rack1:127.0.0.1:8080 adds child W0824 22:24:49.973480 master_grpc_server.go:365 volume server 127.0.0.1:8080 still disagrees after a full volume list: reported digest 15521999559454932142, master holds 15521999559454932143 --- PASS: TestDigestCheckDoesNotReaskAfterAFullList (0.00s) === RUN TestDigestCheckAsksForTheListWhenADeltaDisagrees I0824 22:24:49.973694 node.go:471 test adds child dc1 I0824 22:24:49.973728 node.go:471 test:dc1 adds child rack1 I0824 22:24:49.973766 node.go:471 test:dc1:rack1 adds child 127.0.0.1:8080 I0824 22:24:49.973824 node.go:471 test:dc1:rack1:127.0.0.1:8080 adds child I0824 22:24:49.973919 master_grpc_server.go:370 volume server 127.0.0.1:8080: reported digest 15521999559454932142, master holds 15521999559454932143, requesting the full volume list --- PASS: TestDigestCheckAsksForTheListWhenADeltaDisagrees (0.00s) === RUN TestDigestCheckKeepsDuplicateNodesOnFullLists I0824 22:24:49.974099 node.go:471 test adds child dc1 I0824 22:24:49.974248 node.go:471 test:dc1 adds child rack1 I0824 22:24:49.974307 node.go:471 test:dc1:rack1 adds child 127.0.0.1:8080 I0824 22:24:49.974351 node.go:471 test:dc1:rack1:127.0.0.1:8080 adds child --- PASS: TestDigestCheckKeepsDuplicateNodesOnFullLists (0.00s) === RUN TestDigestCheckAsksForTheListWhenTheLookupIndexDrifts I0824 22:24:49.974715 node.go:471 test adds child dc1 I0824 22:24:49.974776 node.go:471 test:dc1 adds child rack1 I0824 22:24:49.974821 node.go:471 test:dc1:rack1 adds child 127.0.0.1:8080 I0824 22:24:49.974870 node.go:471 test:dc1:rack1:127.0.0.1:8080 adds child I0824 22:24:49.974998 volume_layout.go:866 Volume 1 has 0 replica, less than required 1 I0824 22:24:49.975035 volume_layout.go:815 Volume 1 becomes unwritable I0824 22:24:49.975082 master_grpc_server.go:370 volume server 127.0.0.1:8080: lookup index disagrees with the volumes held, requesting the full volume list --- PASS: TestDigestCheckAsksForTheListWhenTheLookupIndexDrifts (0.00s) === RUN TestHeartbeatResponsesCarryTheVolumeOptions --- PASS: TestHeartbeatResponsesCarryTheVolumeOptions (0.00s) === RUN TestRaftMembershipRPCs_RejectUnwhitelisted I0824 22:24:49.975458 master_grpc_server_raft.go:45 gRPC raft admin auth failed: 10.13.37.66 is not whitelisted (remote: 10.13.37.66:40000) I0824 22:24:49.975517 master_grpc_server_raft.go:45 gRPC raft admin auth failed: 10.13.37.66 is not whitelisted (remote: 10.13.37.66:40000) I0824 22:24:49.975555 master_grpc_server_raft.go:45 gRPC raft admin auth failed: 10.13.37.66 is not whitelisted (remote: 10.13.37.66:40000) --- PASS: TestRaftMembershipRPCs_RejectUnwhitelisted (0.00s) === RUN TestRaftMembershipRPCs_AllowWhitelisted --- PASS: TestRaftMembershipRPCs_AllowWhitelisted (0.00s) === RUN TestRaftMembershipRPCs_NoWhitelistFailsOpen --- PASS: TestRaftMembershipRPCs_NoWhitelistFailsOpen (0.00s) === RUN TestRaftLeadershipTransfer_NoRaft --- PASS: TestRaftLeadershipTransfer_NoRaft (0.00s) === RUN TestRaftListClusterServers_NoRaft --- PASS: TestRaftListClusterServers_NoRaft (0.00s) === RUN TestRaftAddServer_NoRaft --- PASS: TestRaftAddServer_NoRaft (0.00s) === RUN TestRaftRemoveServer_NoRaft --- PASS: TestRaftRemoveServer_NoRaft (0.00s) === RUN TestStatisticsLogicalSizes I0824 22:24:49.976488 node.go:471 test adds child dc1 I0824 22:24:49.976531 node.go:471 test:dc1 adds child rack1 I0824 22:24:49.976576 node.go:471 test:dc1:rack1 adds child 127.0.0.1:34534 I0824 22:24:49.982462 node.go:471 test:dc1:rack1:127.0.0.1:34534 adds child I0824 22:24:49.982592 node.go:471 test:dc1:rack1 adds child 127.0.0.2:34534 I0824 22:24:49.982636 node.go:471 test:dc1:rack1:127.0.0.2:34534 adds child I0824 22:24:49.982750 node.go:471 test:dc1:rack1 adds child 127.0.0.3:34534 I0824 22:24:49.982788 node.go:471 test:dc1:rack1:127.0.0.3:34534 adds child I0824 22:24:49.982878 node.go:471 test:dc1:rack1 adds child 127.0.0.4:34534 I0824 22:24:49.982924 node.go:471 test:dc1:rack1:127.0.0.4:34534 adds child --- PASS: TestStatisticsLogicalSizes (0.01s) === RUN TestStatisticsReplicaCopyCount I0824 22:24:49.983356 node.go:471 test adds child dc1 I0824 22:24:49.983402 node.go:471 test:dc1 adds child rack1 I0824 22:24:49.983449 node.go:471 test:dc1:rack1 adds child 127.0.0.1:34534 I0824 22:24:49.983526 node.go:471 test:dc1:rack1:127.0.0.1:34534 adds child I0824 22:24:49.983623 node.go:471 test:dc1:rack1 adds child 127.0.0.2:34534 I0824 22:24:49.983666 node.go:471 test:dc1:rack1:127.0.0.2:34534 adds child I0824 22:24:49.983738 node.go:471 test:dc1:rack1 adds child 127.0.0.3:34534 I0824 22:24:49.983772 node.go:471 test:dc1:rack1:127.0.0.3:34534 adds child I0824 22:24:49.983861 node.go:471 test:dc1:rack1 adds child 127.0.0.4:34534 I0824 22:24:49.983903 node.go:471 test:dc1:rack1:127.0.0.4:34534 adds child === RUN TestStatisticsReplicaCopyCount/requested === RUN TestStatisticsReplicaCopyCount/empty_falls_back_to_the_master_default === RUN TestStatisticsReplicaCopyCount/unparsable_falls_back_to_the_master_default --- PASS: TestStatisticsReplicaCopyCount (0.00s) --- PASS: TestStatisticsReplicaCopyCount/requested (0.00s) --- PASS: TestStatisticsReplicaCopyCount/empty_falls_back_to_the_master_default (0.00s) --- PASS: TestStatisticsReplicaCopyCount/unparsable_falls_back_to_the_master_default (0.00s) === RUN TestInitialLockRingUpdateReturnsLastBroadcastForFilers I0824 22:24:49.984859 lock_ring_manager.go:139 LockRing: broadcasting ring update for group "group-a" version 1787610289984810005: [filer1:8888 filer2:8888] --- PASS: TestInitialLockRingUpdateReturnsLastBroadcastForFilers (0.00s) === RUN TestInitialLockRingUpdateSkipsNonFilers I0824 22:24:49.985186 lock_ring_manager.go:139 LockRing: broadcasting ring update for group "group-a" version 1787610289985157425: [filer1:8888] --- PASS: TestInitialLockRingUpdateSkipsNonFilers (0.00s) === RUN TestBroadcastVolumeLocationsToClients --- PASS: TestBroadcastVolumeLocationsToClients (0.00s) === RUN TestVolumeMovedBetweenDisksIsNotBroadcastAsRemoved I0824 22:24:49.986161 node.go:471 test adds child dc1 I0824 22:24:49.986220 node.go:471 test:dc1 adds child rack1 I0824 22:24:49.986277 node.go:471 test:dc1:rack1 adds child 127.0.0.1:8080 I0824 22:24:49.986342 node.go:471 test:dc1:rack1:127.0.0.1:8080 adds child I0824 22:24:49.986421 node.go:471 test:dc1:rack1:127.0.0.1:8080 adds child ssd I0824 22:24:49.986534 data_node.go:114 Deleting volume id: 1 I0824 22:24:49.986611 topology.go:534 removing volume info: Id:1, Size:1024, ReplicaPlacement:000, Collection:c, Version:3, Ttl:, FileCount:0, DeleteCount:0, DeletedByteCount:0, ReadOnly:false, ModifiedAtSecond:0 from 127.0.0.1:8080 I0824 22:24:49.986714 volume_layout.go:815 Volume 1 becomes unwritable I0824 22:24:49.986744 volume_layout.go:335 volume 1 does not have enough copies --- PASS: TestVolumeMovedBetweenDisksIsNotBroadcastAsRemoved (0.00s) === RUN TestVolumeGoneFromTheNodeIsBroadcastAsRemoved I0824 22:24:49.987003 node.go:471 test adds child dc1 I0824 22:24:49.987041 node.go:471 test:dc1 adds child rack1 I0824 22:24:49.987083 node.go:471 test:dc1:rack1 adds child 127.0.0.1:8080 I0824 22:24:49.987133 node.go:471 test:dc1:rack1:127.0.0.1:8080 adds child ssd I0824 22:24:49.987176 node.go:471 test:dc1:rack1:127.0.0.1:8080 adds child I0824 22:24:49.987275 data_node.go:114 Deleting volume id: 1 I0824 22:24:49.987307 topology.go:534 removing volume info: Id:1, Size:1024, ReplicaPlacement:000, Collection:c, Version:3, Ttl:, FileCount:0, DeleteCount:0, DeletedByteCount:0, ReadOnly:false, ModifiedAtSecond:0 from 127.0.0.1:8080 I0824 22:24:49.987373 volume_layout.go:815 Volume 1 becomes unwritable I0824 22:24:49.987395 volume_layout.go:335 volume 1 does not have enough copies --- PASS: TestVolumeGoneFromTheNodeIsBroadcastAsRemoved (0.00s) === RUN TestVolumeRemountedOnAnotherDiskIsNotBroadcastAsRemoved I0824 22:24:49.987623 node.go:471 test adds child dc1 I0824 22:24:49.987657 node.go:471 test:dc1 adds child rack1 I0824 22:24:49.987699 node.go:471 test:dc1:rack1 adds child 127.0.0.1:8080 I0824 22:24:49.987739 node.go:471 test:dc1:rack1:127.0.0.1:8080 adds child I0824 22:24:49.987772 node.go:471 test:dc1:rack1:127.0.0.1:8080 adds child ssd I0824 22:24:49.987890 topology.go:534 removing volume info: Id:1, Size:0, ReplicaPlacement:000, Collection:c, Version:0, Ttl:, FileCount:0, DeleteCount:0, DeletedByteCount:0, ReadOnly:false, ModifiedAtSecond:0 from 127.0.0.1:8080 I0824 22:24:49.987980 volume_layout.go:815 Volume 1 becomes unwritable I0824 22:24:49.988003 volume_layout.go:335 volume 1 does not have enough copies --- PASS: TestVolumeRemountedOnAnotherDiskIsNotBroadcastAsRemoved (0.00s) === RUN TestVolumeUnmountedViaDeltaIsBroadcastAsRemoved I0824 22:24:49.988194 node.go:471 test adds child dc1 I0824 22:24:49.988236 node.go:471 test:dc1 adds child rack1 I0824 22:24:49.988278 node.go:471 test:dc1:rack1 adds child 127.0.0.1:8080 I0824 22:24:49.988325 node.go:471 test:dc1:rack1:127.0.0.1:8080 adds child I0824 22:24:49.988358 node.go:471 test:dc1:rack1:127.0.0.1:8080 adds child ssd I0824 22:24:49.988447 topology.go:534 removing volume info: Id:1, Size:0, ReplicaPlacement:000, Collection:c, Version:0, Ttl:, FileCount:0, DeleteCount:0, DeletedByteCount:0, ReadOnly:false, ModifiedAtSecond:0 from 127.0.0.1:8080 I0824 22:24:49.988512 volume_layout.go:815 Volume 1 becomes unwritable I0824 22:24:49.988532 volume_layout.go:335 volume 1 does not have enough copies --- PASS: TestVolumeUnmountedViaDeltaIsBroadcastAsRemoved (0.00s) === RUN TestVolumeReplacedByEcShardsIsBroadcastAsRemoved I0824 22:24:49.988700 node.go:471 test adds child dc1 I0824 22:24:49.988729 node.go:471 test:dc1 adds child rack1 I0824 22:24:49.988764 node.go:471 test:dc1:rack1 adds child 127.0.0.1:8080 I0824 22:24:49.988799 node.go:471 test:dc1:rack1:127.0.0.1:8080 adds child I0824 22:24:49.988860 node.go:471 test:dc1:rack1:127.0.0.1:8080 adds child ssd I0824 22:24:49.989008 data_node.go:114 Deleting volume id: 1 I0824 22:24:49.989079 topology.go:534 removing volume info: Id:1, Size:1024, ReplicaPlacement:000, Collection:c, Version:3, Ttl:, FileCount:0, DeleteCount:0, DeletedByteCount:0, ReadOnly:false, ModifiedAtSecond:0 from 127.0.0.1:8080 I0824 22:24:49.989123 volume_layout.go:815 Volume 1 becomes unwritable I0824 22:24:49.989163 volume_layout.go:335 volume 1 does not have enough copies --- PASS: TestVolumeReplacedByEcShardsIsBroadcastAsRemoved (0.00s) === RUN TestRaftServerID === RUN TestRaftServerID/without_grpc_suffix === RUN TestRaftServerID/with_grpc_suffix --- PASS: TestRaftServerID (0.00s) --- PASS: TestRaftServerID/without_grpc_suffix (0.00s) --- PASS: TestRaftServerID/with_grpc_suffix (0.00s) === RUN TestAddPeersConfigurationUsesCanonicalIDs --- PASS: TestAddPeersConfigurationUsesCanonicalIDs (0.00s) === RUN TestVolumeServerAdminAuthCoverage --- PASS: TestVolumeServerAdminAuthCoverage (0.24s) === RUN TestBatchDelete_DeniesNonWhitelistedPeer I0824 22:24:50.235708 volume_grpc_admin.go:64 gRPC admin auth failed: 127.0.0.1 is not whitelisted (remote: 127.0.0.1:12345) --- PASS: TestBatchDelete_DeniesNonWhitelistedPeer (0.00s) === RUN TestLookupRaftLeaderMasterDoesNotUpdateCurrentMaster --- PASS: TestLookupRaftLeaderMasterDoesNotUpdateCurrentMaster (0.02s) === RUN TestReceiveFile_RejectsTraversalExt E0824 22:24:50.258217 volume_grpc_copy.go:698 ReceiveFile: invalid file extension "/../../pwned.ec00" --- PASS: TestReceiveFile_RejectsTraversalExt (0.02s) === RUN TestReceiveFile_RejectsTraversalCollection E0824 22:24:50.279795 volume_grpc_copy.go:704 ReceiveFile: invalid collection "../../pwned" --- PASS: TestReceiveFile_RejectsTraversalCollection (0.01s) === RUN TestReceiveFile_AcceptsNormalExt --- PASS: TestReceiveFile_AcceptsNormalExt (0.00s) === RUN TestCopyFile_RejectsTraversalExt --- PASS: TestCopyFile_RejectsTraversalExt (0.00s) === RUN TestVolumeCopy_KeepsExistingReplicaWhenSourceUnreachable I0824 22:24:50.293983 store.go:329 In dir /tmp/TestVolumeCopy_KeepsExistingReplicaWhenSourceUnreachable848026423/001 (disk ID 0) adds volume:42 collection: replicaPlacement:000 ttl: I0824 22:24:50.303626 store.go:334 add volume 42 on disk ID 0 --- PASS: TestVolumeCopy_KeepsExistingReplicaWhenSourceUnreachable (0.02s) === RUN TestWriteToFile_RemovesPartialFileOnStreamError --- PASS: TestWriteToFile_RemovesPartialFileOnStreamError (0.00s) === RUN TestWriteToFile_RemovesEmptyFileOnImmediateStreamError --- PASS: TestWriteToFile_RemovesEmptyFileOnImmediateStreamError (0.00s) === RUN TestWriteToFile_PreservesAppendModeOnError --- PASS: TestWriteToFile_PreservesAppendModeOnError (0.00s) === RUN TestWriteToFile_SucceedsOnCleanStream --- PASS: TestWriteToFile_SucceedsOnCleanStream (0.00s) === RUN TestWriteToFile_PreservesDestinationWhenOptionalSourceMissing --- PASS: TestWriteToFile_PreservesDestinationWhenOptionalSourceMissing (0.00s) === RUN TestWriteToFile_CommitsOptionalCopyWhenSourcePresent --- PASS: TestWriteToFile_CommitsOptionalCopyWhenSourcePresent (0.00s) === RUN TestFullTeardownFencedByGeneration === RUN TestFullTeardownFencedByGeneration/older_disk_wiped I0824 22:24:50.339404 store_ec.go:213 MountEcShards 55.0 on disk ID 0 I0824 22:24:50.340229 store_ec.go:213 MountEcShards 55.1 on disk ID 0 I0824 22:24:50.340490 volume_grpc_erasure_coding.go:469 ec volume ec-fence_55 full teardown fenced at generation 200 === RUN TestFullTeardownFencedByGeneration/newer_disk_preserved I0824 22:24:50.351255 store_ec.go:213 MountEcShards 55.0 on disk ID 0 I0824 22:24:50.351541 store_ec.go:213 MountEcShards 55.1 on disk ID 0 I0824 22:24:50.351602 volume_grpc_erasure_coding.go:469 ec volume ec-fence_55 full teardown fenced at generation 100 === RUN TestFullTeardownFencedByGeneration/zero_gen_preserved I0824 22:24:50.359574 store_ec.go:213 MountEcShards 55.0 on disk ID 0 I0824 22:24:50.360322 store_ec.go:213 MountEcShards 55.1 on disk ID 0 I0824 22:24:50.360461 volume_grpc_erasure_coding.go:469 ec volume ec-fence_55 full teardown fenced at generation 200 === RUN TestFullTeardownFencedByGeneration/zero_request_blanket_wipe I0824 22:24:50.370775 store_ec.go:213 MountEcShards 55.0 on disk ID 0 I0824 22:24:50.371681 store_ec.go:213 MountEcShards 55.1 on disk ID 0 I0824 22:24:50.371918 volume_grpc_erasure_coding.go:453 ec volume ec-fence_55 full teardown --- PASS: TestFullTeardownFencedByGeneration (0.05s) --- PASS: TestFullTeardownFencedByGeneration/older_disk_wiped (0.01s) --- PASS: TestFullTeardownFencedByGeneration/newer_disk_preserved (0.01s) --- PASS: TestFullTeardownFencedByGeneration/zero_gen_preserved (0.01s) --- PASS: TestFullTeardownFencedByGeneration/zero_request_blanket_wipe (0.01s) === RUN TestUnmountEcShardsFencedByGeneration I0824 22:24:50.384264 store_ec.go:213 MountEcShards 56.0 on disk ID 0 I0824 22:24:50.385857 store_ec.go:213 MountEcShards 56.1 on disk ID 0 I0824 22:24:50.386284 volume_grpc_erasure_coding.go:1131 VolumeEcShardsInfo: volume 56 I0824 22:24:50.386834 store_ec.go:303 UnmountEcShards 56.1 disk_id:0 I0824 22:24:50.387097 volume_grpc_erasure_coding.go:1131 VolumeEcShardsInfo: volume 56 --- PASS: TestUnmountEcShardsFencedByGeneration (0.01s) === RUN TestReadEcGenerationTsNs W0824 22:24:50.404299 volume_info.go:54 unmarshal error: proto: syntax error (line 1:1): invalid value not-a-valid-vif oldFormat: failed to unmarshal old version volume info: proto: syntax error (line 1:1): invalid value not-a-valid-vif --- PASS: TestReadEcGenerationTsNs (0.02s) === RUN TestEcShardDeleteKeepsSharedIndexWhileSiblingHasShards I0824 22:24:50.414117 volume_grpc_erasure_coding.go:486 ec volume ec-shared-index_77 shard delete [0 5] I0824 22:24:50.416742 volume_grpc_erasure_coding.go:486 ec volume ec-shared-index_77 shard delete [7 12] --- PASS: TestEcShardDeleteKeepsSharedIndexWhileSiblingHasShards (0.01s) === RUN TestEcShardDeleteKeepsSharedIdxSidecarWhileSiblingHasShards I0824 22:24:50.430204 volume_grpc_erasure_coding.go:486 ec volume ec-shared-idx-sidecar_78 shard delete [0] I0824 22:24:50.432418 volume_grpc_erasure_coding.go:486 ec volume ec-shared-idx-sidecar_78 shard delete [7] --- PASS: TestEcShardDeleteKeepsSharedIdxSidecarWhileSiblingHasShards (0.01s) === RUN TestFetchEcIndexFromPeers_CopiesIndexOverGrpc I0824 22:24:50.468097 volume_grpc_erasure_coding_recover.go:164 ec volume 6190: fetched missing .ecx from 127.0.0.1:32211 into /tmp/TestFetchEcIndexFromPeers_CopiesIndexOverGrpc1228188192/002/dst --- PASS: TestFetchEcIndexFromPeers_CopiesIndexOverGrpc (0.04s) === RUN TestFetchEcIndexFromPeers_SkipsPeerWithoutIndex --- PASS: TestFetchEcIndexFromPeers_SkipsPeerWithoutIndex (0.03s) === RUN TestVolumeEcShardsMount_RecoverMissingIndex I0824 22:24:50.512977 volume_grpc_erasure_coding.go:761 VolumeEcShardsMount: recover_missing_index:true I0824 22:24:50.546554 volume_grpc_erasure_coding_recover.go:164 ec volume 6190: fetched missing .ecx from 127.0.0.1:1 into /tmp/TestVolumeEcShardsMount_RecoverMissingIndex1226941711/002/dst W0824 22:24:50.548721 volume_info.go:54 unmarshal error: proto: syntax error (line 1:1): invalid value volinfo oldFormat: failed to unmarshal old version volume info: proto: syntax error (line 1:1): invalid value volinfo I0824 22:24:50.549616 volume_grpc_erasure_coding_recover.go:74 recovered missing EC index for 1 volume(s) from peers and mounted their shards --- PASS: TestVolumeEcShardsMount_RecoverMissingIndex (0.05s) === RUN TestCheckEcVolumeStatusCountOnlyDataShards --- PASS: TestCheckEcVolumeStatusCountOnlyDataShards (0.00s) === RUN TestRemoveStaleEcArtifacts --- PASS: TestRemoveStaleEcArtifacts (0.01s) === RUN TestDeleteEcShardsWithoutLocalEcx --- PASS: TestDeleteEcShardsWithoutLocalEcx (0.00s) === RUN TestVolumeEcShardsInfo_AggregatesAcrossDisks I0824 22:24:50.580648 store_ec.go:213 MountEcShards 42.0 on disk ID 0 I0824 22:24:50.581448 store_ec.go:213 MountEcShards 42.5 on disk ID 0 I0824 22:24:50.583310 store_ec.go:213 MountEcShards 42.7 on disk ID 1 I0824 22:24:50.583934 store_ec.go:213 MountEcShards 42.12 on disk ID 1 I0824 22:24:50.584131 volume_grpc_erasure_coding.go:1131 VolumeEcShardsInfo: volume 42 --- PASS: TestVolumeEcShardsInfo_AggregatesAcrossDisks (0.02s) === RUN TestValidateRemoteEndpoint === RUN TestValidateRemoteEndpoint/empty === RUN TestValidateRemoteEndpoint/loopback_literal === RUN TestValidateRemoteEndpoint/ipv6_loopback === RUN TestValidateRemoteEndpoint/imds_ipv4 === RUN TestValidateRemoteEndpoint/unspecified_ipv4 === RUN TestValidateRemoteEndpoint/link-local_ipv6 === RUN TestValidateRemoteEndpoint/ftp_scheme === RUN TestValidateRemoteEndpoint/missing_scheme === RUN TestValidateRemoteEndpoint/imds_hostname === RUN TestValidateRemoteEndpoint/imds_short_hostname === RUN TestValidateRemoteEndpoint/host_resolves_to_loopback === RUN TestValidateRemoteEndpoint/host_resolves_to_link-local === RUN TestValidateRemoteEndpoint/rfc1918_10/8_literal === RUN TestValidateRemoteEndpoint/rfc1918_172.16/12_literal === RUN TestValidateRemoteEndpoint/rfc1918_192.168/16_literal === RUN TestValidateRemoteEndpoint/cgnat_literal === RUN TestValidateRemoteEndpoint/host_resolves_to_rfc1918_10/8 === RUN TestValidateRemoteEndpoint/host_resolves_to_rfc1918_172/12 === RUN TestValidateRemoteEndpoint/host_resolves_to_rfc1918_192.168/16 === RUN TestValidateRemoteEndpoint/host_resolves_to_cgnat === RUN TestValidateRemoteEndpoint/nat64_imds === RUN TestValidateRemoteEndpoint/nat64_loopback === RUN TestValidateRemoteEndpoint/6to4_private === RUN TestValidateRemoteEndpoint/teredo_loopback === RUN TestValidateRemoteEndpoint/ipv4-compatible_loopback === RUN TestValidateRemoteEndpoint/nat64_public_passes === RUN TestValidateRemoteEndpoint/6to4_public_passes === RUN TestValidateRemoteEndpoint/teredo_public_passes === RUN TestValidateRemoteEndpoint/ipv4-compatible_public_passes === RUN TestValidateRemoteEndpoint/nat64_non-wellknown-prefix_not_decoded === RUN TestValidateRemoteEndpoint/public_s3 --- PASS: TestValidateRemoteEndpoint (0.01s) --- PASS: TestValidateRemoteEndpoint/empty (0.00s) --- PASS: TestValidateRemoteEndpoint/loopback_literal (0.00s) --- PASS: TestValidateRemoteEndpoint/ipv6_loopback (0.00s) --- PASS: TestValidateRemoteEndpoint/imds_ipv4 (0.00s) --- PASS: TestValidateRemoteEndpoint/unspecified_ipv4 (0.00s) --- PASS: TestValidateRemoteEndpoint/link-local_ipv6 (0.00s) --- PASS: TestValidateRemoteEndpoint/ftp_scheme (0.00s) --- PASS: TestValidateRemoteEndpoint/missing_scheme (0.00s) --- PASS: TestValidateRemoteEndpoint/imds_hostname (0.00s) --- PASS: TestValidateRemoteEndpoint/imds_short_hostname (0.00s) --- PASS: TestValidateRemoteEndpoint/host_resolves_to_loopback (0.00s) --- PASS: TestValidateRemoteEndpoint/host_resolves_to_link-local (0.00s) --- PASS: TestValidateRemoteEndpoint/rfc1918_10/8_literal (0.00s) --- PASS: TestValidateRemoteEndpoint/rfc1918_172.16/12_literal (0.00s) --- PASS: TestValidateRemoteEndpoint/rfc1918_192.168/16_literal (0.00s) --- PASS: TestValidateRemoteEndpoint/cgnat_literal (0.00s) --- PASS: TestValidateRemoteEndpoint/host_resolves_to_rfc1918_10/8 (0.00s) --- PASS: TestValidateRemoteEndpoint/host_resolves_to_rfc1918_172/12 (0.00s) --- PASS: TestValidateRemoteEndpoint/host_resolves_to_rfc1918_192.168/16 (0.00s) --- PASS: TestValidateRemoteEndpoint/host_resolves_to_cgnat (0.00s) --- PASS: TestValidateRemoteEndpoint/nat64_imds (0.00s) --- PASS: TestValidateRemoteEndpoint/nat64_loopback (0.00s) --- PASS: TestValidateRemoteEndpoint/6to4_private (0.00s) --- PASS: TestValidateRemoteEndpoint/teredo_loopback (0.00s) --- PASS: TestValidateRemoteEndpoint/ipv4-compatible_loopback (0.00s) --- PASS: TestValidateRemoteEndpoint/nat64_public_passes (0.00s) --- PASS: TestValidateRemoteEndpoint/6to4_public_passes (0.00s) --- PASS: TestValidateRemoteEndpoint/teredo_public_passes (0.00s) --- PASS: TestValidateRemoteEndpoint/ipv4-compatible_public_passes (0.00s) --- PASS: TestValidateRemoteEndpoint/nat64_non-wellknown-prefix_not_decoded (0.00s) --- PASS: TestValidateRemoteEndpoint/public_s3 (0.00s) === RUN TestValidateRemoteEndpointResolverFailure --- PASS: TestValidateRemoteEndpointResolverFailure (0.00s) === RUN TestGuardedDialerRebind --- PASS: TestGuardedDialerRebind (0.00s) === RUN TestRemoteEndpointGuardCoversS3CompatibleSiblings --- PASS: TestRemoteEndpointGuardCoversS3CompatibleSiblings (0.00s) === RUN TestRemoteEndpointGuardCoversAzure === RUN TestRemoteEndpointGuardCoversAzure/imds === RUN TestRemoteEndpointGuardCoversAzure/loopback === RUN TestRemoteEndpointGuardCoversAzure/private --- PASS: TestRemoteEndpointGuardCoversAzure (0.00s) --- PASS: TestRemoteEndpointGuardCoversAzure/imds (0.00s) --- PASS: TestRemoteEndpointGuardCoversAzure/loopback (0.00s) --- PASS: TestRemoteEndpointGuardCoversAzure/private (0.00s) === RUN TestValidateReplicaTarget === RUN TestValidateReplicaTarget/embedded_path_and_query === RUN TestValidateReplicaTarget/loopback_literal === RUN TestValidateReplicaTarget/ipv6_loopback === RUN TestValidateReplicaTarget/metadata_literal === RUN TestValidateReplicaTarget/unspecified === RUN TestValidateReplicaTarget/metadata_hostname === RUN TestValidateReplicaTarget/scheme_rejected === RUN TestValidateReplicaTarget/path_rejected === RUN TestValidateReplicaTarget/query_rejected === RUN TestValidateReplicaTarget/userinfo_rejected === RUN TestValidateReplicaTarget/missing_port_literal === RUN TestValidateReplicaTarget/missing_port_hostname === RUN TestValidateReplicaTarget/empty === RUN TestValidateReplicaTarget/resolves_to_loopback === RUN TestValidateReplicaTarget/resolves_to_link-local === RUN TestValidateReplicaTarget/private_peer_literal === RUN TestValidateReplicaTarget/private_192_peer === RUN TestValidateReplicaTarget/private_peer_hostname --- PASS: TestValidateReplicaTarget (0.00s) --- PASS: TestValidateReplicaTarget/embedded_path_and_query (0.00s) --- PASS: TestValidateReplicaTarget/loopback_literal (0.00s) --- PASS: TestValidateReplicaTarget/ipv6_loopback (0.00s) --- PASS: TestValidateReplicaTarget/metadata_literal (0.00s) --- PASS: TestValidateReplicaTarget/unspecified (0.00s) --- PASS: TestValidateReplicaTarget/metadata_hostname (0.00s) --- PASS: TestValidateReplicaTarget/scheme_rejected (0.00s) --- PASS: TestValidateReplicaTarget/path_rejected (0.00s) --- PASS: TestValidateReplicaTarget/query_rejected (0.00s) --- PASS: TestValidateReplicaTarget/userinfo_rejected (0.00s) --- PASS: TestValidateReplicaTarget/missing_port_literal (0.00s) --- PASS: TestValidateReplicaTarget/missing_port_hostname (0.00s) --- PASS: TestValidateReplicaTarget/empty (0.00s) --- PASS: TestValidateReplicaTarget/resolves_to_loopback (0.00s) --- PASS: TestValidateReplicaTarget/resolves_to_link-local (0.00s) --- PASS: TestValidateReplicaTarget/private_peer_literal (0.00s) --- PASS: TestValidateReplicaTarget/private_192_peer (0.00s) --- PASS: TestValidateReplicaTarget/private_peer_hostname (0.00s) === RUN TestGuardedRemoteClientSkipsFixedHostBackends --- PASS: TestGuardedRemoteClientSkipsFixedHostBackends (0.00s) === RUN TestGuardedRemoteClientAzureBuildsGuardedClient --- PASS: TestGuardedRemoteClientAzureBuildsGuardedClient (0.02s) === RUN TestGcsCredentialsArePath --- PASS: TestGcsCredentialsArePath (0.00s) === RUN TestGuardedRemoteClientGuardsGcsTokenURL --- PASS: TestGuardedRemoteClientGuardsGcsTokenURL (0.00s) === RUN TestCheckGcsCredentials --- PASS: TestCheckGcsCredentials (0.00s) === RUN TestGuardedDialerLiteralBlocked --- PASS: TestGuardedDialerLiteralBlocked (0.00s) === RUN TestGuardedReplicaDialerRebind --- PASS: TestGuardedReplicaDialerRebind (0.00s) === RUN TestTierMoveDatFromRemote_KeepRemote_LeavesReplicaLocal I0824 22:24:50.634678 store.go:329 In dir /tmp/TestTierMoveDatFromRemote_KeepRemote_LeavesReplicaLocal2477185659/002 (disk ID 0) adds volume:71 collection: replicaPlacement:000 ttl: I0824 22:24:50.637528 store.go:334 add volume 71 on disk ID 0 I0824 22:24:50.641099 store.go:922 UnmountVolume 71 disk_id:0 I0824 22:24:50.649119 volume_tier.go:31 volume 71 is tiered to tier_test_local_dir.default as obj-1787610290638455651 and read only I0824 22:24:50.649492 volume_loading.go:167 loading volume 71 from remote files:{backend_type:"tier_test_local_dir" backend_id:"default" key:"obj-1787610290638455651" file_size:288 modified_time:1787610290 extension:".dat"} version:3 bytes_offset:4 E0824 22:24:50.650478 volume_loading.go:233 readSuperBlock remote volume 71: I0824 22:24:50.651312 needle_map_sorted_file.go:26 Start to Generate /tmp/TestTierMoveDatFromRemote_KeepRemote_LeavesReplicaLocal2477185659/002/71.sdx from /tmp/TestTierMoveDatFromRemote_KeepRemote_LeavesReplicaLocal2477185659/002/71.idx I0824 22:24:50.656934 needle_map_sorted_file.go:28 Finished Generating /tmp/TestTierMoveDatFromRemote_KeepRemote_LeavesReplicaLocal2477185659/002/71.sdx from /tmp/TestTierMoveDatFromRemote_KeepRemote_LeavesReplicaLocal2477185659/002/71.idx I0824 22:24:50.662112 store.go:922 UnmountVolume 71 disk_id:0 I0824 22:24:50.664368 store.go:872 mount volume 71 --- PASS: TestTierMoveDatFromRemote_KeepRemote_LeavesReplicaLocal (0.04s) === RUN TestVolumeTierMoveDatPreservesModifiedTime I0824 22:24:50.672756 store.go:329 In dir /tmp/TestVolumeTierMoveDatPreservesModifiedTime2398288688/001 (disk ID 0) adds volume:1 collection: replicaPlacement:000 ttl: I0824 22:24:50.676222 store.go:334 add volume 1 on disk ID 0 --- PASS: TestVolumeTierMoveDatPreservesModifiedTime (0.01s) === RUN TestDeleteHandlerRejectsOverflowingNeedleDelta I0824 22:24:50.683491 common.go:109 response method:DELETE URL:/1,ffffffffffffffff00000000_1 with httpStatus:400 and JSON:{"error":"delta 1 overflows needle id 66666666666666666666666666666666"} --- PASS: TestDeleteHandlerRejectsOverflowingNeedleDelta (0.00s) === RUN TestBatchDeleteRejectsOverflowingNeedleDelta --- PASS: TestBatchDeleteRejectsOverflowingNeedleDelta (0.00s) === RUN TestQueryRejectsOverflowingNeedleDelta I0824 22:24:50.684978 volume_grpc_query.go:25 volume query failed to parse fid 1,ffffffffffffffff00000000_1: delta 1 overflows needle id 66666666666666666666666666666666 --- PASS: TestQueryRejectsOverflowingNeedleDelta (0.00s) === RUN TestListedEntriesTrailingSlash --- PASS: TestListedEntriesTrailingSlash (0.00s) === RUN TestListedEntriesWithoutCache --- PASS: TestListedEntriesWithoutCache (0.00s) === RUN TestListedEntriesCap --- PASS: TestListedEntriesCap (0.26s) === RUN TestListedEntriesPerRequest --- PASS: TestListedEntriesPerRequest (0.00s) === RUN TestWrappedFsConfine --- PASS: TestWrappedFsConfine (0.00s) PASS ok github.com/seaweedfs/seaweedfs/weed/server 401.817s ? github.com/seaweedfs/seaweedfs/weed/server/constants [no test files] === RUN TestToBreadcrumb === RUN TestToBreadcrumb/empty === RUN TestToBreadcrumb/test1 === RUN TestToBreadcrumb/test2 === RUN TestToBreadcrumb/test3 --- PASS: TestToBreadcrumb (0.00s) --- PASS: TestToBreadcrumb/empty (0.00s) --- PASS: TestToBreadcrumb/test1 (0.00s) --- PASS: TestToBreadcrumb/test2 (0.00s) --- PASS: TestToBreadcrumb/test3 (0.00s) === RUN TestPrintPathPreservesPathSafeTimestampChars --- PASS: TestPrintPathPreservesPathSafeTimestampChars (0.00s) === RUN TestQueryEscapePreservesPlusInLoadMoreCursor --- PASS: TestQueryEscapePreservesPlusInLoadMoreCursor (0.00s) PASS ok github.com/seaweedfs/seaweedfs/weed/server/filer_ui 9.956s ? github.com/seaweedfs/seaweedfs/weed/server/master_ui [no test files] ? github.com/seaweedfs/seaweedfs/weed/server/postgres [no test files] === RUN TestStatusTpl --- PASS: TestStatusTpl (0.07s) PASS ok github.com/seaweedfs/seaweedfs/weed/server/volume_server_ui 11.379s === RUN TestPutFileEscapesQueryInjection --- PASS: TestPutFileEscapesQueryInjection (0.11s) === RUN TestPathWithin --- PASS: TestPathWithin (0.00s) === RUN TestHasExplicitPermissionSiblingPrefix --- PASS: TestHasExplicitPermissionSiblingPrefix (0.00s) === RUN TestIsPathInHomeDirectorySiblingPrefix --- PASS: TestIsPathInHomeDirectorySiblingPrefix (0.00s) === RUN TestToAbsolutePath === RUN TestToAbsolutePath/normal_path === RUN TestToAbsolutePath/root_path === RUN TestToAbsolutePath/path_with_dot === RUN TestToAbsolutePath/path_traversal_attempts === RUN TestToAbsolutePath/path_traversal_attempts_2 === RUN TestToAbsolutePath/path_traversal_attempts_3 === RUN TestToAbsolutePath/empty_path === RUN TestToAbsolutePath/multiple_slashes === RUN TestToAbsolutePath/trailing_slash === RUN TestToAbsolutePath/empty_HomeDir_passthrough === RUN TestToAbsolutePath/root_HomeDir_passthrough --- PASS: TestToAbsolutePath (0.00s) --- PASS: TestToAbsolutePath/normal_path (0.00s) --- PASS: TestToAbsolutePath/root_path (0.00s) --- PASS: TestToAbsolutePath/path_with_dot (0.00s) --- PASS: TestToAbsolutePath/path_traversal_attempts (0.00s) --- PASS: TestToAbsolutePath/path_traversal_attempts_2 (0.00s) --- PASS: TestToAbsolutePath/path_traversal_attempts_3 (0.00s) --- PASS: TestToAbsolutePath/empty_path (0.00s) --- PASS: TestToAbsolutePath/multiple_slashes (0.00s) --- PASS: TestToAbsolutePath/trailing_slash (0.00s) --- PASS: TestToAbsolutePath/empty_HomeDir_passthrough (0.00s) --- PASS: TestToAbsolutePath/root_HomeDir_passthrough (0.00s) PASS ok github.com/seaweedfs/seaweedfs/weed/sftpd 25.256s === RUN TestManager_CertificateTakesOverPublicKeyChannel --- PASS: TestManager_CertificateTakesOverPublicKeyChannel (0.06s) === RUN TestManager_PublicKeyOnlyChannel --- PASS: TestManager_PublicKeyOnlyChannel (0.00s) === RUN TestManager_CertificateRequiresCAFile --- PASS: TestManager_CertificateRequiresCAFile (0.00s) === RUN TestCertificateAuthenticator_GoldenPath --- PASS: TestCertificateAuthenticator_GoldenPath (0.00s) === RUN TestCertificateAuthenticator_RejectsPlainPublicKey --- PASS: TestCertificateAuthenticator_RejectsPlainPublicKey (0.00s) === RUN TestCertificateAuthenticator_RejectsHostCert --- PASS: TestCertificateAuthenticator_RejectsHostCert (0.00s) === RUN TestCertificateAuthenticator_RejectsEmptyPrincipals --- PASS: TestCertificateAuthenticator_RejectsEmptyPrincipals (0.00s) === RUN TestCertificateAuthenticator_RejectsWrongPrincipal --- PASS: TestCertificateAuthenticator_RejectsWrongPrincipal (0.01s) === RUN TestCertificateAuthenticator_RejectsExpiredCert --- PASS: TestCertificateAuthenticator_RejectsExpiredCert (0.00s) === RUN TestCertificateAuthenticator_RejectsUnknownCA --- PASS: TestCertificateAuthenticator_RejectsUnknownCA (0.00s) === RUN TestCertificateAuthenticator_RejectsUnknownUser --- PASS: TestCertificateAuthenticator_RejectsUnknownUser (0.01s) === RUN TestCertificateAuthenticator_LoadsMultipleCAKeys --- PASS: TestCertificateAuthenticator_LoadsMultipleCAKeys (0.00s) === RUN TestNewCertificateAuthenticator_DisabledIgnoresFile --- PASS: TestNewCertificateAuthenticator_DisabledIgnoresFile (0.00s) === RUN TestNewCertificateAuthenticator_EnabledRequiresFile --- PASS: TestNewCertificateAuthenticator_EnabledRequiresFile (0.00s) === RUN TestNewCertificateAuthenticator_MissingFile --- PASS: TestNewCertificateAuthenticator_MissingFile (0.00s) PASS ok github.com/seaweedfs/seaweedfs/weed/sftpd/auth 3.306s ? github.com/seaweedfs/seaweedfs/weed/sftpd/user [no test files] ? github.com/seaweedfs/seaweedfs/weed/sftpd/utils [no test files] === RUN TestRaftLeaderTransfer_Name --- PASS: TestRaftLeaderTransfer_Name (0.00s) === RUN TestRaftLeaderTransfer_Help --- PASS: TestRaftLeaderTransfer_Help (0.00s) === RUN TestRaftLeaderTransfer_HasTag --- PASS: TestRaftLeaderTransfer_HasTag (0.00s) === RUN TestRaftLeaderTransfer_ValidateTargetIdWithoutAddress --- PASS: TestRaftLeaderTransfer_ValidateTargetIdWithoutAddress (0.00s) === RUN TestRaftLeaderTransfer_ValidateTargetAddressWithoutId --- PASS: TestRaftLeaderTransfer_ValidateTargetAddressWithoutId (0.00s) === RUN TestRaftLeaderTransfer_UnknownFlag flag provided but not defined: -unknown-flag Usage of cluster.raft.transferLeader: -address string target server grpc address (must be used with -id) -id string target server id (must be used with -address) --- PASS: TestRaftLeaderTransfer_UnknownFlag (0.00s) === RUN TestPrintClusterInfo --- PASS: TestPrintClusterInfo (0.00s) === RUN TestPrintClusterInfo_multiDiskPerNode --- PASS: TestPrintClusterInfo_multiDiskPerNode (0.00s) === RUN TestPrintVolumeInfo --- PASS: TestPrintVolumeInfo (0.06s) === RUN TestPrintStorageInfo --- PASS: TestPrintStorageInfo (0.06s) === RUN TestPrintFilesInfo --- PASS: TestPrintFilesInfo (0.00s) === RUN TestCollectCollectionInfoEC --- PASS: TestCollectCollectionInfoEC (0.02s) === RUN TestCollectCollectionInfoMixed --- PASS: TestCollectCollectionInfoMixed (0.01s) === RUN TestCollectCollectionInfoRegularVolumeDedupesReplicas --- PASS: TestCollectCollectionInfoRegularVolumeDedupesReplicas (0.00s) === RUN TestShortName --- PASS: TestShortName (0.02s) === RUN TestRelevantNodes --- PASS: TestRelevantNodes (0.00s) === RUN TestComputeOwnerTarget_EvenDivision --- PASS: TestComputeOwnerTarget_EvenDivision (0.00s) === RUN TestComputeOwnerTarget_UnevenDivision --- PASS: TestComputeOwnerTarget_UnevenDivision (0.00s) === RUN TestComputeOwnerTarget_InactiveNodesDrainToZero --- PASS: TestComputeOwnerTarget_InactiveNodesDrainToZero (0.00s) === RUN TestBuildDistributionCounts --- PASS: TestBuildDistributionCounts (0.00s) === RUN TestBuildDistributionCounts_NilFidFallback --- PASS: TestBuildDistributionCounts_NilFidFallback (0.03s) === RUN TestSelectActiveNodes_AllNodes --- PASS: TestSelectActiveNodes_AllNodes (0.00s) === RUN TestSelectActiveNodes_LimitToTwo --- PASS: TestSelectActiveNodes_LimitToTwo (0.00s) === RUN TestPlanOwnerMoves_BasicBalance --- PASS: TestPlanOwnerMoves_BasicBalance (0.00s) === RUN TestPlanOwnerMoves_AlreadyBalanced --- PASS: TestPlanOwnerMoves_AlreadyBalanced (0.00s) === RUN TestPlanDistribution_Primary_BalancesChunks --- PASS: TestPlanDistribution_Primary_BalancesChunks (0.00s) === RUN TestPlanDistribution_Primary_NoMovesWhenBalanced --- PASS: TestPlanDistribution_Primary_NoMovesWhenBalanced (0.00s) === RUN TestPlanDistribution_RoundRobin_AssignsByOffset --- PASS: TestPlanDistribution_RoundRobin_AssignsByOffset (0.01s) === RUN TestPlanDistribution_RoundRobin_OffsetOrdering --- PASS: TestPlanDistribution_RoundRobin_OffsetOrdering (0.00s) === RUN TestPlanDistribution_Replica_BalancesOwnerAndCopies --- PASS: TestPlanDistribution_Replica_BalancesOwnerAndCopies (0.00s) === RUN TestPrintRedistributionPlan_Output --- PASS: TestPrintRedistributionPlan_Output (0.00s) === RUN TestCreateMergePlan_HonorsExplicitDirection --- PASS: TestCreateMergePlan_HonorsExplicitDirection (0.00s) === RUN TestCreateMergePlan_DirectedAllowsEmptyTarget --- PASS: TestCreateMergePlan_DirectedAllowsEmptyTarget (0.00s) === RUN TestCreateMergePlan_DirectedRejectsSelfMap --- PASS: TestCreateMergePlan_DirectedRejectsSelfMap (0.00s) === RUN TestCreateMergePlan_DirectedRejectsIneligible === RUN TestCreateMergePlan_DirectedRejectsIneligible/readonly_source === RUN TestCreateMergePlan_DirectedRejectsIneligible/readonly_target === RUN TestCreateMergePlan_DirectedRejectsIneligible/empty_source === RUN TestCreateMergePlan_DirectedRejectsIneligible/wrong_collection --- PASS: TestCreateMergePlan_DirectedRejectsIneligible (0.00s) --- PASS: TestCreateMergePlan_DirectedRejectsIneligible/readonly_source (0.00s) --- PASS: TestCreateMergePlan_DirectedRejectsIneligible/readonly_target (0.00s) --- PASS: TestCreateMergePlan_DirectedRejectsIneligible/empty_source (0.00s) --- PASS: TestCreateMergePlan_DirectedRejectsIneligible/wrong_collection (0.00s) === RUN TestCreateMergePlan_DistributesAcrossMultipleTargets --- PASS: TestCreateMergePlan_DistributesAcrossMultipleTargets (0.00s) === RUN TestCreateMergePlan_MultiTargetRejectsInsufficientCapacity --- PASS: TestCreateMergePlan_MultiTargetRejectsInsufficientCapacity (0.00s) === RUN TestCreateMergePlan_TargetListRestrictsHeuristic --- PASS: TestCreateMergePlan_TargetListRestrictsHeuristic (0.00s) === RUN TestParseTargetVolumeIds --- PASS: TestParseTargetVolumeIds (0.00s) === RUN TestGetVolumeSize_ClampsDeletedOverSize --- PASS: TestGetVolumeSize_ClampsDeletedOverSize (0.00s) === RUN TestFsMvMovesIntoExistingDestinationDirectory --- PASS: TestFsMvMovesIntoExistingDestinationDirectory (0.07s) === RUN TestPlanLocalToRemoteSync === RUN TestPlanLocalToRemoteSync/copy_local-only_files,_no_delete_flag === RUN TestPlanLocalToRemoteSync/delete_flag_removes_remote-only_files === RUN TestPlanLocalToRemoteSync/delete_with_empty_local_storage_removes_all_remote_files === RUN TestPlanLocalToRemoteSync/delete_respects_include_pattern === RUN TestPlanLocalToRemoteSync/delete_respects_exclude_pattern === RUN TestPlanLocalToRemoteSync/delete_ignores_size/age_filters === RUN TestPlanLocalToRemoteSync/delete_removes_orphaned_files,_directory_entries_left_alone === RUN TestPlanLocalToRemoteSync/exclude_pattern_protects_remote_files_under_nested_directory === RUN TestPlanLocalToRemoteSync/remote_files_outside_-dir_are_never_deleted === RUN TestPlanLocalToRemoteSync/directory_present_locally_is_not_deleted === RUN TestPlanLocalToRemoteSync/forceUpdate_copies_files_that_exist_on_remote === RUN TestPlanLocalToRemoteSync/local_directories_are_not_copied --- PASS: TestPlanLocalToRemoteSync (0.00s) --- PASS: TestPlanLocalToRemoteSync/copy_local-only_files,_no_delete_flag (0.00s) --- PASS: TestPlanLocalToRemoteSync/delete_flag_removes_remote-only_files (0.00s) --- PASS: TestPlanLocalToRemoteSync/delete_with_empty_local_storage_removes_all_remote_files (0.00s) --- PASS: TestPlanLocalToRemoteSync/delete_respects_include_pattern (0.00s) --- PASS: TestPlanLocalToRemoteSync/delete_respects_exclude_pattern (0.00s) --- PASS: TestPlanLocalToRemoteSync/delete_ignores_size/age_filters (0.00s) --- PASS: TestPlanLocalToRemoteSync/delete_removes_orphaned_files,_directory_entries_left_alone (0.00s) --- PASS: TestPlanLocalToRemoteSync/exclude_pattern_protects_remote_files_under_nested_directory (0.00s) --- PASS: TestPlanLocalToRemoteSync/remote_files_outside_-dir_are_never_deleted (0.00s) --- PASS: TestPlanLocalToRemoteSync/directory_present_locally_is_not_deleted (0.00s) --- PASS: TestPlanLocalToRemoteSync/forceUpdate_copies_files_that_exist_on_remote (0.00s) --- PASS: TestPlanLocalToRemoteSync/local_directories_are_not_copied (0.00s) === RUN TestFileFilter_matchesName === RUN TestFileFilter_matchesName/no_filters === RUN TestFileFilter_matchesName/include_match === RUN TestFileFilter_matchesName/include_mismatch === RUN TestFileFilter_matchesName/exclude_match === RUN TestFileFilter_matchesName/exclude_mismatch --- PASS: TestFileFilter_matchesName (0.00s) --- PASS: TestFileFilter_matchesName/no_filters (0.00s) --- PASS: TestFileFilter_matchesName/include_match (0.00s) --- PASS: TestFileFilter_matchesName/include_mismatch (0.00s) --- PASS: TestFileFilter_matchesName/exclude_match (0.00s) --- PASS: TestFileFilter_matchesName/exclude_mismatch (0.00s) === RUN TestFileFilter_matches_minCacheAge === RUN TestFileFilter_matches_minCacheAge/no_minCacheAge === RUN TestFileFilter_matches_minCacheAge/recent_cache,_should_not_match === RUN TestFileFilter_matches_minCacheAge/old_cache,_should_match === RUN TestFileFilter_matches_minCacheAge/no_remote_entry,_uses_crtime_-_recent,_should_not_match === RUN TestFileFilter_matches_minCacheAge/no_remote_entry,_uses_crtime_-_old,_should_match === RUN TestFileFilter_matches_minCacheAge/remote_entry_with_0_sync_ts,_uses_crtime_-_recent,_should_not_match === RUN TestFileFilter_matches_minCacheAge/nil_attributes,_should_not_match --- PASS: TestFileFilter_matches_minCacheAge (0.00s) --- PASS: TestFileFilter_matches_minCacheAge/no_minCacheAge (0.00s) --- PASS: TestFileFilter_matches_minCacheAge/recent_cache,_should_not_match (0.00s) --- PASS: TestFileFilter_matches_minCacheAge/old_cache,_should_match (0.00s) --- PASS: TestFileFilter_matches_minCacheAge/no_remote_entry,_uses_crtime_-_recent,_should_not_match (0.00s) --- PASS: TestFileFilter_matches_minCacheAge/no_remote_entry,_uses_crtime_-_old,_should_match (0.00s) --- PASS: TestFileFilter_matches_minCacheAge/remote_entry_with_0_sync_ts,_uses_crtime_-_recent,_should_not_match (0.00s) --- PASS: TestFileFilter_matches_minCacheAge/nil_attributes,_should_not_match (0.00s) === RUN TestUpdateBucketActions_SetActions --- PASS: TestUpdateBucketActions_SetActions (0.00s) === RUN TestUpdateBucketActions_ReplaceActions --- PASS: TestUpdateBucketActions_ReplaceActions (0.00s) === RUN TestUpdateBucketActions_None --- PASS: TestUpdateBucketActions_None (0.00s) === RUN TestUpdateBucketActions_PreservesOtherBuckets --- PASS: TestUpdateBucketActions_PreservesOtherBuckets (0.00s) === RUN TestUpdateBucketActions_PreservesGlobalActions --- PASS: TestUpdateBucketActions_PreservesGlobalActions (0.00s) === RUN TestDisplayBucketAccess --- PASS: TestDisplayBucketAccess (0.00s) === RUN TestDisplayBucketAccess_None --- PASS: TestDisplayBucketAccess_None (0.00s) === RUN TestCircuitBreakerShell --- PASS: TestCircuitBreakerShell (0.03s) === RUN TestParseShardsSpec_Range --- PASS: TestParseShardsSpec_Range (0.00s) === RUN TestParseShardsSpec_Set --- PASS: TestParseShardsSpec_Set (0.00s) === RUN TestParseShardsSpec_DedupSort --- PASS: TestParseShardsSpec_DedupSort (0.00s) === RUN TestParseShardsSpec_OutOfRange --- PASS: TestParseShardsSpec_OutOfRange (0.00s) === RUN TestParseShardsSpec_BadRange --- PASS: TestParseShardsSpec_BadRange (0.01s) === RUN TestResolveShardSelection_Mutex --- PASS: TestResolveShardSelection_Mutex (0.00s) === RUN TestResolveShardSelection_SingleShard --- PASS: TestResolveShardSelection_SingleShard (0.00s) === RUN TestFormatShardLabel --- PASS: TestFormatShardLabel (0.00s) === RUN TestIsGoodMove replication: 100 expected false name: test 100 move to wrong data centers replication: 100 expected true name: test 100 move to spread into proper data centers replication: 001 expected false name: test move to the same node replication: 001 expected false name: test move to the same rack, but existing node replication: 001 expected true name: test move to the same rack, a new node replication: 010 expected false name: test 010 move all to the same rack replication: 010 expected true name: test 010 move to spread racks replication: 010 expected true name: test 010 move to spread racks replication: 011 expected true name: test 011 switch which rack has more replicas replication: 011 expected true name: test 011 move the lonely replica to another racks replication: 011 expected false name: test 011 move to wrong racks replication: 011 expected false name: test 011 move all to the same rack replication: 001 expected false name: test 001 reject move onto a machine already holding a replica replication: 001 expected true name: test 001 allow move onto a different machine in the rack --- PASS: TestIsGoodMove (0.02s) === RUN TestBalance hdd 0.00 0.01:0.00 moving volume collection4_7 192.168.1.2:8080 => 192.168.1.6:8080 hdd 0.00 0.01:0.00 moving volume collection3_9 192.168.1.2:8080 => 192.168.1.6:8080 hdd 0.00 0.01:0.00 moving volume 4 192.168.1.2:8080 => 192.168.1.6:8080 hdd 0.00 0.01:0.00 moving volume 2 192.168.1.2:8080 => 192.168.1.6:8080 hdd 0.00 0.01:0.00 moving volume 6 192.168.1.2:8080 => 192.168.1.6:8080 hdd 0.00 0.01:0.00 moving volume 3 192.168.1.2:8080 => 192.168.1.6:8080 hdd 0.00 0.00:0.00 moving volume 31 192.168.1.4:8080 => 192.168.1.6:8080 hdd 0.00 0.00:0.00 moving volume 29 192.168.1.4:8080 => 192.168.1.6:8080 hdd 0.00 0.00:0.00 moving volume 30 192.168.1.4:8080 => 192.168.1.6:8080 hdd 0.00 0.00:0.00 moving volume 27 192.168.1.4:8080 => 192.168.1.6:8080 hdd 0.00 0.00:0.00 moving volume 28 192.168.1.4:8080 => 192.168.1.6:8080 hdd 0.00 0.00:0.00 moving volume collection0_25 192.168.1.4:8080 => 192.168.1.6:8080 hdd 0.00 0.00:0.00 moving volume collection1_80 192.168.1.4:8080 => 192.168.1.6:8080 hdd 0.00 0.00:0.00 moving volume collection1_69 192.168.1.4:8080 => 192.168.1.6:8080 hdd 0.00 0.00:0.00 moving volume collection1_84 192.168.1.4:8080 => 192.168.1.6:8080 hdd 0.00 0.00:0.00 moving volume collection1_63 192.168.1.4:8080 => 192.168.1.6:8080 hdd 0.00 0.00:0.00 moving volume collection1_74 192.168.1.4:8080 => 192.168.1.6:8080 hdd 0.00 0.00:0.00 moving volume collection1_85 192.168.1.4:8080 => 192.168.1.6:8080 hdd 0.00 0.00:0.00 moving volume collection1_54 192.168.1.4:8080 => 192.168.1.6:8080 hdd 0.00 0.00:0.00 moving volume collection1_97 192.168.1.4:8080 => 192.168.1.6:8080 hdd 0.00 0.00:0.00 moving volume collection1_174 192.168.1.4:8080 => 192.168.1.6:8080 hdd 0.00 0.00:0.00 moving volume collection1_105 192.168.1.4:8080 => 192.168.1.6:8080 hdd 0.00 0.00:0.00 moving volume collection0_24 192.168.1.4:8080 => 192.168.1.6:8080 hdd 0.00 0.00:0.00 moving volume collection1_173 192.168.1.4:8080 => 192.168.1.6:8080 hdd 0.00 0.00:0.00 moving volume collection1_136 192.168.1.4:8080 => 192.168.1.6:8080 hdd 0.00 0.00:0.00 moving volume collection1_138 192.168.1.4:8080 => 192.168.1.6:8080 hdd 0.00 0.00:0.00 moving volume collection1_116 192.168.1.4:8080 => 192.168.1.6:8080 --- PASS: TestBalance (0.04s) === RUN TestBalanceParallel hdd 0.54 2.33:0.10 hdd 0.54 1.50:0.10 hdd 0.54 1.00:0.22 moving volume 3 full => empty moving volume 5 full => empty moving volume 4 full => empty hdd 0.54 0.67:0.38 moving volume 6 full => empty --- PASS: TestBalanceParallel (0.01s) === RUN TestBalanceSkipsRemoteStorageVolumes === RUN TestBalanceSkipsRemoteStorageVolumes/maxParallelization=0 hdd 0.18 0.43:0.10 moving volume 5 full => empty hdd 0.18 0.25:0.10 moving volume 6 full => empty === RUN TestBalanceSkipsRemoteStorageVolumes/maxParallelization=2 hdd 0.18 0.43:0.10 hdd 0.18 0.25:0.10 moving volume 6 full => empty moving volume 5 full => empty --- PASS: TestBalanceSkipsRemoteStorageVolumes (0.00s) --- PASS: TestBalanceSkipsRemoteStorageVolumes/maxParallelization=0 (0.00s) --- PASS: TestBalanceSkipsRemoteStorageVolumes/maxParallelization=2 (0.00s) === RUN TestVolumeBalanceMoveOptions --- PASS: TestVolumeBalanceMoveOptions (0.02s) === RUN TestBalanceDoesNotDrainOntoOneNode hdd 0.33 1.00:0.10 moving volume 2 full => empty hdd 0.33 0.67:0.10 moving volume 1 full => empty hdd 0.33 0.43:0.22 moving volume 4 full => empty --- PASS: TestBalanceDoesNotDrainOntoOneNode (0.00s) === RUN TestBalanceByDiskUsage hdd 0.03 0.43:0.02 moving volume 101 half-full => byte-full hdd 0.03 0.36:0.02 moving volume 102 half-full => byte-full hdd 0.03 0.30:0.02 moving volume 105 half-full => byte-full hdd 0.03 0.25:0.02 moving volume 103 half-full => byte-full hdd 0.03 0.20:0.02 moving volume 104 half-full => byte-full hdd 0.03 0.15:0.02 moving volume 106 half-full => byte-full hdd 0.03 0.11:0.03 moving volume 107 half-full => byte-full hdd 0.03 0.07:0.03 moving volume 108 half-full => byte-full hdd 0.03 0.03:0.03 moving volume 109 half-full => byte-full hdd 14.00 19.00:10.00 moving volume 11 byte-full => half-full hdd 14.00 18.00:11.00 moving volume 13 byte-full => half-full hdd 14.00 17.00:12.00 moving volume 4 byte-full => half-full hdd 14.00 16.00:13.00 moving volume 5 byte-full => half-full hdd 14.00 15.00:14.00 moving volume 6 byte-full => half-full --- PASS: TestBalanceByDiskUsage (0.00s) === RUN TestBalanceByDiskUsageUsesPhysicalDiskUsageWhenReported hdd 0.52 0.99:0.10 moving volume 2 physical-full => target hdd 0.52 0.99:0.10 moving volume 5 physical-full => target hdd 0.52 0.99:0.10 moving volume 1 physical-full => target hdd 0.52 0.99:0.10 moving volume 3 physical-full => target hdd 0.52 0.99:0.10 moving volume 4 physical-full => target --- PASS: TestBalanceByDiskUsageUsesPhysicalDiskUsageWhenReported (0.00s) === RUN TestBalanceByDiskUsageMixedReportingFallsBackToDataSize hdd 6.67 19.00:1.00 moving volume 104 new-full => new-target hdd 6.67 18.00:2.00 moving volume 112 new-full => new-target hdd 6.67 17.00:2.00 moving volume 102 new-full => old-empty hdd 6.67 16.00:3.00 moving volume 113 new-full => old-empty hdd 6.67 15.00:3.00 moving volume 116 new-full => new-target hdd 6.67 14.00:4.00 moving volume 120 new-full => new-target hdd 6.67 13.00:4.00 moving volume 110 new-full => old-empty hdd 6.67 12.00:5.00 moving volume 105 new-full => old-empty hdd 6.67 11.00:5.00 moving volume 103 new-full => new-target hdd 6.67 10.00:6.00 moving volume 106 new-full => new-target hdd 6.67 9.00:6.00 moving volume 111 new-full => old-empty --- PASS: TestBalanceByDiskUsageMixedReportingFallsBackToDataSize (0.00s) === RUN TestBalanceSkipsPhysicallyFullTarget hdd 0.01 2.33:0.00 moving volume 7 source => disk-full hdd 0.01 1.50:0.00 moving volume 1 source => disk-full hdd 0.01 1.00:0.00 moving volume 4 source => disk-full hdd 0.01 0.67:0.00 moving volume 5 source => disk-full hdd 0.01 0.43:0.01 moving volume 8 source => disk-full hdd 0.01 0.25:0.01 moving volume 2 source => disk-full hdd 0.01 2.33:0.00 moving volume 8 source => disk-full hdd 0.01 1.50:0.00 moving volume 5 source => disk-full hdd 0.01 1.00:0.00 moving volume 1 source => disk-full hdd 0.01 0.67:0.00 moving volume 2 source => disk-full hdd 0.01 0.43:0.01 moving volume 6 source => disk-full hdd 0.01 0.25:0.01 moving volume 3 source => disk-full --- PASS: TestBalanceSkipsPhysicallyFullTarget (0.00s) === RUN TestBalanceGateSteersToEmptierDisk hdd 0.00 2.33:0.00 moving volume 8 source => disk-empty hdd 0.00 1.50:0.00 moving volume 2 source => disk-empty hdd 0.00 1.00:0.00 moving volume 3 source => disk-empty --- PASS: TestBalanceGateSteersToEmptierDisk (0.00s) === RUN TestBalanceVolumesPerExec hdd 0.33 1.00:0.10 moving volume 6 full => empty --- PASS: TestBalanceVolumesPerExec (0.00s) === RUN TestVolumeSelection collect volumes quiet for: 0 seconds --- PASS: TestVolumeSelection (0.04s) === RUN TestDeleteEmptySelection --- PASS: TestDeleteEmptySelection (0.04s) === RUN TestSplitCSVSet === RUN TestSplitCSVSet/empty_input_is_empty_set_(no_filter) === RUN TestSplitCSVSet/whitespace_only_is_empty_set_(no_filter) === RUN TestSplitCSVSet/commas_only_is_empty_set_(no_filter) === RUN TestSplitCSVSet/whitespace_and_commas_only_is_empty_set_(no_filter) === RUN TestSplitCSVSet/single === RUN TestSplitCSVSet/multi === RUN TestSplitCSVSet/trims_whitespace === RUN TestSplitCSVSet/skips_empty_items --- PASS: TestSplitCSVSet (0.00s) --- PASS: TestSplitCSVSet/empty_input_is_empty_set_(no_filter) (0.00s) --- PASS: TestSplitCSVSet/whitespace_only_is_empty_set_(no_filter) (0.00s) --- PASS: TestSplitCSVSet/commas_only_is_empty_set_(no_filter) (0.00s) --- PASS: TestSplitCSVSet/whitespace_and_commas_only_is_empty_set_(no_filter) (0.00s) --- PASS: TestSplitCSVSet/single (0.00s) --- PASS: TestSplitCSVSet/multi (0.00s) --- PASS: TestSplitCSVSet/trims_whitespace (0.00s) --- PASS: TestSplitCSVSet/skips_empty_items (0.00s) === RUN TestCollectVolumeServersByDcRackNode_RackFilter --- PASS: TestCollectVolumeServersByDcRackNode_RackFilter (0.00s) === RUN TestCollectVolumeServersByDcRackNode_NodeFilter --- PASS: TestCollectVolumeServersByDcRackNode_NodeFilter (0.00s) === RUN TestDoVolumeCheckDiskDoesNotResurrectAbsentNeedle --- PASS: TestDoVolumeCheckDiskDoesNotResurrectAbsentNeedle (0.21s) === RUN TestDoVolumeCheckDiskResurrectSkipsVacuumedTarget --- PASS: TestDoVolumeCheckDiskResurrectSkipsVacuumedTarget (0.04s) === RUN TestDoVolumeCheckDiskResurrectQueuesProvenMissingWrite --- PASS: TestDoVolumeCheckDiskResurrectQueuesProvenMissingWrite (0.02s) === RUN TestShouldSkipVolume === RUN TestShouldSkipVolume/identical_volumes_should_be_skipped === RUN TestShouldSkipVolume/different_file_counts_should_not_be_skipped === RUN TestShouldSkipVolume/different_delete_counts_with_syncDeletions_enabled_should_not_be_skipped === RUN TestShouldSkipVolume/different_delete_counts_with_syncDeletions_disabled_should_be_skipped_if_file_counts_match === RUN TestShouldSkipVolume/volumes_with_zero_file_counts_should_be_skipped === RUN TestShouldSkipVolume/volumes_with_zero_and_non-zero_file_counts_should_not_be_skipped === RUN TestShouldSkipVolume/recently_modified_volumes_with_same_file_counts_should_be_skipped === RUN TestShouldSkipVolume/one_volume_modified_before_pulse_cutoff_with_different_file_counts_should_not_be_skipped === RUN TestShouldSkipVolume/different_modification_times_with_same_file_counts_should_be_skipped === RUN TestShouldSkipVolume/volumes_modified_exactly_at_pulse_cutoff_boundary_with_different_counts_should_not_be_skipped === RUN TestShouldSkipVolume/volumes_modified_just_after_pulse_cutoff_boundary_with_same_counts_should_be_skipped === RUN TestShouldSkipVolume/large_file_count_difference_with_old_modification_time_should_not_be_skipped === RUN TestShouldSkipVolume/both_volumes_modified_after_pulse_cutoff_with_same_file_counts_should_be_skipped --- PASS: TestShouldSkipVolume (0.00s) --- PASS: TestShouldSkipVolume/identical_volumes_should_be_skipped (0.00s) --- PASS: TestShouldSkipVolume/different_file_counts_should_not_be_skipped (0.00s) --- PASS: TestShouldSkipVolume/different_delete_counts_with_syncDeletions_enabled_should_not_be_skipped (0.00s) --- PASS: TestShouldSkipVolume/different_delete_counts_with_syncDeletions_disabled_should_be_skipped_if_file_counts_match (0.00s) --- PASS: TestShouldSkipVolume/volumes_with_zero_file_counts_should_be_skipped (0.00s) --- PASS: TestShouldSkipVolume/volumes_with_zero_and_non-zero_file_counts_should_not_be_skipped (0.00s) --- PASS: TestShouldSkipVolume/recently_modified_volumes_with_same_file_counts_should_be_skipped (0.00s) --- PASS: TestShouldSkipVolume/one_volume_modified_before_pulse_cutoff_with_different_file_counts_should_not_be_skipped (0.00s) --- PASS: TestShouldSkipVolume/different_modification_times_with_same_file_counts_should_be_skipped (0.00s) --- PASS: TestShouldSkipVolume/volumes_modified_exactly_at_pulse_cutoff_boundary_with_different_counts_should_not_be_skipped (0.00s) --- PASS: TestShouldSkipVolume/volumes_modified_just_after_pulse_cutoff_boundary_with_same_counts_should_be_skipped (0.00s) --- PASS: TestShouldSkipVolume/large_file_count_difference_with_old_modification_time_should_not_be_skipped (0.00s) --- PASS: TestShouldSkipVolume/both_volumes_modified_after_pulse_cutoff_with_same_file_counts_should_be_skipped (0.00s) === RUN TestVolumeCheckDiskHelperMethods --- PASS: TestVolumeCheckDiskHelperMethods (0.00s) === RUN TestIsEmptyVolumeDeleteCandidateCollectionPattern === RUN TestIsEmptyVolumeDeleteCandidateCollectionPattern/empty_pattern_matches_named_collection === RUN TestIsEmptyVolumeDeleteCandidateCollectionPattern/wildcard_matches_collection === RUN TestIsEmptyVolumeDeleteCandidateCollectionPattern/wildcard_rejects_collection === RUN TestIsEmptyVolumeDeleteCandidateCollectionPattern/default_pattern_matches_empty_collection === RUN TestIsEmptyVolumeDeleteCandidateCollectionPattern/default_pattern_rejects_named_collection --- PASS: TestIsEmptyVolumeDeleteCandidateCollectionPattern (0.00s) --- PASS: TestIsEmptyVolumeDeleteCandidateCollectionPattern/empty_pattern_matches_named_collection (0.00s) --- PASS: TestIsEmptyVolumeDeleteCandidateCollectionPattern/wildcard_matches_collection (0.00s) --- PASS: TestIsEmptyVolumeDeleteCandidateCollectionPattern/wildcard_rejects_collection (0.00s) --- PASS: TestIsEmptyVolumeDeleteCandidateCollectionPattern/default_pattern_matches_empty_collection (0.00s) --- PASS: TestIsEmptyVolumeDeleteCandidateCollectionPattern/default_pattern_rejects_named_collection (0.00s) === RUN TestMatchCollectionPattern === RUN TestMatchCollectionPattern/empty_pattern_matches_any_collection === RUN TestMatchCollectionPattern/empty_pattern_matches_empty_collection === RUN TestMatchCollectionPattern/default_pattern_matches_empty_collection === RUN TestMatchCollectionPattern/default_pattern_rejects_named_collection === RUN TestMatchCollectionPattern/exact_match === RUN TestMatchCollectionPattern/exact_mismatch === RUN TestMatchCollectionPattern/prefix_wildcard_match === RUN TestMatchCollectionPattern/prefix_wildcard_mismatch === RUN TestMatchCollectionPattern/single_char_wildcard_match === RUN TestMatchCollectionPattern/single_char_wildcard_mismatch --- PASS: TestMatchCollectionPattern (0.00s) --- PASS: TestMatchCollectionPattern/empty_pattern_matches_any_collection (0.00s) --- PASS: TestMatchCollectionPattern/empty_pattern_matches_empty_collection (0.00s) --- PASS: TestMatchCollectionPattern/default_pattern_matches_empty_collection (0.00s) --- PASS: TestMatchCollectionPattern/default_pattern_rejects_named_collection (0.00s) --- PASS: TestMatchCollectionPattern/exact_match (0.00s) --- PASS: TestMatchCollectionPattern/exact_mismatch (0.00s) --- PASS: TestMatchCollectionPattern/prefix_wildcard_match (0.00s) --- PASS: TestMatchCollectionPattern/prefix_wildcard_mismatch (0.00s) --- PASS: TestMatchCollectionPattern/single_char_wildcard_match (0.00s) --- PASS: TestMatchCollectionPattern/single_char_wildcard_mismatch (0.00s) === RUN TestReserveTargetSpreadsAcrossServers --- PASS: TestReserveTargetSpreadsAcrossServers (0.10s) === RUN TestFixUnderReplicatedVolumesInParallel --- PASS: TestFixUnderReplicatedVolumesInParallel (0.00s) === RUN TestSatisfyReplicaPlacementComplicated replication: 100 expected false name: test 100 negative replication: 100 expected true name: test 100 positive replication: 022 expected true name: test 022 positive replication: 022 expected false name: test 022 negative replication: 210 expected true name: test 210 moved from 200 positive replication: 210 expected false name: test 210 moved from 200 negative extra dc replication: 210 expected false name: test 210 moved from 200 negative extra data node --- PASS: TestSatisfyReplicaPlacementComplicated (0.00s) === RUN TestSatisfyReplicaPlacement01x replication: 011 expected true name: test 011 same existing rack replication: 011 expected false name: test 011 negative replication: 011 expected true name: test 011 different existing racks replication: 011 expected false name: test 011 different existing racks negative --- PASS: TestSatisfyReplicaPlacement01x (0.00s) === RUN TestSatisfyReplicaPlacement00x replication: 001 expected true name: test 001 replication: 002 expected true name: test 002 positive replication: 002 expected false name: test 002 negative, repeat the same node replication: 002 expected false name: test 002 negative, enough node already --- PASS: TestSatisfyReplicaPlacement00x (0.00s) === RUN TestSatisfyReplicaPlacement100 replication: 100 expected true name: test 100 --- PASS: TestSatisfyReplicaPlacement100 (0.00s) === RUN TestMisplacedChecking replication: 001 expected true name: test 001 replication: 010 expected false name: test 010 replication: 011 expected false name: test 011 replication: 110 expected true name: test 110 replication: 100 expected true name: test 100 --- PASS: TestMisplacedChecking (0.02s) === RUN TestPickingMisplacedVolumeToDelete replication: 001 name: test 001 command_volume_fix_replication_test.go:435: test 001: picked dn2 001 replication: 100 name: test 100 command_volume_fix_replication_test.go:435: test 100: picked dn2 100 --- PASS: TestPickingMisplacedVolumeToDelete (0.02s) === RUN TestPickOneReplicaToDeleteSkipsReadOnlySurvivor === RUN TestPickOneReplicaToDeleteSkipsReadOnlySurvivor/refuses_to_delete_the_only_writable_replica_while_a_read-only_survivor_remains === RUN TestPickOneReplicaToDeleteSkipsReadOnlySurvivor/refuses_when_all_replicas_are_read-only === RUN TestPickOneReplicaToDeleteSkipsReadOnlySurvivor/trims_the_smallest_writable_replica_and_never_the_read-only_one --- PASS: TestPickOneReplicaToDeleteSkipsReadOnlySurvivor (0.00s) --- PASS: TestPickOneReplicaToDeleteSkipsReadOnlySurvivor/refuses_to_delete_the_only_writable_replica_while_a_read-only_survivor_remains (0.00s) --- PASS: TestPickOneReplicaToDeleteSkipsReadOnlySurvivor/refuses_when_all_replicas_are_read-only (0.00s) --- PASS: TestPickOneReplicaToDeleteSkipsReadOnlySurvivor/trims_the_smallest_writable_replica_and_never_the_read-only_one (0.00s) === RUN TestPickOneMisplacedVolumeRelocatesReadOnlyReplicas --- PASS: TestPickOneMisplacedVolumeRelocatesReadOnlyReplicas (0.00s) === RUN TestPickOneReplicaToDeletePreservesPlacement --- PASS: TestPickOneReplicaToDeletePreservesPlacement (0.00s) === RUN TestSatisfyReplicaCurrentLocation === RUN TestSatisfyReplicaCurrentLocation/test_001 === RUN TestSatisfyReplicaCurrentLocation/test_010 === RUN TestSatisfyReplicaCurrentLocation/test_011 === RUN TestSatisfyReplicaCurrentLocation/test_110 === RUN TestSatisfyReplicaCurrentLocation/test_100 --- PASS: TestSatisfyReplicaCurrentLocation (0.00s) --- PASS: TestSatisfyReplicaCurrentLocation/test_001 (0.00s) --- PASS: TestSatisfyReplicaCurrentLocation/test_010 (0.00s) --- PASS: TestSatisfyReplicaCurrentLocation/test_011 (0.00s) --- PASS: TestSatisfyReplicaCurrentLocation/test_110 (0.00s) --- PASS: TestSatisfyReplicaCurrentLocation/test_100 (0.00s) === RUN TestClassifyReplicaSet === RUN TestClassifyReplicaSet/110_misplaced_without_surplus_adds_before_trimming === RUN TestClassifyReplicaSet/110_misplaced_with_surplus_trims === RUN TestClassifyReplicaSet/010_missing_a_replica_adds === RUN TestClassifyReplicaSet/010_surplus_count_but_missing_rack_spread_adds === RUN TestClassifyReplicaSet/010_well_placed_needs_nothing === RUN TestClassifyReplicaSet/000_duplicate_copy_trims --- PASS: TestClassifyReplicaSet (0.00s) --- PASS: TestClassifyReplicaSet/110_misplaced_without_surplus_adds_before_trimming (0.00s) --- PASS: TestClassifyReplicaSet/110_misplaced_with_surplus_trims (0.00s) --- PASS: TestClassifyReplicaSet/010_missing_a_replica_adds (0.00s) --- PASS: TestClassifyReplicaSet/010_surplus_count_but_missing_rack_spread_adds (0.00s) --- PASS: TestClassifyReplicaSet/010_well_placed_needs_nothing (0.00s) --- PASS: TestClassifyReplicaSet/000_duplicate_copy_trims (0.00s) === RUN TestParsing --- PASS: TestParsing (1.48s) === RUN TestWriteDataNodeInfo_SplitsCollapsedDisksByPhysicalDiskId --- PASS: TestWriteDataNodeInfo_SplitsCollapsedDisksByPhysicalDiskId (0.00s) === RUN TestWriteTopologyInfo_PrintsParentHeadersOnce --- PASS: TestWriteTopologyInfo_PrintsParentHeadersOnce (0.00s) === RUN TestFindDuplicateVolumeIds === RUN TestFindDuplicateVolumeIds/flags_id_shared_across_collections === RUN TestFindDuplicateVolumeIds/replicas_of_one_volume_are_not_duplicates === RUN TestFindDuplicateVolumeIds/default_empty_collection_counts_as_its_own_collection === RUN TestFindDuplicateVolumeIds/normal_volume_and_ec_shard_sharing_an_id_across_collections --- PASS: TestFindDuplicateVolumeIds (0.00s) --- PASS: TestFindDuplicateVolumeIds/flags_id_shared_across_collections (0.00s) --- PASS: TestFindDuplicateVolumeIds/replicas_of_one_volume_are_not_duplicates (0.00s) --- PASS: TestFindDuplicateVolumeIds/default_empty_collection_counts_as_its_own_collection (0.00s) --- PASS: TestFindDuplicateVolumeIds/normal_volume_and_ec_shard_sharing_an_id_across_collections (0.00s) === RUN TestWriteDuplicateVolumeIdWarning --- PASS: TestWriteDuplicateVolumeIdWarning (0.00s) === RUN TestCollectVolumeMarkTargetsByCollection --- PASS: TestCollectVolumeMarkTargetsByCollection (0.00s) === RUN TestCollectVolumeMarkTargetsByCollectionMatchesDefaultCollection --- PASS: TestCollectVolumeMarkTargetsByCollectionMatchesDefaultCollection (0.00s) === RUN TestCollectVolumeMarkTargetsByCollectionRejectsEmptyCollection --- PASS: TestCollectVolumeMarkTargetsByCollectionRejectsEmptyCollection (0.00s) === RUN TestCollectVolumeMarkTargetsByCollectionReportsMissingCollection --- PASS: TestCollectVolumeMarkTargetsByCollectionReportsMissingCollection (0.00s) === RUN TestVolumeMarkRejectsCollectionWithNodeOrVolumeId --- PASS: TestVolumeMarkRejectsCollectionWithNodeOrVolumeId (0.00s) === RUN TestVolumeMarkRejectsEmptyCollectionFlag --- PASS: TestVolumeMarkRejectsEmptyCollectionFlag (0.00s) === RUN TestEvaluateMergedVolume --- PASS: TestEvaluateMergedVolume (0.00s) === RUN TestMergeNeedleStreamsOrdersByTimestamp --- PASS: TestMergeNeedleStreamsOrdersByTimestamp (0.00s) === RUN TestMergeNeedleStreamsDoesNotDeduplicateAcrossWindows --- PASS: TestMergeNeedleStreamsDoesNotDeduplicateAcrossWindows (0.00s) === RUN TestMergeNeedleStreamsSameStreamDuplicates --- PASS: TestMergeNeedleStreamsSameStreamDuplicates (0.00s) === RUN TestMergeNeedleStreamsWithEmptyStream --- PASS: TestMergeNeedleStreamsWithEmptyStream (0.00s) === RUN TestMergeNeedleStreamsComplexDuplication --- PASS: TestMergeNeedleStreamsComplexDuplication (0.00s) === RUN TestMergeNeedleStreamsTimeWindowDeduplication --- PASS: TestMergeNeedleStreamsTimeWindowDeduplication (0.00s) === RUN TestMergeNeedleStreamsSingleStream --- PASS: TestMergeNeedleStreamsSingleStream (0.00s) === RUN TestMergeNeedleStreamsLargeIDs --- PASS: TestMergeNeedleStreamsLargeIDs (0.00s) === RUN TestMergeNeedleStreamsLastModifiedFallback --- PASS: TestMergeNeedleStreamsLastModifiedFallback (0.00s) === RUN TestMergeWorkflowValidation command_volume_merge_test.go:455: Volume merge workflow validated: 9 stages command_volume_merge_test.go:457: 2_validate_replicas: Verify at least 2 replicas exist and are healthy command_volume_merge_test.go:457: 3_allocate_temporary: Create temporary merge volume on third location (not a current replica) command_volume_merge_test.go:457: 4_mark_readonly: Mark all original replicas as readonly command_volume_merge_test.go:457: 6_copy_merged: Copy merged volume back to each original replica location command_volume_merge_test.go:457: 7_delete_temporary: Delete the temporary merge volume from the third location command_volume_merge_test.go:457: 8_restore_writable: Restore writable state for replicas that were originally writable command_volume_merge_test.go:457: 9_verify_completion: Log completion status to user command_volume_merge_test.go:457: 1_collect_replicas: Query master to find all replicas of the target volume command_volume_merge_test.go:457: 5_tail_and_merge: Tail all replica needles and merge by timestamp, deduplicating --- PASS: TestMergeWorkflowValidation (0.00s) === RUN TestNeedleBlobFromNeedleRoundTrip --- PASS: TestNeedleBlobFromNeedleRoundTrip (0.00s) === RUN TestMergeEdgeCaseHandling command_volume_merge_test.go:554: ✓ Edge case handled: duplicate_needles_same_stream command_volume_merge_test.go:554: ✓ Edge case handled: empty_replica_stream command_volume_merge_test.go:554: ✓ Edge case handled: large_volume_memory_efficiency command_volume_merge_test.go:554: ✓ Edge case handled: target_server_allocation_failure command_volume_merge_test.go:554: ✓ Edge case handled: merge_volume_writeend_failure command_volume_merge_test.go:554: ✓ Edge case handled: replica_already_readonly command_volume_merge_test.go:554: ✓ Edge case handled: network_timeout_during_tail command_volume_merge_test.go:554: ✓ Edge case handled: duplicate_needles_across_streams command_volume_merge_test.go:554: ✓ Edge case handled: different_needle_metadata command_volume_merge_test.go:554: ✓ Edge case handled: concurrent_writes_prevented command_volume_merge_test.go:561: All 10 edge cases are handled --- PASS: TestMergeEdgeCaseHandling (0.00s) === RUN TestVolumeServerEvacuate moving volume collection0_15 192.168.1.4:8080 => 192.168.1.2:8080 moving volume collection0_21 192.168.1.4:8080 => 192.168.1.6:8080 moving volume collection0_22 192.168.1.4:8080 => 192.168.1.2:8080 moving volume collection0_23 192.168.1.4:8080 => 192.168.1.2:8080 moving volume collection0_24 192.168.1.4:8080 => 192.168.1.2:8080 moving volume collection0_25 192.168.1.4:8080 => 192.168.1.2:8080 moving volume 27 192.168.1.4:8080 => 192.168.1.2:8080 moving volume 28 192.168.1.4:8080 => 192.168.1.2:8080 moving volume 29 192.168.1.4:8080 => 192.168.1.2:8080 moving volume 30 192.168.1.4:8080 => 192.168.1.2:8080 moving volume 31 192.168.1.4:8080 => 192.168.1.2:8080 moving volume collection1_33 192.168.1.4:8080 => 192.168.1.2:8080 moving volume collection1_38 192.168.1.4:8080 => 192.168.1.2:8080 moving volume collection1_51 192.168.1.4:8080 => 192.168.1.2:8080 moving volume collection1_52 192.168.1.4:8080 => 192.168.1.2:8080 moving volume collection1_54 192.168.1.4:8080 => 192.168.1.2:8080 moving volume collection1_63 192.168.1.4:8080 => 192.168.1.2:8080 moving volume collection1_69 192.168.1.4:8080 => 192.168.1.2:8080 moving volume collection1_74 192.168.1.4:8080 => 192.168.1.2:8080 moving volume collection1_80 192.168.1.4:8080 => 192.168.1.2:8080 moving volume collection1_84 192.168.1.4:8080 => 192.168.1.2:8080 moving volume collection1_85 192.168.1.4:8080 => 192.168.1.2:8080 moving volume collection1_97 192.168.1.4:8080 => 192.168.1.2:8080 moving volume collection1_98 192.168.1.4:8080 => 192.168.1.2:8080 moving volume collection1_105 192.168.1.4:8080 => 192.168.1.2:8080 moving volume collection1_106 192.168.1.4:8080 => 192.168.1.2:8080 moving volume collection1_112 192.168.1.4:8080 => 192.168.1.2:8080 moving volume collection1_116 192.168.1.4:8080 => 192.168.1.2:8080 moving volume collection1_119 192.168.1.4:8080 => 192.168.1.2:8080 moving volume collection1_128 192.168.1.4:8080 => 192.168.1.2:8080 moving volume collection1_133 192.168.1.4:8080 => 192.168.1.2:8080 moving volume collection1_136 192.168.1.4:8080 => 192.168.1.2:8080 moving volume collection1_138 192.168.1.4:8080 => 192.168.1.2:8080 moving volume collection1_140 192.168.1.4:8080 => 192.168.1.2:8080 moving volume collection1_144 192.168.1.4:8080 => 192.168.1.2:8080 moving volume collection1_161 192.168.1.4:8080 => 192.168.1.2:8080 moving volume collection1_173 192.168.1.4:8080 => 192.168.1.2:8080 moving volume collection1_174 192.168.1.4:8080 => 192.168.1.2:8080 moving volume collection1_197 192.168.1.4:8080 => 192.168.1.2:8080 moving volume collection1_219 192.168.1.4:8080 => 192.168.1.2:8080 moving volume collection2_263 192.168.1.4:8080 => 192.168.1.2:8080 moving volume collection2_272 192.168.1.4:8080 => 192.168.1.2:8080 moving volume collection2_291 192.168.1.4:8080 => 192.168.1.2:8080 moving volume collection2_299 192.168.1.4:8080 => 192.168.1.2:8080 moving volume collection2_301 192.168.1.4:8080 => 192.168.1.2:8080 moving volume collection2_302 192.168.1.4:8080 => 192.168.1.2:8080 moving volume collection2_339 192.168.1.4:8080 => 192.168.1.2:8080 moving volume collection2_345 192.168.1.4:8080 => 192.168.1.2:8080 moving volume collection2_355 192.168.1.4:8080 => 192.168.1.2:8080 moving volume collection2_373 192.168.1.4:8080 => 192.168.1.2:8080 --- PASS: TestVolumeServerEvacuate (0.09s) === RUN TestFindRemoteVolumeInTopology === RUN TestFindRemoteVolumeInTopology/find_existing_remote_volume === RUN TestFindRemoteVolumeInTopology/find_remote_volume_with_different_backend === RUN TestFindRemoteVolumeInTopology/find_remote_volume_on_another_server === RUN TestFindRemoteVolumeInTopology/local_volume_not_found_as_remote === RUN TestFindRemoteVolumeInTopology/non-existent_volume_not_found === RUN TestFindRemoteVolumeInTopology/filter_by_matching_collection_exact === RUN TestFindRemoteVolumeInTopology/filter_by_matching_collection_regex === RUN TestFindRemoteVolumeInTopology/filter_by_non-matching_collection --- PASS: TestFindRemoteVolumeInTopology (0.01s) --- PASS: TestFindRemoteVolumeInTopology/find_existing_remote_volume (0.00s) --- PASS: TestFindRemoteVolumeInTopology/find_remote_volume_with_different_backend (0.00s) --- PASS: TestFindRemoteVolumeInTopology/find_remote_volume_on_another_server (0.00s) --- PASS: TestFindRemoteVolumeInTopology/local_volume_not_found_as_remote (0.00s) --- PASS: TestFindRemoteVolumeInTopology/non-existent_volume_not_found (0.00s) --- PASS: TestFindRemoteVolumeInTopology/filter_by_matching_collection_exact (0.00s) --- PASS: TestFindRemoteVolumeInTopology/filter_by_matching_collection_regex (0.00s) --- PASS: TestFindRemoteVolumeInTopology/filter_by_non-matching_collection (0.00s) === RUN TestFindRemoteVolumeInTopologyInvalidPattern --- PASS: TestFindRemoteVolumeInTopologyInvalidPattern (0.00s) === RUN TestCollectRemoteVolumesWithInfo === RUN TestCollectRemoteVolumesWithInfo/empty_pattern_matches_empty_collection_only === RUN TestCollectRemoteVolumesWithInfo/match_all_collections === RUN TestCollectRemoteVolumesWithInfo/match_specific_collection === RUN TestCollectRemoteVolumesWithInfo/match_collection_prefix === RUN TestCollectRemoteVolumesWithInfo/match_bucket_collection === RUN TestCollectRemoteVolumesWithInfo/no_match --- PASS: TestCollectRemoteVolumesWithInfo (0.00s) --- PASS: TestCollectRemoteVolumesWithInfo/empty_pattern_matches_empty_collection_only (0.00s) --- PASS: TestCollectRemoteVolumesWithInfo/match_all_collections (0.00s) --- PASS: TestCollectRemoteVolumesWithInfo/match_specific_collection (0.00s) --- PASS: TestCollectRemoteVolumesWithInfo/match_collection_prefix (0.00s) --- PASS: TestCollectRemoteVolumesWithInfo/match_bucket_collection (0.00s) --- PASS: TestCollectRemoteVolumesWithInfo/no_match (0.00s) === RUN TestCollectRemoteVolumesWithInfoCaptures --- PASS: TestCollectRemoteVolumesWithInfoCaptures (0.00s) === RUN TestCollectRemoteVolumesWithInfoInvalidPattern --- PASS: TestCollectRemoteVolumesWithInfoInvalidPattern (0.00s) === RUN TestVolumeSelectionByDataCenter collect volumes quiet for: 0 seconds collect volumes quiet for: 0 seconds --- PASS: TestVolumeSelectionByDataCenter (0.05s) === RUN TestFilterLocationsByDataCenter --- PASS: TestFilterLocationsByDataCenter (0.05s) === RUN TestCollectVolumeTierUploadLocationsPrefersTieredReplica --- PASS: TestCollectVolumeTierUploadLocationsPrefersTieredReplica (0.00s) === RUN TestCollectVolumeTierUploadLocationsFiltersCollection --- PASS: TestCollectVolumeTierUploadLocationsFiltersCollection (0.00s) PASS ok github.com/seaweedfs/seaweedfs/weed/shell 25.815s === RUN TestSetDiskStatusSuccess --- PASS: TestSetDiskStatusSuccess (0.03s) === RUN TestSetDiskStatusReportsRepeatedFailures --- PASS: TestSetDiskStatusReportsRepeatedFailures (0.02s) === RUN TestSetDiskStatusRequiresRepeatedSuccessesToRecover --- PASS: TestSetDiskStatusRequiresRepeatedSuccessesToRecover (0.01s) === RUN TestRobinCounter --- PASS: TestRobinCounter (0.00s) === RUN TestUpdateBucketQuotaMetrics I0824 22:18:20.736248 metrics.go:1088 delete bucket metrics, quota-test-bucket: 2 --- PASS: TestUpdateBucketQuotaMetrics (0.02s) === RUN TestECRebuildMetricsRegistered --- PASS: TestECRebuildMetricsRegistered (0.35s) === RUN TestECRebuildCounterIncrement --- PASS: TestECRebuildCounterIncrement (0.01s) === RUN TestReplicationMetricsRegistered --- PASS: TestReplicationMetricsRegistered (0.09s) === RUN TestReplicationCounterIncrement --- PASS: TestReplicationCounterIncrement (0.00s) === RUN TestReplicationTargetsHistogram --- PASS: TestReplicationTargetsHistogram (0.30s) === RUN TestUnderReplicatedVolumesGauge --- PASS: TestUnderReplicatedVolumesGauge (0.00s) === RUN TestReplicationFailuresCounter --- PASS: TestReplicationFailuresCounter (0.00s) === RUN TestBuildInfo metrics_buildinfo_test.go:72: BuildInfo metric: version=4.44, commit=, sizelimit=30GB, goos=linux, goarch=riscv64 --- PASS: TestBuildInfo (0.56s) PASS ok github.com/seaweedfs/seaweedfs/weed/stats 12.804s === RUN TestCalculateExpectedShardSizeWithRealEncoding === RUN TestCalculateExpectedShardSizeWithRealEncoding/5MB_file I0824 22:18:27.123327 ec_encoder.go:138 encodeDatFile /tmp/TestCalculateExpectedShardSizeWithRealEncoding1854665425/001/test_volume.dat size:5242880 with EC context 10+4 (total: 14) disk_location_ec_realworld_test.go:118: ✓ SUCCESS: .dat size 5242880 → actual shard size 1048576 matches calculated size (Small file that needs 1 small block per shard) === RUN TestCalculateExpectedShardSizeWithRealEncoding/10MB_file_(exactly_10_small_blocks) I0824 22:18:28.572643 ec_encoder.go:138 encodeDatFile /tmp/TestCalculateExpectedShardSizeWithRealEncoding1854665425/001/test_volume.dat size:10485760 with EC context 10+4 (total: 14) disk_location_ec_realworld_test.go:118: ✓ SUCCESS: .dat size 10485760 → actual shard size 1048576 matches calculated size (Exactly fits in 1MB small blocks) === RUN TestCalculateExpectedShardSizeWithRealEncoding/15MB_file I0824 22:18:28.852426 ec_encoder.go:138 encodeDatFile /tmp/TestCalculateExpectedShardSizeWithRealEncoding1854665425/001/test_volume.dat size:15728640 with EC context 10+4 (total: 14) disk_location_ec_realworld_test.go:118: ✓ SUCCESS: .dat size 15728640 → actual shard size 2097152 matches calculated size (Requires 2 small blocks per shard) === RUN TestCalculateExpectedShardSizeWithRealEncoding/50MB_file I0824 22:18:29.737312 ec_encoder.go:138 encodeDatFile /tmp/TestCalculateExpectedShardSizeWithRealEncoding1854665425/001/test_volume.dat size:52428800 with EC context 10+4 (total: 14) disk_location_ec_realworld_test.go:118: ✓ SUCCESS: .dat size 52428800 → actual shard size 5242880 matches calculated size (Requires 5 small blocks per shard) === RUN TestCalculateExpectedShardSizeWithRealEncoding/100MB_file I0824 22:18:31.131424 ec_encoder.go:138 encodeDatFile /tmp/TestCalculateExpectedShardSizeWithRealEncoding1854665425/001/test_volume.dat size:104857600 with EC context 10+4 (total: 14) disk_location_ec_realworld_test.go:118: ✓ SUCCESS: .dat size 104857600 → actual shard size 10485760 matches calculated size (Requires 10 small blocks per shard) === RUN TestCalculateExpectedShardSizeWithRealEncoding/512MB_file I0824 22:18:37.221646 ec_encoder.go:138 encodeDatFile /tmp/TestCalculateExpectedShardSizeWithRealEncoding1854665425/001/test_volume.dat size:536870912 with EC context 10+4 (total: 14) disk_location_ec_realworld_test.go:118: ✓ SUCCESS: .dat size 536870912 → actual shard size 54525952 matches calculated size (Requires 52 small blocks per shard (rounded up)) --- PASS: TestCalculateExpectedShardSizeWithRealEncoding (25.24s) --- PASS: TestCalculateExpectedShardSizeWithRealEncoding/5MB_file (1.46s) --- PASS: TestCalculateExpectedShardSizeWithRealEncoding/10MB_file_(exactly_10_small_blocks) (0.30s) --- PASS: TestCalculateExpectedShardSizeWithRealEncoding/15MB_file (0.49s) --- PASS: TestCalculateExpectedShardSizeWithRealEncoding/50MB_file (1.47s) --- PASS: TestCalculateExpectedShardSizeWithRealEncoding/100MB_file (3.62s) --- PASS: TestCalculateExpectedShardSizeWithRealEncoding/512MB_file (17.80s) === RUN TestCalculateExpectedShardSizeEdgeCases === RUN TestCalculateExpectedShardSizeEdgeCases/1_byte_file I0824 22:18:52.266811 ec_encoder.go:138 encodeDatFile /tmp/TestCalculateExpectedShardSizeEdgeCases1522684588/001/1 byte file.dat size:1 with EC context 10+4 (total: 14) disk_location_ec_realworld_test.go:188: ✓ File size 1 → shard size 1048576 (correct) === RUN TestCalculateExpectedShardSizeEdgeCases/1KB_file I0824 22:18:52.618606 ec_encoder.go:138 encodeDatFile /tmp/TestCalculateExpectedShardSizeEdgeCases1522684588/001/1KB file.dat size:1024 with EC context 10+4 (total: 14) disk_location_ec_realworld_test.go:188: ✓ File size 1024 → shard size 1048576 (correct) === RUN TestCalculateExpectedShardSizeEdgeCases/10KB_file I0824 22:18:52.930929 ec_encoder.go:138 encodeDatFile /tmp/TestCalculateExpectedShardSizeEdgeCases1522684588/001/10KB file.dat size:10240 with EC context 10+4 (total: 14) disk_location_ec_realworld_test.go:188: ✓ File size 10240 → shard size 1048576 (correct) === RUN TestCalculateExpectedShardSizeEdgeCases/1MB_file_(1_small_block) I0824 22:18:53.430872 ec_encoder.go:138 encodeDatFile /tmp/TestCalculateExpectedShardSizeEdgeCases1522684588/001/1MB file (1 small block).dat size:1048576 with EC context 10+4 (total: 14) disk_location_ec_realworld_test.go:188: ✓ File size 1048576 → shard size 1048576 (correct) === RUN TestCalculateExpectedShardSizeEdgeCases/1MB_+_1_byte I0824 22:18:53.671091 ec_encoder.go:138 encodeDatFile /tmp/TestCalculateExpectedShardSizeEdgeCases1522684588/001/1MB + 1 byte.dat size:1048577 with EC context 10+4 (total: 14) disk_location_ec_realworld_test.go:188: ✓ File size 1048577 → shard size 1048576 (correct) === RUN TestCalculateExpectedShardSizeEdgeCases/9.9MB_(almost_1_small_block_per_shard) I0824 22:18:54.009996 ec_encoder.go:138 encodeDatFile /tmp/TestCalculateExpectedShardSizeEdgeCases1522684588/001/9.9MB (almost 1 small block per shard).dat size:10358784 with EC context 10+4 (total: 14) disk_location_ec_realworld_test.go:188: ✓ File size 10358784 → shard size 1048576 (correct) === RUN TestCalculateExpectedShardSizeEdgeCases/10.1MB_(just_over_1_small_block_per_shard) I0824 22:18:54.701891 ec_encoder.go:138 encodeDatFile /tmp/TestCalculateExpectedShardSizeEdgeCases1522684588/001/10.1MB (just over 1 small block per shard).dat size:10588160 with EC context 10+4 (total: 14) disk_location_ec_realworld_test.go:188: ✓ File size 10588160 → shard size 2097152 (correct) --- PASS: TestCalculateExpectedShardSizeEdgeCases (3.09s) --- PASS: TestCalculateExpectedShardSizeEdgeCases/1_byte_file (0.35s) --- PASS: TestCalculateExpectedShardSizeEdgeCases/1KB_file (0.31s) --- PASS: TestCalculateExpectedShardSizeEdgeCases/10KB_file (0.49s) --- PASS: TestCalculateExpectedShardSizeEdgeCases/1MB_file_(1_small_block) (0.22s) --- PASS: TestCalculateExpectedShardSizeEdgeCases/1MB_+_1_byte (0.27s) --- PASS: TestCalculateExpectedShardSizeEdgeCases/9.9MB_(almost_1_small_block_per_shard) (0.51s) --- PASS: TestCalculateExpectedShardSizeEdgeCases/10.1MB_(just_over_1_small_block_per_shard) (0.91s) === RUN TestCalculateExpectedShardSize === RUN TestCalculateExpectedShardSize/0_bytes_(empty_file) disk_location_ec_shard_size_test.go:138: ✓ File size: 0 → Shard size: 0 (Empty file has 0 shard size) === RUN TestCalculateExpectedShardSize/Exact_10GB_(1_large_batch) disk_location_ec_shard_size_test.go:138: ✓ File size: 10737418240 → Shard size: 1073741824 (Exactly fits in large blocks) === RUN TestCalculateExpectedShardSize/Exact_20GB_(2_large_batches) disk_location_ec_shard_size_test.go:138: ✓ File size: 21474836480 → Shard size: 2147483648 (2 complete large batches) === RUN TestCalculateExpectedShardSize/Just_under_large_batch_(10GB_-_1_byte) disk_location_ec_shard_size_test.go:138: ✓ File size: 10737418239 → Shard size: 1073741824 (Just under 10GB needs 1024 small blocks) === RUN TestCalculateExpectedShardSize/Just_over_large_batch_(10GB_+_1_byte) disk_location_ec_shard_size_test.go:138: ✓ File size: 10737418241 → Shard size: 1074790400 (Just over 10GB adds 1 small block) === RUN TestCalculateExpectedShardSize/Exact_10MB_(1_small_batch) disk_location_ec_shard_size_test.go:138: ✓ File size: 10485760 → Shard size: 1048576 (Exactly fits in 1 small batch) === RUN TestCalculateExpectedShardSize/Exact_20MB_(2_small_batches) disk_location_ec_shard_size_test.go:138: ✓ File size: 20971520 → Shard size: 2097152 (2 complete small batches) === RUN TestCalculateExpectedShardSize/Just_under_small_batch_(10MB_-_1_byte) disk_location_ec_shard_size_test.go:138: ✓ File size: 10485759 → Shard size: 1048576 (Just under 10MB rounds up to 1 small block) === RUN TestCalculateExpectedShardSize/Just_over_small_batch_(10MB_+_1_byte) disk_location_ec_shard_size_test.go:138: ✓ File size: 10485761 → Shard size: 2097152 (Just over 10MB needs 2 small blocks) === RUN TestCalculateExpectedShardSize/10GB_+_1MB disk_location_ec_shard_size_test.go:138: ✓ File size: 10738466816 → Shard size: 1074790400 (1 large batch + 1MB needs 1 small block) === RUN TestCalculateExpectedShardSize/10GB_+_5MB disk_location_ec_shard_size_test.go:138: ✓ File size: 10742661120 → Shard size: 1074790400 (1 large batch + 5MB rounds up to 1 small block) === RUN TestCalculateExpectedShardSize/10GB_+_15MB disk_location_ec_shard_size_test.go:138: ✓ File size: 10753146880 → Shard size: 1075838976 (1 large batch + 15MB needs 2 small blocks) === RUN TestCalculateExpectedShardSize/11GB_(1_large_batch_+_103_small_blocks) disk_location_ec_shard_size_test.go:138: ✓ File size: 11811160064 → Shard size: 1181745152 (1GB large + 1GB remaining needs 103 small blocks) === RUN TestCalculateExpectedShardSize/5MB_(requires_1_small_block_per_shard) disk_location_ec_shard_size_test.go:138: ✓ File size: 5242880 → Shard size: 1048576 (Small file rounds up to 1MB per shard) === RUN TestCalculateExpectedShardSize/1KB_(minimum_size) disk_location_ec_shard_size_test.go:138: ✓ File size: 1024 → Shard size: 1048576 (Tiny file needs 1 small block) === RUN TestCalculateExpectedShardSize/10.5GB_(mixed) disk_location_ec_shard_size_test.go:138: ✓ File size: 11274289152 → Shard size: 1128267776 (1GB large + 512MB remaining needs 52 small blocks) --- PASS: TestCalculateExpectedShardSize (0.01s) --- PASS: TestCalculateExpectedShardSize/0_bytes_(empty_file) (0.00s) --- PASS: TestCalculateExpectedShardSize/Exact_10GB_(1_large_batch) (0.00s) --- PASS: TestCalculateExpectedShardSize/Exact_20GB_(2_large_batches) (0.00s) --- PASS: TestCalculateExpectedShardSize/Just_under_large_batch_(10GB_-_1_byte) (0.00s) --- PASS: TestCalculateExpectedShardSize/Just_over_large_batch_(10GB_+_1_byte) (0.00s) --- PASS: TestCalculateExpectedShardSize/Exact_10MB_(1_small_batch) (0.00s) --- PASS: TestCalculateExpectedShardSize/Exact_20MB_(2_small_batches) (0.00s) --- PASS: TestCalculateExpectedShardSize/Just_under_small_batch_(10MB_-_1_byte) (0.00s) --- PASS: TestCalculateExpectedShardSize/Just_over_small_batch_(10MB_+_1_byte) (0.00s) --- PASS: TestCalculateExpectedShardSize/10GB_+_1MB (0.00s) --- PASS: TestCalculateExpectedShardSize/10GB_+_5MB (0.00s) --- PASS: TestCalculateExpectedShardSize/10GB_+_15MB (0.00s) --- PASS: TestCalculateExpectedShardSize/11GB_(1_large_batch_+_103_small_blocks) (0.00s) --- PASS: TestCalculateExpectedShardSize/5MB_(requires_1_small_block_per_shard) (0.00s) --- PASS: TestCalculateExpectedShardSize/1KB_(minimum_size) (0.00s) --- PASS: TestCalculateExpectedShardSize/10.5GB_(mixed) (0.00s) === RUN TestShardSizeValidationScenarios === RUN TestShardSizeValidationScenarios/Valid:_exact_match_for_10GB === RUN TestShardSizeValidationScenarios/Invalid:_1_byte_too_small === RUN TestShardSizeValidationScenarios/Invalid:_1_byte_too_large === RUN TestShardSizeValidationScenarios/Valid:_small_file_exact_match === RUN TestShardSizeValidationScenarios/Invalid:_wrong_size_for_small_file --- PASS: TestShardSizeValidationScenarios (0.00s) --- PASS: TestShardSizeValidationScenarios/Valid:_exact_match_for_10GB (0.00s) --- PASS: TestShardSizeValidationScenarios/Invalid:_1_byte_too_small (0.00s) --- PASS: TestShardSizeValidationScenarios/Invalid:_1_byte_too_large (0.00s) --- PASS: TestShardSizeValidationScenarios/Valid:_small_file_exact_match (0.00s) --- PASS: TestShardSizeValidationScenarios/Invalid:_wrong_size_for_small_file (0.00s) === RUN TestIncompleteEcEncodingCleanup === RUN TestIncompleteEcEncodingCleanup/Incomplete_EC:_shards_without_.ecx,_.dat_exists_-_should_cleanup W0824 22:18:55.362408 disk_location_ec.go:456 Found 14 EC shards without .ecx file for volume 100 (incomplete encoding interrupted before .ecx creation), cleaning up... === RUN TestIncompleteEcEncodingCleanup/Distributed_EC:_shards_without_.ecx,_.dat_deleted_-_should_NOT_cleanup === RUN TestIncompleteEcEncodingCleanup/Distributed_EC:_full-size_shards_with_.ecx,_<_10_of_them,_.dat_exists_-_keep W0824 22:18:55.398546 ec_volume.go:206 vif file not found, using defaults, volumeId:102, filename:/tmp/TestIncompleteEcEncodingCleanupDistributed_EC_full-size_shards796856747/001/102.vif W0824 22:18:55.402534 ec_volume.go:206 vif file not found, using defaults, volumeId:102, filename:/tmp/TestIncompleteEcEncodingCleanupDistributed_EC_full-size_shards796856747/001/102.vif === RUN TestIncompleteEcEncodingCleanup/Valid_local_EC:_shards_with_.ecx,_>=_10_shards,_.dat_exists_-_should_load W0824 22:18:55.414559 ec_volume.go:206 vif file not found, using defaults, volumeId:103, filename:/tmp/TestIncompleteEcEncodingCleanupValid_local_EC_shards_with_.ecx3331061714/001/103.vif W0824 22:18:55.419133 ec_volume.go:206 vif file not found, using defaults, volumeId:103, filename:/tmp/TestIncompleteEcEncodingCleanupValid_local_EC_shards_with_.ecx3331061714/001/103.vif === RUN TestIncompleteEcEncodingCleanup/Distributed_EC:_shards_with_.ecx,_.dat_deleted_-_should_load W0824 22:18:55.428475 ec_volume.go:206 vif file not found, using defaults, volumeId:104, filename:/tmp/TestIncompleteEcEncodingCleanupDistributed_EC_shards_with_.ecx916493878/001/104.vif W0824 22:18:55.431296 ec_volume.go:206 vif file not found, using defaults, volumeId:104, filename:/tmp/TestIncompleteEcEncodingCleanupDistributed_EC_shards_with_.ecx916493878/001/104.vif === RUN TestIncompleteEcEncodingCleanup/Incomplete_EC_with_collection:_shards_without_.ecx,_.dat_exists_-_should_cleanup W0824 22:18:55.439441 disk_location_ec.go:456 Found 14 EC shards without .ecx file for volume 105 (incomplete encoding interrupted before .ecx creation), cleaning up... --- PASS: TestIncompleteEcEncodingCleanup (0.09s) --- PASS: TestIncompleteEcEncodingCleanup/Incomplete_EC:_shards_without_.ecx,_.dat_exists_-_should_cleanup (0.03s) --- PASS: TestIncompleteEcEncodingCleanup/Distributed_EC:_shards_without_.ecx,_.dat_deleted_-_should_NOT_cleanup (0.01s) --- PASS: TestIncompleteEcEncodingCleanup/Distributed_EC:_full-size_shards_with_.ecx,_<_10_of_them,_.dat_exists_-_keep (0.01s) --- PASS: TestIncompleteEcEncodingCleanup/Valid_local_EC:_shards_with_.ecx,_>=_10_shards,_.dat_exists_-_should_load (0.01s) --- PASS: TestIncompleteEcEncodingCleanup/Distributed_EC:_shards_with_.ecx,_.dat_deleted_-_should_load (0.01s) --- PASS: TestIncompleteEcEncodingCleanup/Incomplete_EC_with_collection:_shards_without_.ecx,_.dat_exists_-_should_cleanup (0.01s) === RUN TestValidateEcVolume === RUN TestValidateEcVolume/Valid:_.dat_exists_with_10+_shards === RUN TestValidateEcVolume/Keep:_.dat_exists_with_<_10_full-size_shards_(possible_distributed_sole_copies) === RUN TestValidateEcVolume/Valid:_.dat_deleted_(distributed_EC)_with_any_shards === RUN TestValidateEcVolume/Valid:_.dat_deleted_(distributed_EC)_with_no_shards === RUN TestValidateEcVolume/Invalid:_zero-byte_shard_files_should_not_count === RUN TestValidateEcVolume/Keep:_.dat_exists_with_different_size_shards_(inconsistent,_not_trusted_for_deletion) W0824 22:18:55.470580 disk_location_ec.go:533 EC volume 205 shard 1 size 110 != 100; keeping EC shards --- PASS: TestValidateEcVolume (0.03s) --- PASS: TestValidateEcVolume/Valid:_.dat_exists_with_10+_shards (0.00s) --- PASS: TestValidateEcVolume/Keep:_.dat_exists_with_<_10_full-size_shards_(possible_distributed_sole_copies) (0.00s) --- PASS: TestValidateEcVolume/Valid:_.dat_deleted_(distributed_EC)_with_any_shards (0.00s) --- PASS: TestValidateEcVolume/Valid:_.dat_deleted_(distributed_EC)_with_no_shards (0.00s) --- PASS: TestValidateEcVolume/Invalid:_zero-byte_shard_files_should_not_count (0.00s) --- PASS: TestValidateEcVolume/Keep:_.dat_exists_with_different_size_shards_(inconsistent,_not_trusted_for_deletion) (0.01s) === RUN TestRemoveEcVolumeFiles === RUN TestRemoveEcVolumeFiles/Same_directory_for_data_and_index === RUN TestRemoveEcVolumeFiles/Separate_idx_directory --- PASS: TestRemoveEcVolumeFiles (0.02s) --- PASS: TestRemoveEcVolumeFiles/Same_directory_for_data_and_index (0.01s) --- PASS: TestRemoveEcVolumeFiles/Separate_idx_directory (0.01s) === RUN TestEcCleanupWithSeparateIdxDirectory W0824 22:18:55.513424 disk_location_ec.go:456 Found 14 EC shards without .ecx file for volume 400 (incomplete encoding interrupted before .ecx creation), cleaning up... --- PASS: TestEcCleanupWithSeparateIdxDirectory (0.02s) === RUN TestDistributedEcVolumeNoFileDeletion W0824 22:18:55.524676 ec_volume.go:206 vif file not found, using defaults, volumeId:500, filename:/tmp/TestDistributedEcVolumeNoFileDeletion1569362906/001/500.vif disk_location_ec_test.go:675: SUCCESS: Distributed EC volume files preserved (not deleted) --- PASS: TestDistributedEcVolumeNoFileDeletion (0.01s) === RUN TestLoadExistingVolumeSkipsVifWhenEcxPresent === RUN TestLoadExistingVolumeSkipsVifWhenEcxPresent/same-idx-dir === RUN TestLoadExistingVolumeSkipsVifWhenEcxPresent/split-idx-dir --- PASS: TestLoadExistingVolumeSkipsVifWhenEcxPresent (0.01s) --- PASS: TestLoadExistingVolumeSkipsVifWhenEcxPresent/same-idx-dir (0.00s) --- PASS: TestLoadExistingVolumeSkipsVifWhenEcxPresent/split-idx-dir (0.00s) === RUN TestLoadAllEcShardsDeletesStaleZeroSizedShards W0824 22:18:55.540956 disk_location_ec.go:288 removed zero-sized ec shard /tmp/TestLoadAllEcShardsDeletesStaleZeroSizedShards3180295720/001/123.ec00 (residue of a failed operation) --- PASS: TestLoadAllEcShardsDeletesStaleZeroSizedShards (0.00s) === RUN TestLoadAllEcShardsSplitDirZeroSizedCleanup W0824 22:18:55.548174 disk_location_ec.go:288 removed zero-sized ec shard /tmp/TestLoadAllEcShardsSplitDirZeroSizedCleanup3085589852/002/124.ec00 (residue of a failed operation) --- PASS: TestLoadAllEcShardsSplitDirZeroSizedCleanup (0.01s) === RUN TestUnUsedSpace --- PASS: TestUnUsedSpace (0.00s) === RUN TestResolveVolumeIDs --- PASS: TestResolveVolumeIDs (0.00s) === RUN TestValidateEcVolume_PartialDatNextToFullShardsKeeps --- PASS: TestValidateEcVolume_PartialDatNextToFullShardsKeeps (0.01s) === RUN TestValidateEcVolume_InterruptedEncodeReclaimsDat W0824 22:18:55.561243 disk_location_ec.go:556 EC volume 71: 10 shards of 1048576 bytes are smaller than the .dat's full encode (3145728 bytes); reclaiming the complete .dat --- PASS: TestValidateEcVolume_InterruptedEncodeReclaimsDat (0.00s) === RUN TestValidateEcVolume_TransientStatErrorKeeps E0824 22:18:55.564742 file_util.go:90 check /tmp/TestValidateEcVolume_TransientStatErrorKeeps3826944428/001/locked/73.vif: stat /tmp/TestValidateEcVolume_TransientStatErrorKeeps3826944428/001/locked/73.vif: permission denied W0824 22:18:55.569637 volume_info.go:25 can not read /tmp/TestValidateEcVolume_TransientStatErrorKeeps3826944428/001/locked/73.vif W0824 22:18:55.569989 disk_location_ec.go:518 EC volume 73: cannot stat .dat /tmp/TestValidateEcVolume_TransientStatErrorKeeps3826944428/001/locked/73.dat (stat /tmp/TestValidateEcVolume_TransientStatErrorKeeps3826944428/001/locked/73.dat: permission denied); keeping EC shards --- PASS: TestValidateEcVolume_TransientStatErrorKeeps (0.01s) === RUN TestGenerateLevelDbFileStaleWatermarkRebuilds W0824 22:18:55.592233 needle_map_leveldb.go:128 stale watermark 10000 for /tmp/TestGenerateLevelDbFileStaleWatermarkRebuilds1813588811/001/1.ldb (filesize 48); rebuilding leveldb from start --- PASS: TestGenerateLevelDbFileStaleWatermarkRebuilds (0.05s) === RUN TestLevelDbNeedleMapWatermarkAdvances --- PASS: TestLevelDbNeedleMapWatermarkAdvances (1.00s) === RUN TestLevelDbNeedleMap_Concurrency --- PASS: TestLevelDbNeedleMap_Concurrency (0.10s) === RUN TestFastLoadingNeedleMapMetrics I0824 22:18:56.821540 needle_map_metric_test.go:27 FileCount expected 10000 actual 12001 I0824 22:18:56.821719 needle_map_metric_test.go:28 DeletedSize expected 1658 actual 1658 I0824 22:18:56.821764 needle_map_metric_test.go:29 ContentSize expected 10000 actual 10000 I0824 22:18:56.821798 needle_map_metric_test.go:30 DeletedCount expected 1658 actual 3659 I0824 22:18:56.821823 needle_map_metric_test.go:31 MaxFileKey expected 10000 actual 10000 --- PASS: TestFastLoadingNeedleMapMetrics (0.09s) === RUN TestSortedFileNeedleMap_DeleteAppendsTombstone I0824 22:18:56.823588 needle_map_sorted_file.go:26 Start to Generate /tmp/TestSortedFileNeedleMap_DeleteAppendsTombstone637767010/001/v1.sdx from /tmp/TestSortedFileNeedleMap_DeleteAppendsTombstone637767010/001/v1.idx I0824 22:18:56.830602 needle_map_sorted_file.go:28 Finished Generating /tmp/TestSortedFileNeedleMap_DeleteAppendsTombstone637767010/001/v1.sdx from /tmp/TestSortedFileNeedleMap_DeleteAppendsTombstone637767010/001/v1.idx --- PASS: TestSortedFileNeedleMap_DeleteAppendsTombstone (0.01s) === RUN TestRemoteTier_DiskScanLoadsRemoteOnlyVolume I0824 22:18:56.856954 volume_tier.go:31 volume 44 is tiered to test_local_dir.default as obj-1787609936849789628-266823 and read only I0824 22:18:56.857325 volume_loading.go:167 loading volume 44 from remote files:{backend_type:"test_local_dir" backend_id:"default" key:"obj-1787609936849789628-266823" file_size:2960 modified_time:1787609936 extension:".dat"} version:3 bytes_offset:4 E0824 22:18:56.858466 volume_loading.go:233 readSuperBlock remote volume 44: I0824 22:18:56.859593 needle_map_sorted_file.go:26 Start to Generate /tmp/TestRemoteTier_DiskScanLoadsRemoteOnlyVolume3820159032/002/44.sdx from /tmp/TestRemoteTier_DiskScanLoadsRemoteOnlyVolume3820159032/002/44.idx I0824 22:18:56.865667 needle_map_sorted_file.go:28 Finished Generating /tmp/TestRemoteTier_DiskScanLoadsRemoteOnlyVolume3820159032/002/44.sdx from /tmp/TestRemoteTier_DiskScanLoadsRemoteOnlyVolume3820159032/002/44.idx --- PASS: TestRemoteTier_DiskScanLoadsRemoteOnlyVolume (0.04s) === RUN TestRemoteTier_LiveTierUpload_StillReportsToMaster --- PASS: TestRemoteTier_LiveTierUpload_StillReportsToMaster (0.01s) === RUN TestRemoteTier_ReloadUnderDataLock_NoDeadlock I0824 22:18:56.888379 volume_tier.go:31 volume 68 is tiered to test_local_dir.default as obj-1787609936885270076-266823 and read only I0824 22:18:56.888640 volume_loading.go:167 loading volume 68 from remote files:{backend_type:"test_local_dir" backend_id:"default" key:"obj-1787609936885270076-266823" file_size:2656 modified_time:1787609936 extension:".dat"} version:3 bytes_offset:4 E0824 22:18:56.889307 volume_loading.go:233 readSuperBlock remote volume 68: I0824 22:18:56.890059 needle_map_sorted_file.go:26 Start to Generate /tmp/TestRemoteTier_ReloadUnderDataLock_NoDeadlock1089146817/002/68.sdx from /tmp/TestRemoteTier_ReloadUnderDataLock_NoDeadlock1089146817/002/68.idx I0824 22:18:56.899522 needle_map_sorted_file.go:28 Finished Generating /tmp/TestRemoteTier_ReloadUnderDataLock_NoDeadlock1089146817/002/68.sdx from /tmp/TestRemoteTier_ReloadUnderDataLock_NoDeadlock1089146817/002/68.idx --- PASS: TestRemoteTier_ReloadUnderDataLock_NoDeadlock (0.02s) === RUN TestRemoteTier_Move_KeepsRemoteObject I0824 22:18:56.909086 volume_tier.go:31 volume 31 is tiered to test_local_dir.default as obj-1787609936906447209-266823 and read only I0824 22:18:56.909402 volume_loading.go:167 loading volume 31 from remote files:{backend_type:"test_local_dir" backend_id:"default" key:"obj-1787609936906447209-266823" file_size:760 modified_time:1787609936 extension:".dat"} version:3 bytes_offset:4 E0824 22:18:56.910070 volume_loading.go:233 readSuperBlock remote volume 31: I0824 22:18:56.910807 needle_map_sorted_file.go:26 Start to Generate /tmp/TestRemoteTier_Move_KeepsRemoteObject3572004236/002/31.sdx from /tmp/TestRemoteTier_Move_KeepsRemoteObject3572004236/002/31.idx I0824 22:18:56.925687 needle_map_sorted_file.go:28 Finished Generating /tmp/TestRemoteTier_Move_KeepsRemoteObject3572004236/002/31.sdx from /tmp/TestRemoteTier_Move_KeepsRemoteObject3572004236/002/31.idx I0824 22:18:56.928528 volume_write.go:132 removed volume file /tmp/TestRemoteTier_Move_KeepsRemoteObject3572004236/002/31.idx (size=80) --- PASS: TestRemoteTier_Move_KeepsRemoteObject (0.03s) === RUN TestRemoteTier_RealDelete_RemovesRemoteObject I0824 22:18:56.944226 volume_tier.go:31 volume 32 is tiered to test_local_dir.default as obj-1787609936941747896-266823 and read only I0824 22:18:56.944468 volume_loading.go:167 loading volume 32 from remote files:{backend_type:"test_local_dir" backend_id:"default" key:"obj-1787609936941747896-266823" file_size:2024 modified_time:1787609936 extension:".dat"} version:3 bytes_offset:4 E0824 22:18:56.945029 volume_loading.go:233 readSuperBlock remote volume 32: I0824 22:18:56.945563 needle_map_sorted_file.go:26 Start to Generate /tmp/TestRemoteTier_RealDelete_RemovesRemoteObject2397476818/002/32.sdx from /tmp/TestRemoteTier_RealDelete_RemovesRemoteObject2397476818/002/32.idx I0824 22:18:56.949900 needle_map_sorted_file.go:28 Finished Generating /tmp/TestRemoteTier_RealDelete_RemovesRemoteObject2397476818/002/32.sdx from /tmp/TestRemoteTier_RealDelete_RemovesRemoteObject2397476818/002/32.idx I0824 22:18:56.950971 volume_write.go:132 removed volume file /tmp/TestRemoteTier_RealDelete_RemovesRemoteObject2397476818/002/32.idx (size=80) --- PASS: TestRemoteTier_RealDelete_RemovesRemoteObject (0.02s) === RUN TestRemoteTier_Vacuum_DoesNotDeleteRemote I0824 22:18:56.958194 volume_tier.go:31 volume 33 is tiered to test_local_dir.default as obj-1787609936956400891-266823 and read only I0824 22:18:56.958408 volume_loading.go:167 loading volume 33 from remote files:{backend_type:"test_local_dir" backend_id:"default" key:"obj-1787609936956400891-266823" file_size:3096 modified_time:1787609936 extension:".dat"} version:3 bytes_offset:4 E0824 22:18:56.958873 volume_loading.go:233 readSuperBlock remote volume 33: I0824 22:18:56.959346 needle_map_sorted_file.go:26 Start to Generate /tmp/TestRemoteTier_Vacuum_DoesNotDeleteRemote2640522321/002/33.sdx from /tmp/TestRemoteTier_Vacuum_DoesNotDeleteRemote2640522321/002/33.idx I0824 22:18:56.963945 needle_map_sorted_file.go:28 Finished Generating /tmp/TestRemoteTier_Vacuum_DoesNotDeleteRemote2640522321/002/33.sdx from /tmp/TestRemoteTier_Vacuum_DoesNotDeleteRemote2640522321/002/33.idx I0824 22:18:56.968541 volume_tier.go:31 volume 33 is tiered to test_local_dir.default as obj-1787609936956400891-266823 and read only I0824 22:18:56.968732 volume_loading.go:167 loading volume 33 from remote files:{backend_type:"test_local_dir" backend_id:"default" key:"obj-1787609936956400891-266823" file_size:3096 modified_time:1787609936 extension:".dat"} version:3 bytes_offset:4 E0824 22:18:56.969312 volume_loading.go:233 readSuperBlock remote volume 33: --- PASS: TestRemoteTier_Vacuum_DoesNotDeleteRemote (0.03s) === RUN TestRemoteTier_ECEncode_RequiresLocalDat --- PASS: TestRemoteTier_ECEncode_RequiresLocalDat (0.00s) === RUN TestRemoteTier_ECEncodeDecode_AfterDownload I0824 22:18:56.996298 ec_encoder.go:138 encodeDatFile /tmp/TestRemoteTier_ECEncodeDecode_AfterDownload2319987295/002/35.dat size:2376 with EC context 10+4 (total: 14) I0824 22:18:57.126692 ec_encoder.go:108 Rebuilding EC files for /tmp/TestRemoteTier_ECEncodeDecode_AfterDownload2319987295/002/35 with default config I0824 22:18:57.128471 ec_encoder.go:269 rebuilding /tmp/TestRemoteTier_ECEncodeDecode_AfterDownload2319987295/002/35: 10 shards present, 4 missing [10 11 12 13], config 10+4 (total: 14) --- PASS: TestRemoteTier_ECEncodeDecode_AfterDownload (0.38s) === RUN TestCollectHeartbeatClearsMetricsOfDepartedCollection I0824 22:18:57.377133 store.go:922 UnmountVolume 1 disk_id:0 --- PASS: TestCollectHeartbeatClearsMetricsOfDepartedCollection (0.01s) === RUN TestCollectHeartbeatClearsMetricsWhenTheLastVolumeExpires I0824 22:18:57.389048 volume_write.go:132 removed volume file /tmp/TestCollectHeartbeatClearsMetricsWhenTheLastVolumeExpires3239092711/001/dir0/pics_1.dat (size=192) I0824 22:18:57.389559 volume_write.go:132 removed volume file /tmp/TestCollectHeartbeatClearsMetricsWhenTheLastVolumeExpires3239092711/001/dir0/pics_1.idx (size=16) I0824 22:18:57.391037 store.go:581 volume 1 is deleted --- PASS: TestCollectHeartbeatClearsMetricsWhenTheLastVolumeExpires (0.01s) === RUN TestCollectHeartbeatSizesOnlySurvivingVolumes I0824 22:18:57.412813 volume_write.go:132 removed volume file /tmp/TestCollectHeartbeatSizesOnlySurvivingVolumes1637660360/001/dir1/pics_2.dat (size=1400) I0824 22:18:57.414131 volume_write.go:132 removed volume file /tmp/TestCollectHeartbeatSizesOnlySurvivingVolumes1637660360/001/dir1/pics_2.idx (size=48) I0824 22:18:57.419510 store.go:581 volume 2 is deleted --- PASS: TestCollectHeartbeatSizesOnlySurvivingVolumes (0.03s) === RUN TestCollectHeartbeatCountsPast255ReadOnlyVolumes --- PASS: TestCollectHeartbeatCountsPast255ReadOnlyVolumes (0.77s) === RUN TestCollectErasureCodingHeartbeatClearsMetricsOfDepartedCollection I0824 22:18:58.209337 store_ec.go:213 MountEcShards 4242.0 on disk ID 0 I0824 22:18:58.212038 store_ec.go:303 UnmountEcShards 4242.0 disk_id:0 --- PASS: TestCollectErasureCodingHeartbeatClearsMetricsOfDepartedCollection (0.02s) === RUN TestRenameOrCopyFile --- PASS: TestRenameOrCopyFile (0.01s) === RUN TestConsolidateVolumeIndexMovesIdxToIdxDir I0824 22:18:58.241430 store.go:872 mount volume 7 --- PASS: TestConsolidateVolumeIndexMovesIdxToIdxDir (0.02s) === RUN TestConsolidateVolumeIndexNoopWithoutIdxDir I0824 22:18:58.255303 store.go:872 mount volume 8 --- PASS: TestConsolidateVolumeIndexNoopWithoutIdxDir (0.01s) === RUN TestDeleteCollectionNamesTheVolumesItDestroyed I0824 22:18:58.271251 volume_write.go:132 removed volume file /tmp/TestDeleteCollectionNamesTheVolumesItDestroyed2558718002/001/dir0/books_1.dat (size=8) I0824 22:18:58.271535 volume_write.go:132 removed volume file /tmp/TestDeleteCollectionNamesTheVolumesItDestroyed2558718002/001/dir0/books_1.idx (size=0) I0824 22:18:58.272827 volume_write.go:132 removed volume file /tmp/TestDeleteCollectionNamesTheVolumesItDestroyed2558718002/001/dir0/books_2.dat (size=8) I0824 22:18:58.273023 volume_write.go:132 removed volume file /tmp/TestDeleteCollectionNamesTheVolumesItDestroyed2558718002/001/dir0/books_2.idx (size=0) I0824 22:18:58.274850 metrics.go:1101 delete collection metrics, books: 1 --- PASS: TestDeleteCollectionNamesTheVolumesItDestroyed (0.02s) === RUN TestHasFreeDiskLocation === RUN TestHasFreeDiskLocation/low_disk_space_prevents_allocation === RUN TestHasFreeDiskLocation/normal_disk_space_and_available_volume_count_allows_allocation === RUN TestHasFreeDiskLocation/volume_count_at_max_prevents_allocation === RUN TestHasFreeDiskLocation/volume_count_over_max_prevents_allocation === RUN TestHasFreeDiskLocation/volume_count_just_under_max_allows_allocation === RUN TestHasFreeDiskLocation/max_volume_count_is_0_allows_allocation === RUN TestHasFreeDiskLocation/max_volume_count_is_0_but_low_disk_space_prevents_allocation --- PASS: TestHasFreeDiskLocation (0.01s) --- PASS: TestHasFreeDiskLocation/low_disk_space_prevents_allocation (0.00s) --- PASS: TestHasFreeDiskLocation/normal_disk_space_and_available_volume_count_allows_allocation (0.00s) --- PASS: TestHasFreeDiskLocation/volume_count_at_max_prevents_allocation (0.00s) --- PASS: TestHasFreeDiskLocation/volume_count_over_max_prevents_allocation (0.00s) --- PASS: TestHasFreeDiskLocation/volume_count_just_under_max_allows_allocation (0.00s) --- PASS: TestHasFreeDiskLocation/max_volume_count_is_0_allows_allocation (0.00s) --- PASS: TestHasFreeDiskLocation/max_volume_count_is_0_but_low_disk_space_prevents_allocation (0.00s) === RUN TestCollectHeartbeatRespectsLowDiskSpace === RUN TestCollectHeartbeatRespectsLowDiskSpace/low_disk_space === RUN TestCollectHeartbeatRespectsLowDiskSpace/normal_disk_space === RUN TestCollectHeartbeatRespectsLowDiskSpace/low_disk_space_zero_volumes === RUN TestCollectHeartbeatRespectsLowDiskSpace/low_disk_space_with_ec_shards === RUN TestCollectHeartbeatRespectsLowDiskSpace/low_disk_space_with_multiple_ec_volumes --- PASS: TestCollectHeartbeatRespectsLowDiskSpace (0.00s) --- PASS: TestCollectHeartbeatRespectsLowDiskSpace/low_disk_space (0.00s) --- PASS: TestCollectHeartbeatRespectsLowDiskSpace/normal_disk_space (0.00s) --- PASS: TestCollectHeartbeatRespectsLowDiskSpace/low_disk_space_zero_volumes (0.00s) --- PASS: TestCollectHeartbeatRespectsLowDiskSpace/low_disk_space_with_ec_shards (0.00s) --- PASS: TestCollectHeartbeatRespectsLowDiskSpace/low_disk_space_with_multiple_ec_volumes (0.00s) === RUN TestUnmountVolumeRemovesAllDuplicateCopies I0824 22:18:58.297434 store.go:922 UnmountVolume 4242 disk_id:0 I0824 22:18:58.297717 store.go:922 UnmountVolume 4242 disk_id:0 --- PASS: TestUnmountVolumeRemovesAllDuplicateCopies (0.01s) === RUN TestDeleteVolumeRemovesAllDuplicateCopies I0824 22:18:58.312557 volume_write.go:132 removed volume file /tmp/TestDeleteVolumeRemovesAllDuplicateCopies2197882589/001/dir0/4243.dat (size=8) I0824 22:18:58.313080 volume_write.go:132 removed volume file /tmp/TestDeleteVolumeRemovesAllDuplicateCopies2197882589/001/dir0/4243.idx (size=0) I0824 22:18:58.315634 store.go:1014 DeleteVolume 4243 disk_id:0 I0824 22:18:58.316518 volume_write.go:132 removed volume file /tmp/TestDeleteVolumeRemovesAllDuplicateCopies2197882589/001/dir1/4243.dat (size=8) I0824 22:18:58.316878 volume_write.go:132 removed volume file /tmp/TestDeleteVolumeRemovesAllDuplicateCopies2197882589/001/dir1/4243.idx (size=0) I0824 22:18:58.318407 store.go:1014 DeleteVolume 4243 disk_id:0 --- PASS: TestDeleteVolumeRemovesAllDuplicateCopies (0.02s) === RUN TestEc9plus3MultiDiskRebootLoadsAllShards I0824 22:18:58.340296 store_ec_mirror.go:56 ec volume 7 (collection="loki"): mirrored 3 sidecar(s) from /tmp/TestEc9plus3MultiDiskRebootLoadsAllShards1938280383/001/disk0 to /tmp/TestEc9plus3MultiDiskRebootLoadsAllShards1938280383/001/disk1 for same-disk invariant I0824 22:18:58.342920 store_ec_mirror.go:56 ec volume 7 (collection="loki"): mirrored 3 sidecar(s) from /tmp/TestEc9plus3MultiDiskRebootLoadsAllShards1938280383/001/disk0 to /tmp/TestEc9plus3MultiDiskRebootLoadsAllShards1938280383/001/disk2 for same-disk invariant I0824 22:18:58.346424 store_ec_mirror.go:56 ec volume 7 (collection="loki"): mirrored 3 sidecar(s) from /tmp/TestEc9plus3MultiDiskRebootLoadsAllShards1938280383/001/disk0 to /tmp/TestEc9plus3MultiDiskRebootLoadsAllShards1938280383/001/disk3 for same-disk invariant I0824 22:18:58.349057 store_ec_mirror.go:56 ec volume 7 (collection="loki"): mirrored 3 sidecar(s) from /tmp/TestEc9plus3MultiDiskRebootLoadsAllShards1938280383/001/disk0 to /tmp/TestEc9plus3MultiDiskRebootLoadsAllShards1938280383/001/disk4 for same-disk invariant I0824 22:18:58.351334 store_ec_mirror.go:56 ec volume 7 (collection="loki"): mirrored 3 sidecar(s) from /tmp/TestEc9plus3MultiDiskRebootLoadsAllShards1938280383/001/disk0 to /tmp/TestEc9plus3MultiDiskRebootLoadsAllShards1938280383/001/disk5 for same-disk invariant I0824 22:18:58.352886 store_ec_reconcile.go:85 ec volume 7 (collection="loki"): loading orphan shards [loki_7.ec02 loki_7.ec03] on /tmp/TestEc9plus3MultiDiskRebootLoadsAllShards1938280383/001/disk1 against locally-mirrored sidecars I0824 22:18:58.356267 store_ec_reconcile.go:85 ec volume 7 (collection="loki"): loading orphan shards [loki_7.ec04 loki_7.ec05] on /tmp/TestEc9plus3MultiDiskRebootLoadsAllShards1938280383/001/disk2 against locally-mirrored sidecars I0824 22:18:58.358100 store_ec_reconcile.go:85 ec volume 7 (collection="loki"): loading orphan shards [loki_7.ec06 loki_7.ec07] on /tmp/TestEc9plus3MultiDiskRebootLoadsAllShards1938280383/001/disk3 against locally-mirrored sidecars I0824 22:18:58.359782 store_ec_reconcile.go:85 ec volume 7 (collection="loki"): loading orphan shards [loki_7.ec08 loki_7.ec09] on /tmp/TestEc9plus3MultiDiskRebootLoadsAllShards1938280383/001/disk4 against locally-mirrored sidecars I0824 22:18:58.361393 store_ec_reconcile.go:85 ec volume 7 (collection="loki"): loading orphan shards [loki_7.ec10 loki_7.ec11] on /tmp/TestEc9plus3MultiDiskRebootLoadsAllShards1938280383/001/disk5 against locally-mirrored sidecars --- PASS: TestEc9plus3MultiDiskRebootLoadsAllShards (0.05s) === RUN TestEc9plus3ValidateKeepsVolumeWithDat --- PASS: TestEc9plus3ValidateKeepsVolumeWithDat (0.27s) === RUN TestEc9plus3PruneKeepsFullDataShardSet --- PASS: TestEc9plus3PruneKeepsFullDataShardSet (0.22s) === RUN TestMountEcShards_AppliesSourceDiskType I0824 22:18:58.884846 store_ec.go:213 MountEcShards 4242.0 on disk ID 0 --- PASS: TestMountEcShards_AppliesSourceDiskType (0.04s) === RUN TestMountEcShards_FallsBackToLocationDiskTypeWhenEmpty I0824 22:18:58.928888 store_ec.go:213 MountEcShards 4242.0 on disk ID 0 --- PASS: TestMountEcShards_FallsBackToLocationDiskTypeWhenEmpty (0.04s) === RUN TestIssue9478_PartialEcOnSiblingDiskOfHealthyDat store_ec_hybrid_repro_test.go:143: after per-disk EC load on sdf: inMemoryShards=1 newEcShardMessages=1 W0824 22:18:58.969840 store_ec_reconcile.go:314 ec volume 122 (collection="") on /tmp/TestIssue9478_PartialEcOnSiblingDiskOfHealthyDat1776282695/001/sdf has only 1 shards (need 10) while a byte-exact source .dat exists on sibling disk /tmp/TestIssue9478_PartialEcOnSiblingDiskOfHealthyDat1776282695/001/sdd; cleaning up leftover EC files store_ec_hybrid_repro_test.go:157: after Store-level cleanup: shardFileLeft=false ecxLeft=false ecjLeft=false inMemoryShards=0 deletedEcShardMessages=1 --- PASS: TestIssue9478_PartialEcOnSiblingDiskOfHealthyDat (0.03s) === RUN TestIssue9478_ZeroByteSiblingDatKeepsPartialEc W0824 22:18:58.983754 volume_info.go:46 empty volume info file /tmp/TestIssue9478_ZeroByteSiblingDatKeepsPartialEc2646757723/001/sdf/122.vif, treating as non-existent W0824 22:18:58.988633 ec_volume.go:206 vif file not found, using defaults, volumeId:122, filename:/tmp/TestIssue9478_ZeroByteSiblingDatKeepsPartialEc2646757723/001/sdf/122.vif W0824 22:18:58.990389 store_ec_reconcile.go:290 ec volume 122 (collection="") on /tmp/TestIssue9478_ZeroByteSiblingDatKeepsPartialEc2646757723/001/sdf has only 1 shards; sibling .dat on /tmp/TestIssue9478_ZeroByteSiblingDatKeepsPartialEc2646757723/001/sdd is 0 bytes but .vif recorded 0 (need byte-exact match); leaving partial EC in place --- PASS: TestIssue9478_ZeroByteSiblingDatKeepsPartialEc (0.02s) === RUN TestMirrorEcMetadataOnStartup_PhysicallyCopiesSidecars I0824 22:18:59.021118 store_ec_mirror.go:56 ec volume 4121 (collection="video-recordings"): mirrored 3 sidecar(s) from /tmp/TestMirrorEcMetadataOnStartup_PhysicallyCopiesSidecars2151247245/001/data2 to /tmp/TestMirrorEcMetadataOnStartup_PhysicallyCopiesSidecars2151247245/001/data1 for same-disk invariant I0824 22:18:59.025250 store_ec_reconcile.go:85 ec volume 4121 (collection="video-recordings"): loading orphan shards [video-recordings_4121.ec00 video-recordings_4121.ec12] on /tmp/TestMirrorEcMetadataOnStartup_PhysicallyCopiesSidecars2151247245/001/data1 against locally-mirrored sidecars --- PASS: TestMirrorEcMetadataOnStartup_PhysicallyCopiesSidecars (0.04s) === RUN TestMirrorEcMetadataOnStartup_NoOpWhenAlreadyMirrored --- PASS: TestMirrorEcMetadataOnStartup_NoOpWhenAlreadyMirrored (0.02s) === RUN TestMountEcShards_LocatesEcxOnSiblingDisk I0824 22:18:59.066223 store_ec.go:213 MountEcShards 5.6 on disk ID 0 --- PASS: TestMountEcShards_LocatesEcxOnSiblingDisk (0.01s) === RUN TestMountEcShards_SameDiskEcxStillWorks I0824 22:18:59.075366 store_ec.go:213 MountEcShards 7.3 on disk ID 0 --- PASS: TestMountEcShards_SameDiskEcxStillWorks (0.01s) === RUN TestMountEcShards_MissingEcxOnAllDisks --- PASS: TestMountEcShards_MissingEcxOnAllDisks (0.01s) === RUN TestMountEcShards_ZeroByteEcxIsIgnored I0824 22:18:59.095133 store_ec.go:213 MountEcShards 12.7 on disk ID 0 --- PASS: TestMountEcShards_ZeroByteEcxIsIgnored (0.01s) === RUN TestMountEcShards_EmptyEcxMountsSuccessfully I0824 22:18:59.107476 store_ec.go:213 MountEcShards 13.0 on disk ID 0 --- PASS: TestMountEcShards_EmptyEcxMountsSuccessfully (0.01s) === RUN TestIndexEcxOwners_IgnoresZeroByteStub --- PASS: TestIndexEcxOwners_IgnoresZeroByteStub (0.01s) === RUN TestMultiDiskLifecycle_SpreadLayout I0824 22:18:59.133682 store_ec_mirror.go:56 ec volume 42561 (collection="pm-itatiaiucu-01"): mirrored 3 sidecar(s) from /tmp/TestMultiDiskLifecycle_SpreadLayout947647212/001/disk0 to /tmp/TestMultiDiskLifecycle_SpreadLayout947647212/001/disk1 for same-disk invariant I0824 22:18:59.137560 store_ec_mirror.go:56 ec volume 42561 (collection="pm-itatiaiucu-01"): mirrored 3 sidecar(s) from /tmp/TestMultiDiskLifecycle_SpreadLayout947647212/001/disk0 to /tmp/TestMultiDiskLifecycle_SpreadLayout947647212/001/disk2 for same-disk invariant I0824 22:18:59.138556 store_ec_reconcile.go:85 ec volume 42561 (collection="pm-itatiaiucu-01"): loading orphan shards [pm-itatiaiucu-01_42561.ec06 pm-itatiaiucu-01_42561.ec07 pm-itatiaiucu-01_42561.ec08 pm-itatiaiucu-01_42561.ec09 pm-itatiaiucu-01_42561.ec10] on /tmp/TestMultiDiskLifecycle_SpreadLayout947647212/001/disk1 against locally-mirrored sidecars I0824 22:18:59.141173 store_ec_reconcile.go:85 ec volume 42561 (collection="pm-itatiaiucu-01"): loading orphan shards [pm-itatiaiucu-01_42561.ec11 pm-itatiaiucu-01_42561.ec12 pm-itatiaiucu-01_42561.ec13] on /tmp/TestMultiDiskLifecycle_SpreadLayout947647212/001/disk2 against locally-mirrored sidecars I0824 22:18:59.142807 store_ec.go:303 UnmountEcShards 42561.0 disk_id:0 I0824 22:18:59.143087 store_ec.go:303 UnmountEcShards 42561.1 disk_id:0 I0824 22:18:59.143230 store_ec.go:303 UnmountEcShards 42561.2 disk_id:0 I0824 22:18:59.143314 store_ec.go:303 UnmountEcShards 42561.3 disk_id:0 I0824 22:18:59.143389 store_ec.go:303 UnmountEcShards 42561.4 disk_id:0 I0824 22:18:59.143542 store_ec.go:303 UnmountEcShards 42561.5 disk_id:0 I0824 22:18:59.161719 store_ec.go:213 MountEcShards 42561.0 on disk ID 2 --- PASS: TestMultiDiskLifecycle_SpreadLayout (0.05s) === RUN TestMultiDiskLifecycle_SidecarDiskLost store_ec_multidisk_lifecycle_test.go:313: files intact; shards without a reachable .ecx stay unloaded, as required --- PASS: TestMultiDiskLifecycle_SidecarDiskLost (0.01s) === RUN TestLoadEcShardsWhenIndexFilesOnDifferentDisk I0824 22:18:59.191286 store_ec_mirror.go:56 ec volume 1093 (collection="grafana-loki"): mirrored 3 sidecar(s) from /tmp/TestLoadEcShardsWhenIndexFilesOnDifferentDisk2733758364/001/data2 to /tmp/TestLoadEcShardsWhenIndexFilesOnDifferentDisk2733758364/001/data1 for same-disk invariant I0824 22:18:59.193365 store_ec_reconcile.go:85 ec volume 1093 (collection="grafana-loki"): loading orphan shards [grafana-loki_1093.ec00 grafana-loki_1093.ec12] on /tmp/TestLoadEcShardsWhenIndexFilesOnDifferentDisk2733758364/001/data1 against locally-mirrored sidecars --- PASS: TestLoadEcShardsWhenIndexFilesOnDifferentDisk (0.02s) === RUN TestLoadEcShardsOrphanWithoutSiblingEcx --- PASS: TestLoadEcShardsOrphanWithoutSiblingEcx (0.01s) === RUN TestReconcileNoOpWhenEachDiskIsSelfContained --- PASS: TestReconcileNoOpWhenEachDiskIsSelfContained (0.02s) === RUN TestLoadEcShardsWhenOwnerEcxIsInDataDir I0824 22:18:59.245225 store_ec_mirror.go:56 ec volume 4242 (collection="grafana-loki"): mirrored 3 sidecar(s) from /tmp/TestLoadEcShardsWhenOwnerEcxIsInDataDir4210482118/001/data2 to /tmp/TestLoadEcShardsWhenOwnerEcxIsInDataDir4210482118/001/data1 for same-disk invariant I0824 22:18:59.246208 store_ec_reconcile.go:85 ec volume 4242 (collection="grafana-loki"): loading orphan shards [grafana-loki_4242.ec00 grafana-loki_4242.ec12] on /tmp/TestLoadEcShardsWhenOwnerEcxIsInDataDir4210482118/001/data1 against locally-mirrored sidecars --- PASS: TestLoadEcShardsWhenOwnerEcxIsInDataDir (0.02s) === RUN TestLoneVifDoesNotCreatePhantomDat --- PASS: TestLoneVifDoesNotCreatePhantomDat (0.01s) === RUN TestEmptyEcDatStubIsSwept W0824 22:18:59.268373 disk_location.go:193 removing leftover empty .dat stub for EC volume 41 (collection="warp-cal") W0824 22:18:59.268370 disk_location.go:193 removing leftover empty .dat stub for EC volume 41 (collection="warp-cal") --- PASS: TestEmptyEcDatStubIsSwept (0.01s) === RUN TestRemoveEmptyEcDatStubFindsVifInIdxDir W0824 22:18:59.277085 disk_location.go:193 removing leftover empty .dat stub for EC volume 42 (collection="warp-cal") --- PASS: TestRemoveEmptyEcDatStubFindsVifInIdxDir (0.01s) === RUN TestEmptyDatStubNextToEcxDoesNotDeleteShards W0824 22:18:59.286167 disk_location.go:193 removing leftover empty .dat stub for EC volume 87 (collection="warp-rec") --- PASS: TestEmptyDatStubNextToEcxDoesNotDeleteShards (0.01s) === RUN TestEmptyDatWithoutVifDoesNotDeleteShards W0824 22:18:59.294363 ec_volume.go:206 vif file not found, using defaults, volumeId:88, filename:/tmp/TestEmptyDatWithoutVifDoesNotDeleteShards618394032/001/warp-rec_88.vif --- PASS: TestEmptyDatWithoutVifDoesNotDeleteShards (0.01s) === RUN TestEcIndexRecovery_CrossServerOrphan I0824 22:18:59.307339 store_ec_mirror.go:56 ec volume 6190 (collection="video-recordings"): mirrored 3 sidecar(s) from /tmp/TestEcIndexRecovery_CrossServerOrphan4029279087/001/disk0 to /tmp/TestEcIndexRecovery_CrossServerOrphan4029279087/001/disk1 for same-disk invariant --- PASS: TestEcIndexRecovery_CrossServerOrphan (0.02s) === RUN TestCollectEcVolumesMissingIndex_ExcludesCrossDiskOrphan --- PASS: TestCollectEcVolumesMissingIndex_ExcludesCrossDiskOrphan (0.02s) === RUN TestRecoverOneRemoteEcShardInterval_SufficientShards store_ec_recovery_test.go:52: Successfully identified 13 available shards (need 10) --- PASS: TestRecoverOneRemoteEcShardInterval_SufficientShards (0.00s) === RUN TestRecoverOneRemoteEcShardInterval_InsufficientShards store_ec_recovery_test.go:80: Correctly identified insufficient shards: 7 available (need 10) --- PASS: TestRecoverOneRemoteEcShardInterval_InsufficientShards (0.00s) === RUN TestRecoverOneRemoteEcShardInterval_ShardCounting === RUN TestRecoverOneRemoteEcShardInterval_ShardCounting/All_shards_available_except_one store_ec_recovery_test.go:155: Available shards: 13 [0 1 2 3 4 6 7 8 9 10 11 12 13], Missing shards: 1 [5] === RUN TestRecoverOneRemoteEcShardInterval_ShardCounting/Exactly_minimum_shards_(DataShardsCount) store_ec_recovery_test.go:155: Available shards: 10 [0 1 2 3 4 5 6 7 8 9], Missing shards: 4 [10 11 12 13] === RUN TestRecoverOneRemoteEcShardInterval_ShardCounting/One_less_than_minimum store_ec_recovery_test.go:155: Available shards: 9 [0 1 2 3 4 5 6 7 8], Missing shards: 5 [9 10 11 12 13] === RUN TestRecoverOneRemoteEcShardInterval_ShardCounting/Only_half_the_shards store_ec_recovery_test.go:155: Available shards: 7 [1 2 3 4 5 6 7], Missing shards: 7 [0 8 9 10 11 12 13] === RUN TestRecoverOneRemoteEcShardInterval_ShardCounting/All_data_shards_available store_ec_recovery_test.go:155: Available shards: 10 [0 1 2 3 4 5 6 7 8 9], Missing shards: 4 [10 11 12 13] --- PASS: TestRecoverOneRemoteEcShardInterval_ShardCounting (0.01s) --- PASS: TestRecoverOneRemoteEcShardInterval_ShardCounting/All_shards_available_except_one (0.00s) --- PASS: TestRecoverOneRemoteEcShardInterval_ShardCounting/Exactly_minimum_shards_(DataShardsCount) (0.00s) --- PASS: TestRecoverOneRemoteEcShardInterval_ShardCounting/One_less_than_minimum (0.00s) --- PASS: TestRecoverOneRemoteEcShardInterval_ShardCounting/Only_half_the_shards (0.00s) --- PASS: TestRecoverOneRemoteEcShardInterval_ShardCounting/All_data_shards_available (0.00s) === RUN TestRecoverOneRemoteEcShardInterval_ErrorMessage store_ec_recovery_test.go:190: Error message format validated: cannot recover shard 42.7: only 7 shards available [0 1 2 3 4 5 6], need at least 10 (missing: [7 8 9 10 11 12 13]) --- PASS: TestRecoverOneRemoteEcShardInterval_ErrorMessage (0.00s) === RUN TestRecoverOneRemoteEcShardInterval_ReconstructDataSlicing store_ec_recovery_test.go:248: Successfully reconstructed shard with proper buffer slicing --- PASS: TestRecoverOneRemoteEcShardInterval_ReconstructDataSlicing (0.00s) === RUN TestRecoverOneRemoteEcShardInterval_ParityShardRecovery === RUN TestRecoverOneRemoteEcShardInterval_ParityShardRecovery/RecoverParity10 === RUN TestRecoverOneRemoteEcShardInterval_ParityShardRecovery/RecoverParity11 === RUN TestRecoverOneRemoteEcShardInterval_ParityShardRecovery/RecoverParity12 === RUN TestRecoverOneRemoteEcShardInterval_ParityShardRecovery/RecoverParity13 --- PASS: TestRecoverOneRemoteEcShardInterval_ParityShardRecovery (0.00s) --- PASS: TestRecoverOneRemoteEcShardInterval_ParityShardRecovery/RecoverParity10 (0.00s) --- PASS: TestRecoverOneRemoteEcShardInterval_ParityShardRecovery/RecoverParity11 (0.00s) --- PASS: TestRecoverOneRemoteEcShardInterval_ParityShardRecovery/RecoverParity12 (0.00s) --- PASS: TestRecoverOneRemoteEcShardInterval_ParityShardRecovery/RecoverParity13 (0.00s) === RUN TestRecoverOneRemoteEcShardInterval_ConcurrentShardReading store_ec_recovery_test.go:363: Successfully simulated concurrent reading of 13 shards --- PASS: TestRecoverOneRemoteEcShardInterval_ConcurrentShardReading (0.00s) === RUN TestRecoverOneRemoteEcShardInterval_BuggyMaxShardCount store_ec_recovery_test.go:419: Corrected code: 9 shards (correct, insufficient) store_ec_recovery_test.go:420: Buggy code would have counted: 27 shards (incorrect, falsely sufficient) store_ec_recovery_test.go:421: Missing shards: [9 10 11 12 13] --- PASS: TestRecoverOneRemoteEcShardInterval_BuggyMaxShardCount (0.00s) === RUN TestFindEcShardTargetLocation_PinsToEcxOnDisk --- PASS: TestFindEcShardTargetLocation_PinsToEcxOnDisk (0.01s) === RUN TestFindEcShardTargetLocation_PrefersMountedOverEcx --- PASS: TestFindEcShardTargetLocation_PrefersMountedOverEcx (0.01s) === RUN TestFindEcShardTargetLocation_FallsThroughToHddWhenNothingMatches --- PASS: TestFindEcShardTargetLocation_FallsThroughToHddWhenNothingMatches (0.01s) === RUN TestFindEcShardTargetLocation_HonoursUnlimitedDisk --- PASS: TestFindEcShardTargetLocation_HonoursUnlimitedDisk (0.01s) === RUN TestFindEcShardTargetLocation_TightProvisioningKeepsEcxDisk --- PASS: TestFindEcShardTargetLocation_TightProvisioningKeepsEcxDisk (0.01s) === RUN TestCollectHeartbeatDigestsExactlyWhatItReports --- PASS: TestCollectHeartbeatDigestsExactlyWhatItReports (0.01s) === RUN TestCollectHeartbeatDigestsAnEmptyStore --- PASS: TestCollectHeartbeatDigestsAnEmptyStore (0.00s) === RUN TestCollectHeartbeatDigestFollowsVolumeChanges --- PASS: TestCollectHeartbeatDigestFollowsVolumeChanges (0.01s) === RUN TestLoadBalancingDistribution I0824 22:18:59.439403 store.go:329 In dir /tmp/TestLoadBalancingDistribution2394529641/001/dir0 (disk ID 0) adds volume:1 collection: replicaPlacement:000 ttl: I0824 22:18:59.442318 store.go:334 add volume 1 on disk ID 0 I0824 22:18:59.442543 store.go:329 In dir /tmp/TestLoadBalancingDistribution2394529641/001/dir1 (disk ID 1) adds volume:2 collection: replicaPlacement:000 ttl: I0824 22:18:59.444005 store.go:334 add volume 2 on disk ID 1 I0824 22:18:59.444304 store.go:329 In dir /tmp/TestLoadBalancingDistribution2394529641/001/dir2 (disk ID 2) adds volume:3 collection: replicaPlacement:000 ttl: I0824 22:18:59.445683 store.go:334 add volume 3 on disk ID 2 I0824 22:18:59.445873 store.go:329 In dir /tmp/TestLoadBalancingDistribution2394529641/001/dir0 (disk ID 0) adds volume:4 collection: replicaPlacement:000 ttl: I0824 22:18:59.446779 store.go:334 add volume 4 on disk ID 0 I0824 22:18:59.446916 store.go:329 In dir /tmp/TestLoadBalancingDistribution2394529641/001/dir1 (disk ID 1) adds volume:5 collection: replicaPlacement:000 ttl: I0824 22:18:59.448106 store.go:334 add volume 5 on disk ID 1 I0824 22:18:59.448277 store.go:329 In dir /tmp/TestLoadBalancingDistribution2394529641/001/dir2 (disk ID 2) adds volume:6 collection: replicaPlacement:000 ttl: I0824 22:18:59.449526 store.go:334 add volume 6 on disk ID 2 I0824 22:18:59.449681 store.go:329 In dir /tmp/TestLoadBalancingDistribution2394529641/001/dir0 (disk ID 0) adds volume:7 collection: replicaPlacement:000 ttl: I0824 22:18:59.450722 store.go:334 add volume 7 on disk ID 0 I0824 22:18:59.450879 store.go:329 In dir /tmp/TestLoadBalancingDistribution2394529641/001/dir1 (disk ID 1) adds volume:8 collection: replicaPlacement:000 ttl: I0824 22:18:59.451940 store.go:334 add volume 8 on disk ID 1 I0824 22:18:59.452100 store.go:329 In dir /tmp/TestLoadBalancingDistribution2394529641/001/dir2 (disk ID 2) adds volume:9 collection: replicaPlacement:000 ttl: I0824 22:18:59.453005 store.go:334 add volume 9 on disk ID 2 --- PASS: TestLoadBalancingDistribution (0.03s) === RUN TestLocalVolumesLen === RUN TestLocalVolumesLen/all_local_volumes === RUN TestLocalVolumesLen/all_remote_volumes === RUN TestLocalVolumesLen/mixed_local_and_remote === RUN TestLocalVolumesLen/no_volumes --- PASS: TestLocalVolumesLen (0.00s) --- PASS: TestLocalVolumesLen/all_local_volumes (0.00s) --- PASS: TestLocalVolumesLen/all_remote_volumes (0.00s) --- PASS: TestLocalVolumesLen/mixed_local_and_remote (0.00s) --- PASS: TestLocalVolumesLen/no_volumes (0.00s) === RUN TestVolumeLoadBalancing === RUN TestVolumeLoadBalancing/even_distribution_across_empty_locations I0824 22:18:59.463176 store.go:329 In dir /tmp/TestVolumeLoadBalancingeven_distribution_across_empty_locations3674849216/001/dir0 (disk ID 0) adds volume:1 collection: replicaPlacement:000 ttl: I0824 22:18:59.464979 store.go:334 add volume 1 on disk ID 0 I0824 22:18:59.465295 store.go:329 In dir /tmp/TestVolumeLoadBalancingeven_distribution_across_empty_locations3674849216/001/dir1 (disk ID 1) adds volume:2 collection: replicaPlacement:000 ttl: I0824 22:18:59.466328 store.go:334 add volume 2 on disk ID 1 I0824 22:18:59.466503 store.go:329 In dir /tmp/TestVolumeLoadBalancingeven_distribution_across_empty_locations3674849216/001/dir2 (disk ID 2) adds volume:3 collection: replicaPlacement:000 ttl: I0824 22:18:59.467416 store.go:334 add volume 3 on disk ID 2 I0824 22:18:59.467568 store.go:329 In dir /tmp/TestVolumeLoadBalancingeven_distribution_across_empty_locations3674849216/001/dir0 (disk ID 0) adds volume:4 collection: replicaPlacement:000 ttl: I0824 22:18:59.468476 store.go:334 add volume 4 on disk ID 0 I0824 22:18:59.468605 store.go:329 In dir /tmp/TestVolumeLoadBalancingeven_distribution_across_empty_locations3674849216/001/dir1 (disk ID 1) adds volume:5 collection: replicaPlacement:000 ttl: I0824 22:18:59.469634 store.go:334 add volume 5 on disk ID 1 I0824 22:18:59.469808 store.go:329 In dir /tmp/TestVolumeLoadBalancingeven_distribution_across_empty_locations3674849216/001/dir2 (disk ID 2) adds volume:6 collection: replicaPlacement:000 ttl: I0824 22:18:59.470729 store.go:334 add volume 6 on disk ID 2 === RUN TestVolumeLoadBalancing/prefers_location_with_fewer_local_volumes I0824 22:18:59.480594 store.go:329 In dir /tmp/TestVolumeLoadBalancingprefers_location_with_fewer_local_volume582976117/001/dir1 (disk ID 1) adds volume:1 collection: replicaPlacement:000 ttl: I0824 22:18:59.482927 store.go:334 add volume 1 on disk ID 1 I0824 22:18:59.483164 store.go:329 In dir /tmp/TestVolumeLoadBalancingprefers_location_with_fewer_local_volume582976117/001/dir1 (disk ID 1) adds volume:2 collection: replicaPlacement:000 ttl: I0824 22:18:59.485857 store.go:334 add volume 2 on disk ID 1 I0824 22:18:59.486214 store.go:329 In dir /tmp/TestVolumeLoadBalancingprefers_location_with_fewer_local_volume582976117/001/dir1 (disk ID 1) adds volume:3 collection: replicaPlacement:000 ttl: I0824 22:18:59.487658 store.go:334 add volume 3 on disk ID 1 === RUN TestVolumeLoadBalancing/ignores_remote_volumes_in_count I0824 22:18:59.500371 store.go:329 In dir /tmp/TestVolumeLoadBalancingignores_remote_volumes_in_count3938584376/001/dir0 (disk ID 0) adds volume:1 collection: replicaPlacement:000 ttl: I0824 22:18:59.503486 store.go:334 add volume 1 on disk ID 0 I0824 22:18:59.503828 store.go:329 In dir /tmp/TestVolumeLoadBalancingignores_remote_volumes_in_count3938584376/001/dir0 (disk ID 0) adds volume:2 collection: replicaPlacement:000 ttl: I0824 22:18:59.507377 store.go:334 add volume 2 on disk ID 0 I0824 22:18:59.507775 store.go:329 In dir /tmp/TestVolumeLoadBalancingignores_remote_volumes_in_count3938584376/001/dir2 (disk ID 2) adds volume:3 collection: replicaPlacement:000 ttl: I0824 22:18:59.510559 store.go:334 add volume 3 on disk ID 2 === RUN TestVolumeLoadBalancing/balances_when_some_locations_have_remote_volumes I0824 22:18:59.520098 store.go:329 In dir /tmp/TestVolumeLoadBalancingbalances_when_some_locations_have_remote3906598629/001/dir2 (disk ID 2) adds volume:1 collection: replicaPlacement:000 ttl: I0824 22:18:59.521823 store.go:334 add volume 1 on disk ID 2 I0824 22:18:59.522072 store.go:329 In dir /tmp/TestVolumeLoadBalancingbalances_when_some_locations_have_remote3906598629/001/dir0 (disk ID 0) adds volume:2 collection: replicaPlacement:000 ttl: I0824 22:18:59.523302 store.go:334 add volume 2 on disk ID 0 I0824 22:18:59.523527 store.go:329 In dir /tmp/TestVolumeLoadBalancingbalances_when_some_locations_have_remote3906598629/001/dir1 (disk ID 1) adds volume:3 collection: replicaPlacement:000 ttl: I0824 22:18:59.526248 store.go:334 add volume 3 on disk ID 1 --- PASS: TestVolumeLoadBalancing (0.07s) --- PASS: TestVolumeLoadBalancing/even_distribution_across_empty_locations (0.01s) --- PASS: TestVolumeLoadBalancing/prefers_location_with_fewer_local_volumes (0.02s) --- PASS: TestVolumeLoadBalancing/ignores_remote_volumes_in_count (0.02s) --- PASS: TestVolumeLoadBalancing/balances_when_some_locations_have_remote_volumes (0.02s) === RUN TestMarkVolumeReadonlyCanDelete I0824 22:18:59.533072 store.go:329 In dir /tmp/TestMarkVolumeReadonlyCanDelete4039139304/001 (disk ID 0) adds volume:11 collection: replicaPlacement:000 ttl: I0824 22:18:59.536005 store.go:334 add volume 11 on disk ID 0 I0824 22:18:59.543529 needle_map_sorted_file.go:26 Start to Generate /tmp/TestMarkVolumeReadonlyCanDelete4039139304/001/11.sdx from /tmp/TestMarkVolumeReadonlyCanDelete4039139304/001/11.idx I0824 22:18:59.550270 needle_map_sorted_file.go:28 Finished Generating /tmp/TestMarkVolumeReadonlyCanDelete4039139304/001/11.sdx from /tmp/TestMarkVolumeReadonlyCanDelete4039139304/001/11.idx --- PASS: TestMarkVolumeReadonlyCanDelete (0.03s) === RUN TestMarkVolumeReadonlyCanDelete_AfterReadonlyBoot I0824 22:18:59.559844 store.go:329 In dir /tmp/TestMarkVolumeReadonlyCanDelete_AfterReadonlyBoot1159460097/001 (disk ID 0) adds volume:12 collection: replicaPlacement:000 ttl: I0824 22:18:59.561874 store.go:334 add volume 12 on disk ID 0 I0824 22:18:59.566106 volume_loading.go:269 open to read file /tmp/TestMarkVolumeReadonlyCanDelete_AfterReadonlyBoot1159460097/001/12.idx I0824 22:18:59.566558 needle_map_sorted_file.go:26 Start to Generate /tmp/TestMarkVolumeReadonlyCanDelete_AfterReadonlyBoot1159460097/001/12.sdx from /tmp/TestMarkVolumeReadonlyCanDelete_AfterReadonlyBoot1159460097/001/12.idx I0824 22:18:59.571068 needle_map_sorted_file.go:28 Finished Generating /tmp/TestMarkVolumeReadonlyCanDelete_AfterReadonlyBoot1159460097/001/12.sdx from /tmp/TestMarkVolumeReadonlyCanDelete_AfterReadonlyBoot1159460097/001/12.idx --- PASS: TestMarkVolumeReadonlyCanDelete_AfterReadonlyBoot (0.02s) === RUN TestMaybeAdjustVolumeMaxNoDeadZone === RUN TestMaybeAdjustVolumeMaxNoDeadZone/ratio=0.5 store_maxvolume_deadzone_test.go:49: free=122.8GiB limit=245.6GiB ratio≈0.50 -> MaxVolumeCount=1 (want 1) === RUN TestMaybeAdjustVolumeMaxNoDeadZone/ratio=1.5 store_maxvolume_deadzone_test.go:49: free=122.8GiB limit=81.9GiB ratio≈1.50 -> MaxVolumeCount=1 (want 1) === RUN TestMaybeAdjustVolumeMaxNoDeadZone/ratio=2.5 store_maxvolume_deadzone_test.go:49: free=122.8GiB limit=49.1GiB ratio≈2.50 -> MaxVolumeCount=2 (want 2) === RUN TestMaybeAdjustVolumeMaxNoDeadZone/ratio=3.5 store_maxvolume_deadzone_test.go:49: free=122.8GiB limit=35.1GiB ratio≈3.50 -> MaxVolumeCount=3 (want 3) --- PASS: TestMaybeAdjustVolumeMaxNoDeadZone (0.00s) --- PASS: TestMaybeAdjustVolumeMaxNoDeadZone/ratio=0.5 (0.00s) --- PASS: TestMaybeAdjustVolumeMaxNoDeadZone/ratio=1.5 (0.00s) --- PASS: TestMaybeAdjustVolumeMaxNoDeadZone/ratio=2.5 (0.00s) --- PASS: TestMaybeAdjustVolumeMaxNoDeadZone/ratio=3.5 (0.00s) === RUN TestCollectStatForOneVolumeModifiedAtSecond --- PASS: TestCollectStatForOneVolumeModifiedAtSecond (0.00s) === RUN TestSpaceCalculation === RUN TestSpaceCalculation/Large_volume,_small_preallocate store_vacuum_test.go:47: Volume size: 261993005056 bytes, Space needed: 288193458995 bytes (110.00% of volume size) === RUN TestSpaceCalculation/Small_volume,_large_preallocate store_vacuum_test.go:47: Volume size: 104857600 bytes, Space needed: 1181116006 bytes (1126.40% of volume size) --- PASS: TestSpaceCalculation (0.00s) --- PASS: TestSpaceCalculation/Large_volume,_small_preallocate (0.00s) --- PASS: TestSpaceCalculation/Small_volume,_large_preallocate (0.00s) === RUN TestHeartbeatSendsFullListUntilTheMasterAccepts --- PASS: TestHeartbeatSendsFullListUntilTheMasterAccepts (0.01s) === RUN TestQuietHeartbeatDoesNotLookLikeAnEmptyServer --- PASS: TestQuietHeartbeatDoesNotLookLikeAnEmptyServer (0.01s) === RUN TestQuietHeartbeatStillDigestsEverythingHeld --- PASS: TestQuietHeartbeatStillDigestsEverythingHeld (0.01s) === RUN TestHeartbeatReportsOnlyWhatChanged --- PASS: TestHeartbeatReportsOnlyWhatChanged (0.01s) === RUN TestHeartbeatReturnsToTheFullListOnRequest --- PASS: TestHeartbeatReturnsToTheFullListOnRequest (0.01s) === RUN TestReconnectingSendsTheFullListAgain --- PASS: TestReconnectingSendsTheFullListAgain (0.01s) === RUN TestRemountedVolumeIsReportedAgain --- PASS: TestRemountedVolumeIsReportedAgain (0.01s) === RUN TestFullListRequestDuringCollectionSurvives --- PASS: TestFullListRequestDuringCollectionSurvives (0.01s) === RUN TestDepartedVolumeIsNamedInTheDelta --- PASS: TestDepartedVolumeIsNamedInTheDelta (0.01s) === RUN TestFullListCarriesNoDepartures --- PASS: TestFullListCarriesNoDepartures (0.01s) === RUN TestMovedVolumeIsNotADeparture --- PASS: TestMovedVolumeIsNotADeparture (0.01s) === RUN TestDepartureNamesTheVolumeAsItIsNow --- PASS: TestDepartureNamesTheVolumeAsItIsNow (0.01s) === RUN TestOverlappingHeartbeatsNameNoDepartures --- PASS: TestOverlappingHeartbeatsNameNoDepartures (0.01s) === RUN TestMountingAVolumeStartsNoBatchWorker --- PASS: TestMountingAVolumeStartsNoBatchWorker (0.00s) === RUN TestDurableWriteStartsTheBatchWorkerOnce --- PASS: TestDurableWriteStartsTheBatchWorkerOnce (0.01s) === RUN TestDurableWriteAfterDestroyWritesInline I0824 22:18:59.716919 volume_write.go:132 removed volume file /tmp/TestDurableWriteAfterDestroyWritesInline43027492/001/1.dat (size=424) I0824 22:18:59.717310 volume_write.go:132 removed volume file /tmp/TestDurableWriteAfterDestroyWritesInline43027492/001/1.idx (size=16) --- PASS: TestDurableWriteAfterDestroyWritesInline (0.01s) === RUN TestBinarySearch --- PASS: TestBinarySearch (0.00s) === RUN TestScrubVolumeData === RUN TestScrubVolumeData/healthy_volume === RUN TestScrubVolumeData/zero-size_volume_without_index === RUN TestScrubVolumeData/zero-size_volume_with_index E0824 22:18:59.732500 needle_read.go:52 ./test_files/empty_volume.dat read 0 dataSize 75688 offset 8 fileSize 8: EOF E0824 22:18:59.732832 needle_read.go:52 ./test_files/empty_volume.dat read 0 dataSize 4448 offset 75696 fileSize 8: EOF E0824 22:18:59.732945 needle_read.go:52 ./test_files/empty_volume.dat read 0 dataSize 83640 offset 80144 fileSize 8: EOF E0824 22:18:59.733027 needle_read.go:52 ./test_files/empty_volume.dat read 0 dataSize 65472 offset 163784 fileSize 8: EOF E0824 22:18:59.733088 needle_read.go:52 ./test_files/empty_volume.dat read 0 dataSize 79928 offset 229256 fileSize 8: EOF E0824 22:18:59.733167 needle_read.go:52 ./test_files/empty_volume.dat read 0 dataSize 86696 offset 309184 fileSize 8: EOF E0824 22:18:59.733229 needle_read.go:52 ./test_files/empty_volume.dat read 0 dataSize 32984 offset 395880 fileSize 8: EOF E0824 22:18:59.733306 needle_read.go:52 ./test_files/empty_volume.dat read 0 dataSize 74648 offset 428864 fileSize 8: EOF E0824 22:18:59.733404 needle_read.go:52 ./test_files/empty_volume.dat read 0 dataSize 60400 offset 503512 fileSize 8: EOF E0824 22:18:59.733453 needle_read.go:52 ./test_files/empty_volume.dat read 0 dataSize 12328 offset 563912 fileSize 8: EOF E0824 22:18:59.733734 needle_read.go:52 ./test_files/empty_volume.dat read 0 dataSize 81992 offset 576240 fileSize 8: EOF E0824 22:18:59.733887 needle_read.go:52 ./test_files/empty_volume.dat read 0 dataSize 46544 offset 658232 fileSize 8: EOF E0824 22:18:59.733962 needle_read.go:52 ./test_files/empty_volume.dat read 0 dataSize 88080 offset 704776 fileSize 8: EOF E0824 22:18:59.734011 needle_read.go:52 ./test_files/empty_volume.dat read 0 dataSize 5512 offset 792856 fileSize 8: EOF E0824 22:18:59.734214 needle_read.go:52 ./test_files/empty_volume.dat read 0 dataSize 81648 offset 798368 fileSize 8: EOF E0824 22:18:59.734865 needle_read.go:52 ./test_files/empty_volume.dat read 0 dataSize 33200 offset 880016 fileSize 8: EOF E0824 22:18:59.735880 needle_read.go:52 ./test_files/empty_volume.dat read 0 dataSize 29320 offset 913216 fileSize 8: EOF === RUN TestScrubVolumeData/volume_without_index === RUN TestScrubVolumeData/bitrot_volume --- PASS: TestScrubVolumeData (0.03s) --- PASS: TestScrubVolumeData/healthy_volume (0.01s) --- PASS: TestScrubVolumeData/zero-size_volume_without_index (0.00s) --- PASS: TestScrubVolumeData/zero-size_volume_with_index (0.01s) --- PASS: TestScrubVolumeData/volume_without_index (0.00s) --- PASS: TestScrubVolumeData/bitrot_volume (0.01s) === RUN TestScrubVolumeData_IgnoresOffset0Tombstone --- PASS: TestScrubVolumeData_IgnoresOffset0Tombstone (0.00s) === RUN TestCheckVolumeDataIntegrityWithDeletionTombstone --- PASS: TestCheckVolumeDataIntegrityWithDeletionTombstone (0.00s) === RUN TestCheckVolumeDataIntegritySortedIndex --- PASS: TestCheckVolumeDataIntegritySortedIndex (0.00s) === RUN TestCheckVolumeDataIntegrityVerifiesDeletionTail W0824 22:18:59.765330 volume_checking.go:357 data file /tmp/TestCheckVolumeDataIntegrityVerifiesDeletionTail23620780/001/1.dat actual 1376 bytes expected 1312 bytes! --- PASS: TestCheckVolumeDataIntegrityVerifiesDeletionTail (0.00s) === RUN TestVolumeLoadStaysWritableWithKeySortedIndex --- PASS: TestVolumeLoadStaysWritableWithKeySortedIndex (0.00s) === RUN TestMaxNeedleEnd --- PASS: TestMaxNeedleEnd (0.00s) === RUN TestDestroyKeepsVifWhenEcCoexists I0824 22:18:59.778773 volume_write.go:132 removed volume file /tmp/TestDestroyKeepsVifWhenEcCoexists928587002/001/1.dat (size=552) I0824 22:18:59.779035 volume_write.go:132 removed volume file /tmp/TestDestroyKeepsVifWhenEcCoexists928587002/001/1.idx (size=16) --- PASS: TestDestroyKeepsVifWhenEcCoexists (0.00s) === RUN TestDestroyRemovesVifWhenNoEc I0824 22:18:59.783299 volume_write.go:132 removed volume file /tmp/TestDestroyRemovesVifWhenNoEc3022726715/001/1.dat (size=592) I0824 22:18:59.784119 volume_write.go:132 removed volume file /tmp/TestDestroyRemovesVifWhenNoEc3022726715/001/1.idx (size=16) --- PASS: TestDestroyRemovesVifWhenNoEc (0.00s) === RUN TestRepairIdxHeadTombstones_RestoresClobberedRows I0824 22:18:59.789611 volume_idx_repair.go:65 volume 1: /tmp/TestRepairIdxHeadTombstones_RestoresClobberedRows1839566691/001/1.idx starts with 4 offset-0 tombstones, recovering the .idx rows they overwrote from /tmp/TestRepairIdxHeadTombstones_RestoresClobberedRows1839566691/001/1.dat I0824 22:18:59.790873 volume_loading.go:265 volume 1: recovered 4 overwritten /tmp/TestRepairIdxHeadTombstones_RestoresClobberedRows1839566691/001/1.idx rows from /tmp/TestRepairIdxHeadTombstones_RestoresClobberedRows1839566691/001/1.dat --- PASS: TestRepairIdxHeadTombstones_RestoresClobberedRows (0.01s) === RUN TestRepairIdxHeadTombstones_ReadOnlyVolume I0824 22:18:59.797988 volume_idx_repair.go:65 volume 1: /tmp/TestRepairIdxHeadTombstones_ReadOnlyVolume442398598/001/1.idx starts with 2 offset-0 tombstones, recovering the .idx rows they overwrote from /tmp/TestRepairIdxHeadTombstones_ReadOnlyVolume442398598/001/1.dat I0824 22:18:59.799152 volume_loading.go:265 volume 1: recovered 2 overwritten /tmp/TestRepairIdxHeadTombstones_ReadOnlyVolume442398598/001/1.idx rows from /tmp/TestRepairIdxHeadTombstones_ReadOnlyVolume442398598/001/1.dat I0824 22:18:59.799929 volume_loading.go:269 open to read file /tmp/TestRepairIdxHeadTombstones_ReadOnlyVolume442398598/001/1.idx I0824 22:18:59.800297 needle_map_sorted_file.go:26 Start to Generate /tmp/TestRepairIdxHeadTombstones_ReadOnlyVolume442398598/001/1.sdx from /tmp/TestRepairIdxHeadTombstones_ReadOnlyVolume442398598/001/1.idx I0824 22:18:59.805718 needle_map_sorted_file.go:28 Finished Generating /tmp/TestRepairIdxHeadTombstones_ReadOnlyVolume442398598/001/1.sdx from /tmp/TestRepairIdxHeadTombstones_ReadOnlyVolume442398598/001/1.idx --- PASS: TestRepairIdxHeadTombstones_ReadOnlyVolume (0.01s) === RUN TestRepairIdxHeadTombstones_LeavesHealthyIdxAlone --- PASS: TestRepairIdxHeadTombstones_LeavesHealthyIdxAlone (0.01s) === RUN TestRepairIdxHeadTombstones_KeepsDeletedNeedlesDeleted I0824 22:18:59.823826 volume_idx_repair.go:65 volume 1: /tmp/TestRepairIdxHeadTombstones_KeepsDeletedNeedlesDeleted3331016699/001/1.idx starts with 2 offset-0 tombstones, recovering the .idx rows they overwrote from /tmp/TestRepairIdxHeadTombstones_KeepsDeletedNeedlesDeleted3331016699/001/1.dat I0824 22:18:59.826162 volume_loading.go:265 volume 1: recovered 1 overwritten /tmp/TestRepairIdxHeadTombstones_KeepsDeletedNeedlesDeleted3331016699/001/1.idx rows from /tmp/TestRepairIdxHeadTombstones_KeepsDeletedNeedlesDeleted3331016699/001/1.dat --- PASS: TestRepairIdxHeadTombstones_KeepsDeletedNeedlesDeleted (0.01s) === RUN TestVolumeInfoCountsDoNotWrap === RUN TestVolumeInfoCountsDoNotWrap/an_ordinary_count === RUN TestVolumeInfoCountsDoNotWrap/the_ceiling === RUN TestVolumeInfoCountsDoNotWrap/past_the_ceiling === RUN TestVolumeInfoCountsDoNotWrap/far_past_it --- PASS: TestVolumeInfoCountsDoNotWrap (0.00s) --- PASS: TestVolumeInfoCountsDoNotWrap/an_ordinary_count (0.00s) --- PASS: TestVolumeInfoCountsDoNotWrap/the_ceiling (0.00s) --- PASS: TestVolumeInfoCountsDoNotWrap/past_the_ceiling (0.00s) --- PASS: TestVolumeInfoCountsDoNotWrap/far_past_it (0.00s) === RUN TestVolumeInfoSurvivesMoreDeletesThanFiles --- PASS: TestVolumeInfoSurvivesMoreDeletesThanFiles (0.00s) === RUN TestRepeatedVolumeStringsAreShared --- PASS: TestRepeatedVolumeStringsAreShared (0.00s) === RUN TestShortVolumeInfoSharesTheSameStrings --- PASS: TestShortVolumeInfoSharesTheSameStrings (0.00s) === RUN TestEmptyVolumeStringsStayEmpty --- PASS: TestEmptyVolumeStringsStayEmpty (0.00s) === RUN TestVolumeStringsStaySharedAcrossCollection --- PASS: TestVolumeStringsStaySharedAcrossCollection (0.83s) === RUN TestInterningIsSafeUnderConcurrentReports --- PASS: TestInterningIsSafeUnderConcurrentReports (0.00s) === RUN TestVolumeInfoHasNoInteriorPadding volume_info_layout_test.go:32: 110 bytes of fields in a 112 byte struct --- PASS: TestVolumeInfoHasNoInteriorPadding (0.00s) === RUN TestSortVolumeInfos --- PASS: TestSortVolumeInfos (0.00s) === RUN TestNewVolumeInfoFromShortKeepsDiskId --- PASS: TestNewVolumeInfoFromShortKeepsDiskId (0.00s) === RUN TestCheckReadWriteErrorTracksConsecutiveEIO --- PASS: TestCheckReadWriteErrorTracksConsecutiveEIO (0.00s) === RUN TestCheckReadWriteErrorNonEIOResetsStreak --- PASS: TestCheckReadWriteErrorNonEIOResetsStreak (0.00s) === RUN TestCheckReadWriteErrorIgnoresPlainError --- PASS: TestCheckReadWriteErrorIgnoresPlainError (0.00s) === RUN TestIoErrorToleranceGate --- PASS: TestIoErrorToleranceGate (0.00s) === RUN TestQuarantineIsSticky --- PASS: TestQuarantineIsSticky (0.00s) === RUN TestReadNeedleBlobTracksEIO E0824 22:19:00.660050 needle_read.go:52 eio read 0 dataSize 32 offset 0 fileSize 0: input/output error --- PASS: TestReadNeedleBlobTracksEIO (0.00s) === RUN TestIoErrorStateIsRaceFree --- PASS: TestIoErrorStateIsRaceFree (0.00s) === RUN TestLoad_CorruptIdx_NoSegfault I0824 22:19:00.669696 volume_loading.go:269 open to read file /tmp/TestLoad_CorruptIdx_NoSegfault4242359711/001/1.idx I0824 22:19:00.670163 volume_loading.go:288 volumeDataIntegrityChecking failed verifyIndexFileIntegrity /tmp/TestLoad_CorruptIdx_NoSegfault4242359711/001/1.idx failed: index file's size is 15 bytes, maybe corrupted I0824 22:19:00.670485 needle_map_sorted_file.go:26 Start to Generate /tmp/TestLoad_CorruptIdx_NoSegfault4242359711/001/1.sdx from /tmp/TestLoad_CorruptIdx_NoSegfault4242359711/001/1.idx I0824 22:19:01.285623 needle_map_sorted_file.go:28 Finished Generating /tmp/TestLoad_CorruptIdx_NoSegfault4242359711/001/1.sdx from /tmp/TestLoad_CorruptIdx_NoSegfault4242359711/001/1.idx I0824 22:19:01.287596 volume_loading.go:302 loading sorted db /tmp/TestLoad_CorruptIdx_NoSegfault4242359711/001/1.sdx error: unexpected file /tmp/TestLoad_CorruptIdx_NoSegfault4242359711/001/1.idx size: 15 --- PASS: TestLoad_CorruptIdx_NoSegfault (0.63s) === RUN TestMarkVolumeWritable_ReopensPersistedReadOnly I0824 22:19:01.302323 volume_loading.go:269 open to read file /tmp/TestMarkVolumeWritable_ReopensPersistedReadOnly4170322327/001/1.idx I0824 22:19:01.302987 needle_map_sorted_file.go:26 Start to Generate /tmp/TestMarkVolumeWritable_ReopensPersistedReadOnly4170322327/001/1.sdx from /tmp/TestMarkVolumeWritable_ReopensPersistedReadOnly4170322327/001/1.idx I0824 22:19:01.932968 needle_map_sorted_file.go:28 Finished Generating /tmp/TestMarkVolumeWritable_ReopensPersistedReadOnly4170322327/001/1.sdx from /tmp/TestMarkVolumeWritable_ReopensPersistedReadOnly4170322327/001/1.idx I0824 22:19:01.947014 volume_write.go:299 index needle 2 of volume 1 at offset 144: needle map /tmp/TestMarkVolumeWritable_ReopensPersistedReadOnly4170322327/001/1.sdx is read only: invalid argument --- PASS: TestMarkVolumeWritable_ReopensPersistedReadOnly (0.66s) === RUN TestReopenIdxForWrite_NoopWhenAlreadyWritable --- PASS: TestReopenIdxForWrite_NoopWhenAlreadyWritable (0.01s) === RUN TestReadNeedleNilNeedleMap I0824 22:19:01.969025 volume_read.go:26 volume 1: needle map not loaded; read returns not-found I0824 22:19:01.969306 volume_read.go:120 volume 1: needle map not loaded; read returns not-found --- PASS: TestReadNeedleNilNeedleMap (0.01s) === RUN TestReadNeedMetaWithWritesAndUpdates --- PASS: TestReadNeedMetaWithWritesAndUpdates (0.02s) === RUN TestReadNeedMetaWithDeletesThenWrites --- PASS: TestReadNeedMetaWithDeletesThenWrites (0.01s) === RUN TestVifReplicationOutranksSuperBlock --- PASS: TestVifReplicationOutranksSuperBlock (0.02s) === RUN TestEmptyVifReplicationLeavesSuperBlockAlone --- PASS: TestEmptyVifReplicationLeavesSuperBlockAlone (0.01s) === RUN TestReconcileRollForwardMarkerOnly I0824 22:19:02.034384 volume_loading.go:331 loading leveldb index /tmp/TestReconcileRollForwardMarkerOnly3001855941/001/1.ldb I0824 22:19:03.901738 needle_map_leveldb.go:345 loading idx to leveldb from offset 0 for file: /tmp/TestReconcileRollForwardMarkerOnly3001855941/001/1.cpx I0824 22:19:03.951056 volume_vacuum.go:323 volume 1: rolling forward interrupted compaction commit I0824 22:19:03.979218 volume_loading.go:331 loading leveldb index /tmp/TestReconcileRollForwardMarkerOnly3001855941/001/1.ldb --- PASS: TestReconcileRollForwardMarkerOnly (1.98s) === RUN TestReconcileRollForwardPartialRename I0824 22:19:04.023456 volume_loading.go:331 loading leveldb index /tmp/TestReconcileRollForwardPartialRename2203881699/001/1.ldb I0824 22:19:04.199306 needle_map_leveldb.go:345 loading idx to leveldb from offset 0 for file: /tmp/TestReconcileRollForwardPartialRename2203881699/001/1.cpx I0824 22:19:04.219988 volume_vacuum.go:323 volume 1: rolling forward interrupted compaction commit I0824 22:19:04.226269 volume_loading.go:331 loading leveldb index /tmp/TestReconcileRollForwardPartialRename2203881699/001/1.ldb --- PASS: TestReconcileRollForwardPartialRename (0.23s) === RUN TestReconcileRollBackNoMarker I0824 22:19:04.266182 needle_map_memory.go:113 loading idx from offset 0 for file: /tmp/TestReconcileRollBackNoMarker3246011390/001/1.cpx I0824 22:19:04.270276 volume_vacuum.go:332 volume 1: rolling back orphan compaction file .cpd I0824 22:19:04.270831 volume_vacuum.go:332 volume 1: rolling back orphan compaction file .cpx --- PASS: TestReconcileRollBackNoMarker (0.04s) === RUN TestReconcileSkipsLoadedVolumeMidVacuum I0824 22:19:04.319366 needle_map_memory.go:113 loading idx from offset 0 for file: /tmp/TestReconcileSkipsLoadedVolumeMidVacuum1603850134/001/1.cpx --- PASS: TestReconcileSkipsLoadedVolumeMidVacuum (0.04s) === RUN TestApplyCompactSwapMissingTempFilesPreservesLive --- PASS: TestApplyCompactSwapMissingTempFilesPreservesLive (0.01s) === RUN TestDestroyRemovesCommitMarker I0824 22:19:04.332733 volume_write.go:132 removed volume file /tmp/TestDestroyRemovesCommitMarker753113639/001/1.dat (size=1304) I0824 22:19:04.333175 volume_write.go:132 removed volume file /tmp/TestDestroyRemovesCommitMarker753113639/001/1.idx (size=48) --- PASS: TestDestroyRemovesCommitMarker (0.01s) === RUN TestMakeDiff --- PASS: TestMakeDiff (0.00s) === RUN TestMemIndexCompaction I0824 22:19:06.024669 needle_map_memory.go:113 loading idx from offset 0 for file: /tmp/TestMemIndexCompaction3314077414/001/1.cpx volume_vacuum_test.go:97: compaction speed: 7003126.09 bytes/s I0824 22:19:07.037937 volume_vacuum.go:166 Committing volume 1 vacuuming... I0824 22:19:07.920583 needle_map_memory.go:113 loading idx from offset 9730 for file: /tmp/TestMemIndexCompaction3314077414/001/1.cpx volume_vacuum_test.go:115: realRecordCount:29730, v.FileCount():29730 mm.DeletedCount():9844 --- PASS: TestMemIndexCompaction (4.73s) === RUN TestLDBIndexCompaction I0824 22:19:09.073805 volume_loading.go:331 loading leveldb index /tmp/TestLDBIndexCompaction1020358345/001/1.ldb I0824 22:19:10.958871 needle_map_leveldb.go:345 loading idx to leveldb from offset 0 for file: /tmp/TestLDBIndexCompaction1020358345/001/1.cpx volume_vacuum_test.go:97: compaction speed: 5696348.66 bytes/s I0824 22:19:13.346692 volume_vacuum.go:166 Committing volume 1 vacuuming... I0824 22:19:14.122792 needle_map_leveldb.go:345 loading idx to leveldb from offset 9728 for file: /tmp/TestLDBIndexCompaction1020358345/001/1.cpx I0824 22:19:15.143119 volume_loading.go:328 updating leveldb index /tmp/TestLDBIndexCompaction1020358345/001/1.ldb volume_vacuum_test.go:110: watermark from levelDB: 20000, realWatermark: 20000, nm.recordCount: 29728, realRecordCount:29728, fileCount=29728, deletedcount:9728 I0824 22:19:15.592070 volume_loading.go:331 loading leveldb index /tmp/TestLDBIndexCompaction1020358345/001/1.ldb --- PASS: TestLDBIndexCompaction (7.25s) === RUN TestCommitCompactDeletionTailKeepsWritable I0824 22:19:16.321683 volume_loading.go:331 loading leveldb index /tmp/TestCommitCompactDeletionTailKeepsWritable4126676373/001/1.ldb I0824 22:19:16.630121 needle_map_leveldb.go:345 loading idx to leveldb from offset 0 for file: /tmp/TestCommitCompactDeletionTailKeepsWritable4126676373/001/1.cpx I0824 22:19:16.650900 volume_vacuum.go:166 Committing volume 1 vacuuming... I0824 22:19:16.654892 needle_map_leveldb.go:345 loading idx to leveldb from offset 5 for file: /tmp/TestCommitCompactDeletionTailKeepsWritable4126676373/001/1.cpx I0824 22:19:16.721805 volume_loading.go:328 updating leveldb index /tmp/TestCommitCompactDeletionTailKeepsWritable4126676373/001/1.ldb I0824 22:19:16.732902 volume_loading.go:331 loading leveldb index /tmp/TestCommitCompactDeletionTailKeepsWritable4126676373/001/1.ldb --- PASS: TestCommitCompactDeletionTailKeepsWritable (0.43s) === RUN TestCompactVolumeFilesOffline I0824 22:19:16.762049 needle_map_memory.go:113 loading idx from offset 0 for file: /tmp/TestCompactVolumeFilesOffline957394376/001/1.cpx I0824 22:19:16.763199 volume_vacuum.go:166 Committing volume 1 vacuuming... --- PASS: TestCompactVolumeFilesOffline (0.02s) === RUN TestCleanupCompactRemovesTempFiles I0824 22:19:16.776545 volume_vacuum.go:349 Cleaning up volume 1 vacuuming... --- PASS: TestCleanupCompactRemovesTempFiles (0.01s) === RUN TestCompactByIndex_DropsDanglingNeedle E0824 22:19:16.789405 needle_read.go:52 /tmp/TestCompactByIndex_DropsDanglingNeedle2028693235/001/1.dat read 0 dataSize 2080 offset 1052120 fileSize 3544: EOF W0824 22:19:16.789784 volume_vacuum.go:709 vacuum volume 1: dropping unreadable needle key=108 offset=1052120 size=2048: EOF W0824 22:19:16.789984 volume_vacuum.go:735 vacuum volume 1: dropped 1 unreadable index entries (2048 data bytes) during compaction I0824 22:19:16.791012 needle_map_memory.go:113 loading idx from offset 0 for file: /tmp/TestCompactByIndex_DropsDanglingNeedle2028693235/001/1.cpx --- PASS: TestCompactByIndex_DropsDanglingNeedle (0.01s) === RUN TestWriteNeedle2FsyncsInlineWhileStopping --- PASS: TestWriteNeedle2FsyncsInlineWhileStopping (0.01s) === RUN TestWriteNeedle2TruncatesWhenInlineFsyncFails --- PASS: TestWriteNeedle2TruncatesWhenInlineFsyncFails (0.00s) === RUN TestWriteNeedle2DropsIndexOfUnflushedNewNeedle --- PASS: TestWriteNeedle2DropsIndexOfUnflushedNewNeedle (0.00s) === RUN TestStoreWriteVolumeNeedleStaysDurableWhileStopping I0824 22:19:16.818342 store.go:329 In dir /tmp/TestStoreWriteVolumeNeedleStaysDurableWhileStopping4030950933/001 (disk ID 0) adds volume:1 collection: replicaPlacement:000 ttl: I0824 22:19:16.820841 store.go:334 add volume 1 on disk ID 0 --- PASS: TestStoreWriteVolumeNeedleStaysDurableWhileStopping (0.01s) === RUN TestSearchVolumesWithDeletedNeedles offset: 10216, isLast: false --- PASS: TestSearchVolumesWithDeletedNeedles (0.01s) === RUN TestDestroyEmptyVolumeWithOnlyEmpty I0824 22:19:16.834788 volume_write.go:132 removed volume file /tmp/TestDestroyEmptyVolumeWithOnlyEmpty3038443707/001/1.dat (size=8) I0824 22:19:16.835285 volume_write.go:132 removed volume file /tmp/TestDestroyEmptyVolumeWithOnlyEmpty3038443707/001/1.idx (size=0) --- PASS: TestDestroyEmptyVolumeWithOnlyEmpty (0.01s) === RUN TestDestroyEmptyVolumeWithoutOnlyEmpty I0824 22:19:16.841624 volume_write.go:132 removed volume file /tmp/TestDestroyEmptyVolumeWithoutOnlyEmpty1376366403/001/1.dat (size=8) I0824 22:19:16.842083 volume_write.go:132 removed volume file /tmp/TestDestroyEmptyVolumeWithoutOnlyEmpty1376366403/001/1.idx (size=0) --- PASS: TestDestroyEmptyVolumeWithoutOnlyEmpty (0.01s) === RUN TestDestroyNonemptyVolumeWithOnlyEmpty --- PASS: TestDestroyNonemptyVolumeWithOnlyEmpty (0.00s) === RUN TestDestroyNonemptyVolumeWithoutOnlyEmpty I0824 22:19:16.855278 volume_write.go:132 removed volume file /tmp/TestDestroyNonemptyVolumeWithoutOnlyEmpty737209888/001/1.dat (size=992) I0824 22:19:16.855758 volume_write.go:132 removed volume file /tmp/TestDestroyNonemptyVolumeWithoutOnlyEmpty737209888/001/1.idx (size=16) --- PASS: TestDestroyNonemptyVolumeWithoutOnlyEmpty (0.01s) === RUN TestWriteNeedleBlobRejectedOnReadOnlyVolume I0824 22:19:16.862517 volume_loading.go:269 open to read file /tmp/TestWriteNeedleBlobRejectedOnReadOnlyVolume3477021023/001/7.idx I0824 22:19:16.863119 needle_map_sorted_file.go:26 Start to Generate /tmp/TestWriteNeedleBlobRejectedOnReadOnlyVolume3477021023/001/7.sdx from /tmp/TestWriteNeedleBlobRejectedOnReadOnlyVolume3477021023/001/7.idx I0824 22:19:16.865980 needle_map_sorted_file.go:28 Finished Generating /tmp/TestWriteNeedleBlobRejectedOnReadOnlyVolume3477021023/001/7.sdx from /tmp/TestWriteNeedleBlobRejectedOnReadOnlyVolume3477021023/001/7.idx --- PASS: TestWriteNeedleBlobRejectedOnReadOnlyVolume (0.01s) === RUN TestWriteNeedleBlobRejectsSizeMismatch --- PASS: TestWriteNeedleBlobRejectsSizeMismatch (0.00s) PASS ok github.com/seaweedfs/seaweedfs/weed/storage 67.512s ? github.com/seaweedfs/seaweedfs/weed/storage/backend [no test files] ? github.com/seaweedfs/seaweedfs/weed/storage/backend/all [no test files] === RUN TestMemoryMapMaxSizeReadWrite --- PASS: TestMemoryMapMaxSizeReadWrite (0.00s) PASS ok github.com/seaweedfs/seaweedfs/weed/storage/backend/memory_map 1.154s ? github.com/seaweedfs/seaweedfs/weed/storage/backend/rclone_backend [no test files] === RUN TestReadAtReturnsEOFForShortRead --- PASS: TestReadAtReturnsEOFForShortRead (0.04s) === RUN TestReadAtClearsEOFOnFullRead --- PASS: TestReadAtClearsEOFOnFullRead (0.00s) === RUN TestReadAtRejectsInvalidRequestsLocally === RUN TestReadAtRejectsInvalidRequestsLocally/empty_buffer === RUN TestReadAtRejectsInvalidRequestsLocally/negative_offset --- PASS: TestReadAtRejectsInvalidRequestsLocally (0.04s) --- PASS: TestReadAtRejectsInvalidRequestsLocally/empty_buffer (0.00s) --- PASS: TestReadAtRejectsInvalidRequestsLocally/negative_offset (0.04s) PASS ok github.com/seaweedfs/seaweedfs/weed/storage/backend/s3_backend 13.310s === RUN TestBitrotSidecarBytes_RustInterop --- PASS: TestBitrotSidecarBytes_RustInterop (0.16s) === RUN TestBitrotSidecarRoundTripAndSelfIntegrity --- PASS: TestBitrotSidecarRoundTripAndSelfIntegrity (0.00s) === RUN TestShardChecksumBuilderCrossesBoundaries --- PASS: TestShardChecksumBuilderCrossesBoundaries (0.10s) === RUN TestValidateBitrotManifest --- PASS: TestValidateBitrotManifest (0.00s) === RUN TestEncodeProducesVerifiableSidecar I0824 22:18:23.210785 ec_encoder.go:138 encodeDatFile /tmp/TestEncodeProducesVerifiableSidecar3893970845/001/7.dat size:6291456 with EC context 10+4 (total: 14) --- PASS: TestEncodeProducesVerifiableSidecar (1.18s) === RUN TestRebuildExcludesCorruptPresentShard I0824 22:18:24.379041 ec_encoder.go:138 encodeDatFile /tmp/TestRebuildExcludesCorruptPresentShard3139327776/001/9.dat size:6291456 with EC context 10+4 (total: 14) W0824 22:18:24.925383 ec_encoder.go:253 bitrot: present shard 3 for /tmp/TestRebuildExcludesCorruptPresentShard3139327776/001/9 fails checksum; excluding from rebuild inputs and regenerating I0824 22:18:24.942393 ec_encoder.go:269 rebuilding /tmp/TestRebuildExcludesCorruptPresentShard3139327776/001/9: 12 shards present, 2 missing [0 3], config 10+4 (total: 14) --- PASS: TestRebuildExcludesCorruptPresentShard (0.88s) === RUN TestComputeProtectionFromShardsBackfill I0824 22:18:25.287702 ec_encoder.go:138 encodeDatFile /tmp/TestComputeProtectionFromShardsBackfill4143154421/001/11.dat size:6291456 with EC context 10+4 (total: 14) --- PASS: TestComputeProtectionFromShardsBackfill (10.27s) === RUN TestRebuildFailClosedCleansUpGeneratedShard I0824 22:18:36.221291 ec_encoder.go:138 encodeDatFile /tmp/TestRebuildFailClosedCleansUpGeneratedShard4000592496/001/13.dat size:6291456 with EC context 10+4 (total: 14) I0824 22:18:37.702808 ec_encoder.go:269 rebuilding /tmp/TestRebuildFailClosedCleansUpGeneratedShard4000592496/001/13: 13 shards present, 1 missing [0], config 10+4 (total: 14) I0824 22:18:39.796357 ec_encoder.go:269 rebuilding /tmp/TestRebuildFailClosedCleansUpGeneratedShard4000592496/001/13: 13 shards present, 1 missing [0], config 10+4 (total: 14) --- PASS: TestRebuildFailClosedCleansUpGeneratedShard (4.69s) === RUN TestRemoveBitrotSidecars --- PASS: TestRemoveBitrotSidecars (0.01s) === RUN TestEcConsistency_WritesBetweenEncodeAndEcx I0824 22:18:40.173802 ec_encoder.go:138 encodeDatFile /tmp/TestEcConsistency_WritesBetweenEncodeAndEcx2916551554/001/consistency.dat size:103000 with EC context 10+4 (total: 14) ec_consistency_test.go:91: Trying to read needle 2 at offset 103000 size 5032 from EC shards (shardSize=10300) ec_consistency_test.go:92: Intervals: [{BlockIndex:30 InnerBlockOffset:0 Size:100 IsLargeBlock:false LargeBlockRowsCount:1} {BlockIndex:31 InnerBlockOffset:0 Size:100 IsLargeBlock:false LargeBlockRowsCount:1} {BlockIndex:32 InnerBlockOffset:0 Size:100 IsLargeBlock:false LargeBlockRowsCount:1} {BlockIndex:33 InnerBlockOffset:0 Size:100 IsLargeBlock:false LargeBlockRowsCount:1} {BlockIndex:34 InnerBlockOffset:0 Size:100 IsLargeBlock:false LargeBlockRowsCount:1} {BlockIndex:35 InnerBlockOffset:0 Size:100 IsLargeBlock:false LargeBlockRowsCount:1} {BlockIndex:36 InnerBlockOffset:0 Size:100 IsLargeBlock:false LargeBlockRowsCount:1} {BlockIndex:37 InnerBlockOffset:0 Size:100 IsLargeBlock:false LargeBlockRowsCount:1} {BlockIndex:38 InnerBlockOffset:0 Size:100 IsLargeBlock:false LargeBlockRowsCount:1} {BlockIndex:39 InnerBlockOffset:0 Size:100 IsLargeBlock:false LargeBlockRowsCount:1} {BlockIndex:40 InnerBlockOffset:0 Size:100 IsLargeBlock:false LargeBlockRowsCount:1} {BlockIndex:41 InnerBlockOffset:0 Size:100 IsLargeBlock:false LargeBlockRowsCount:1} {BlockIndex:42 InnerBlockOffset:0 Size:100 IsLargeBlock:false LargeBlockRowsCount:1} {BlockIndex:43 InnerBlockOffset:0 Size:100 IsLargeBlock:false LargeBlockRowsCount:1} {BlockIndex:44 InnerBlockOffset:0 Size:100 IsLargeBlock:false LargeBlockRowsCount:1} {BlockIndex:45 InnerBlockOffset:0 Size:100 IsLargeBlock:false LargeBlockRowsCount:1} {BlockIndex:46 InnerBlockOffset:0 Size:100 IsLargeBlock:false LargeBlockRowsCount:1} {BlockIndex:47 InnerBlockOffset:0 Size:100 IsLargeBlock:false LargeBlockRowsCount:1} {BlockIndex:48 InnerBlockOffset:0 Size:100 IsLargeBlock:false LargeBlockRowsCount:1} {BlockIndex:49 InnerBlockOffset:0 Size:100 IsLargeBlock:false LargeBlockRowsCount:1} {BlockIndex:50 InnerBlockOffset:0 Size:100 IsLargeBlock:false LargeBlockRowsCount:1} {BlockIndex:51 InnerBlockOffset:0 Size:100 IsLargeBlock:false LargeBlockRowsCount:1} {BlockIndex:52 InnerBlockOffset:0 Size:100 IsLargeBlock:false LargeBlockRowsCount:1} {BlockIndex:53 InnerBlockOffset:0 Size:100 IsLargeBlock:false LargeBlockRowsCount:1} {BlockIndex:54 InnerBlockOffset:0 Size:100 IsLargeBlock:false LargeBlockRowsCount:1} {BlockIndex:55 InnerBlockOffset:0 Size:100 IsLargeBlock:false LargeBlockRowsCount:1} {BlockIndex:56 InnerBlockOffset:0 Size:100 IsLargeBlock:false LargeBlockRowsCount:1} {BlockIndex:57 InnerBlockOffset:0 Size:100 IsLargeBlock:false LargeBlockRowsCount:1} {BlockIndex:58 InnerBlockOffset:0 Size:100 IsLargeBlock:false LargeBlockRowsCount:1} {BlockIndex:59 InnerBlockOffset:0 Size:100 IsLargeBlock:false LargeBlockRowsCount:1} {BlockIndex:60 InnerBlockOffset:0 Size:100 IsLargeBlock:false LargeBlockRowsCount:1} {BlockIndex:61 InnerBlockOffset:0 Size:100 IsLargeBlock:false LargeBlockRowsCount:1} {BlockIndex:62 InnerBlockOffset:0 Size:100 IsLargeBlock:false LargeBlockRowsCount:1} {BlockIndex:63 InnerBlockOffset:0 Size:100 IsLargeBlock:false LargeBlockRowsCount:1} {BlockIndex:64 InnerBlockOffset:0 Size:100 IsLargeBlock:false LargeBlockRowsCount:1} {BlockIndex:65 InnerBlockOffset:0 Size:100 IsLargeBlock:false LargeBlockRowsCount:1} {BlockIndex:66 InnerBlockOffset:0 Size:100 IsLargeBlock:false LargeBlockRowsCount:1} {BlockIndex:67 InnerBlockOffset:0 Size:100 IsLargeBlock:false LargeBlockRowsCount:1} {BlockIndex:68 InnerBlockOffset:0 Size:100 IsLargeBlock:false LargeBlockRowsCount:1} {BlockIndex:69 InnerBlockOffset:0 Size:100 IsLargeBlock:false LargeBlockRowsCount:1} {BlockIndex:70 InnerBlockOffset:0 Size:100 IsLargeBlock:false LargeBlockRowsCount:1} {BlockIndex:71 InnerBlockOffset:0 Size:100 IsLargeBlock:false LargeBlockRowsCount:1} {BlockIndex:72 InnerBlockOffset:0 Size:100 IsLargeBlock:false LargeBlockRowsCount:1} {BlockIndex:73 InnerBlockOffset:0 Size:100 IsLargeBlock:false LargeBlockRowsCount:1} {BlockIndex:74 InnerBlockOffset:0 Size:100 IsLargeBlock:false LargeBlockRowsCount:1} {BlockIndex:75 InnerBlockOffset:0 Size:100 IsLargeBlock:false LargeBlockRowsCount:1} {BlockIndex:76 InnerBlockOffset:0 Size:100 IsLargeBlock:false LargeBlockRowsCount:1} {BlockIndex:77 InnerBlockOffset:0 Size:100 IsLargeBlock:false LargeBlockRowsCount:1} {BlockIndex:78 InnerBlockOffset:0 Size:100 IsLargeBlock:false LargeBlockRowsCount:1} {BlockIndex:79 InnerBlockOffset:0 Size:100 IsLargeBlock:false LargeBlockRowsCount:1} {BlockIndex:80 InnerBlockOffset:0 Size:32 IsLargeBlock:false LargeBlockRowsCount:1}] ec_consistency_test.go:99: CONFIRMED: Read error for needle written after EC encoding: read past end of shard 0: offset 10300 + size 100 > fileSize 10300 ec_consistency_test.go:129: Original data reads correctly from EC shards --- PASS: TestEcConsistency_WritesBetweenEncodeAndEcx (0.10s) === RUN TestEcConsistency_ExactLargeRowEncoding I0824 22:18:40.271543 ec_encoder.go:138 encodeDatFile /tmp/TestEcConsistency_ExactLargeRowEncoding1623109577/001/exact.dat size:100000 with EC context 10+4 (total: 14) --- PASS: TestEcConsistency_ExactLargeRowEncoding (0.09s) === RUN TestVerifyDecodedDatFile === RUN TestVerifyDecodedDatFile/exactly_the_referenced_extent_is_complete === RUN TestVerifyDecodedDatFile/longer_than_referenced_is_fine:_padding_is_not_missing_data === RUN TestVerifyDecodedDatFile/one_byte_short_still_loses_the_last_needle === RUN TestVerifyDecodedDatFile/an_empty_file_is_short_of_everything === RUN TestVerifyDecodedDatFile/an_index_referencing_nothing_accepts_an_empty_file --- PASS: TestVerifyDecodedDatFile (0.01s) --- PASS: TestVerifyDecodedDatFile/exactly_the_referenced_extent_is_complete (0.00s) --- PASS: TestVerifyDecodedDatFile/longer_than_referenced_is_fine:_padding_is_not_missing_data (0.00s) --- PASS: TestVerifyDecodedDatFile/one_byte_short_still_loses_the_last_needle (0.00s) --- PASS: TestVerifyDecodedDatFile/an_empty_file_is_short_of_everything (0.00s) --- PASS: TestVerifyDecodedDatFile/an_index_referencing_nothing_accepts_an_empty_file (0.00s) === RUN TestVerifyDecodedDatFileMissing --- PASS: TestVerifyDecodedDatFileMissing (0.00s) === RUN TestFindDatFileSizeWithRelocatedShard0 I0824 22:18:40.382939 ec_encoder.go:138 encodeDatFile /tmp/TestFindDatFileSizeWithRelocatedShard0329595536/001/1.dat size:2590912 with EC context 10+4 (total: 14) --- PASS: TestFindDatFileSizeWithRelocatedShard0 (1.73s) === RUN TestMinusParityShardsRatio === RUN TestMinusParityShardsRatio/default_10+4_all_present === RUN TestMinusParityShardsRatio/zero_ratio_falls_back_to_default === RUN TestMinusParityShardsRatio/9+3_all_present === RUN TestMinusParityShardsRatio/9+3_drops_parity_id_9 === RUN TestMinusParityShardsRatio/12+4_keeps_data_ids_>=_10 === RUN TestMinusParityShardsRatio/16+6_keeps_data_ids_>=_10 --- PASS: TestMinusParityShardsRatio (0.00s) --- PASS: TestMinusParityShardsRatio/default_10+4_all_present (0.00s) --- PASS: TestMinusParityShardsRatio/zero_ratio_falls_back_to_default (0.00s) --- PASS: TestMinusParityShardsRatio/9+3_all_present (0.00s) --- PASS: TestMinusParityShardsRatio/9+3_drops_parity_id_9 (0.00s) --- PASS: TestMinusParityShardsRatio/12+4_keeps_data_ids_>=_10 (0.00s) --- PASS: TestMinusParityShardsRatio/16+6_keeps_data_ids_>=_10 (0.00s) === RUN TestRebuildEcxFile_AppliesTombstonesAndRemovesJournal --- PASS: TestRebuildEcxFile_AppliesTombstonesAndRemovesJournal (0.00s) === RUN TestRebuildEcxFile_TornJournalAborts --- PASS: TestRebuildEcxFile_TornJournalAborts (0.00s) === RUN TestRebuildEcxFile_NoJournalNoop --- PASS: TestRebuildEcxFile_NoJournalNoop (0.00s) === RUN TestRebuildEcFiles_TruncatedInputErrorsWithoutPublishing --- PASS: TestRebuildEcFiles_TruncatedInputErrorsWithoutPublishing (0.00s) === RUN TestRebuildEcFiles_ZeroSizeInputsErrors --- PASS: TestRebuildEcFiles_ZeroSizeInputsErrors (0.00s) === RUN TestRebuildEcFiles_HappyPathRebuildsByteIdentical I0824 22:18:42.120356 ec_encoder.go:138 encodeDatFile /tmp/TestRebuildEcFiles_HappyPathRebuildsByteIdentical723061288/001/vol.dat size:7000 with EC context 10+4 (total: 14) I0824 22:18:42.259375 ec_encoder.go:269 rebuilding /tmp/TestRebuildEcFiles_HappyPathRebuildsByteIdentical723061288/001/vol: 13 shards present, 1 missing [5], config 10+4 (total: 14) --- PASS: TestRebuildEcFiles_HappyPathRebuildsByteIdentical (0.74s) === RUN TestRebuildEcFiles_ZeroSizedShardRegeneratedInPlace I0824 22:18:42.857083 ec_encoder.go:138 encodeDatFile /tmp/TestRebuildEcFiles_ZeroSizedShardRegeneratedInPlace2684517849/001/vol.dat size:7000 with EC context 10+4 (total: 14) W0824 22:18:42.996794 ec_encoder.go:183 shard 2 for /tmp/TestRebuildEcFiles_ZeroSizedShardRegeneratedInPlace2684517849/001/vol is zero-sized at /tmp/TestRebuildEcFiles_ZeroSizedShardRegeneratedInPlace2684517849/001/vol.ec02; excluding from rebuild inputs and regenerating I0824 22:18:42.999125 ec_encoder.go:269 rebuilding /tmp/TestRebuildEcFiles_ZeroSizedShardRegeneratedInPlace2684517849/001/vol: 13 shards present, 1 missing [2], config 10+4 (total: 14) --- PASS: TestRebuildEcFiles_ZeroSizedShardRegeneratedInPlace (0.21s) === RUN TestRebuildEcFiles_CustomRatioRebuildsByteIdentical I0824 22:18:43.064235 ec_encoder.go:138 encodeDatFile /tmp/TestRebuildEcFiles_CustomRatioRebuildsByteIdentical2753361891/001/vol.dat size:9000 with EC context 9+3 (total: 12) I0824 22:18:43.275006 ec_encoder.go:269 rebuilding /tmp/TestRebuildEcFiles_CustomRatioRebuildsByteIdentical2753361891/001/vol: 11 shards present, 1 missing [7], config 9+3 (total: 12) --- PASS: TestRebuildEcFiles_CustomRatioRebuildsByteIdentical (0.32s) === RUN TestEcReadRoundTrip === RUN TestEcReadRoundTrip/1_large_row_exact I0824 22:18:43.390423 ec_encoder.go:138 encodeDatFile /tmp/TestEcReadRoundTrip1_large_row_exact2570506985/001/rt_100000.dat size:100000 with EC context 10+4 (total: 14) ec_roundtrip_test.go:59: WARN: EC read mismatch at offset 50000 (fallback path, shardFileSize=10000) === RUN TestEcReadRoundTrip/2_large_rows_exact I0824 22:18:43.451264 ec_encoder.go:138 encodeDatFile /tmp/TestEcReadRoundTrip2_large_rows_exact2243918594/001/rt_200000.dat size:200000 with EC context 10+4 (total: 14) === RUN TestEcReadRoundTrip/3_large_rows_exact I0824 22:18:43.547609 ec_encoder.go:138 encodeDatFile /tmp/TestEcReadRoundTrip3_large_rows_exact1536271620/001/rt_300000.dat size:300000 with EC context 10+4 (total: 14) === RUN TestEcReadRoundTrip/1_large_row_plus_1 I0824 22:18:43.785241 ec_encoder.go:138 encodeDatFile /tmp/TestEcReadRoundTrip1_large_row_plus_13341023816/001/rt_100001.dat size:100001 with EC context 10+4 (total: 14) === RUN TestEcReadRoundTrip/2_large_rows_plus_small I0824 22:18:43.829841 ec_encoder.go:138 encodeDatFile /tmp/TestEcReadRoundTrip2_large_rows_plus_small1122915571/001/rt_201000.dat size:201000 with EC context 10+4 (total: 14) === RUN TestEcReadRoundTrip/1_large_row_plus_half_small I0824 22:18:43.920525 ec_encoder.go:138 encodeDatFile /tmp/TestEcReadRoundTrip1_large_row_plus_half_small4034019317/001/rt_100500.dat size:100500 with EC context 10+4 (total: 14) === RUN TestEcReadRoundTrip/just_under_1_large_row I0824 22:18:43.995891 ec_encoder.go:138 encodeDatFile /tmp/TestEcReadRoundTripjust_under_1_large_row3568542060/001/rt_99999.dat size:99999 with EC context 10+4 (total: 14) === RUN TestEcReadRoundTrip/just_under_2_large_rows I0824 22:18:44.087447 ec_encoder.go:138 encodeDatFile /tmp/TestEcReadRoundTripjust_under_2_large_rows292142870/001/rt_199999.dat size:199999 with EC context 10+4 (total: 14) === RUN TestEcReadRoundTrip/small_only I0824 22:18:44.231195 ec_encoder.go:138 encodeDatFile /tmp/TestEcReadRoundTripsmall_only1150934549/001/rt_3000.dat size:3000 with EC context 10+4 (total: 14) === RUN TestEcReadRoundTrip/small_single_row I0824 22:18:44.245793 ec_encoder.go:138 encodeDatFile /tmp/TestEcReadRoundTripsmall_single_row209531674/001/rt_1000.dat size:1000 with EC context 10+4 (total: 14) === RUN TestEcReadRoundTrip/boundary_spanning I0824 22:18:44.260689 ec_encoder.go:138 encodeDatFile /tmp/TestEcReadRoundTripboundary_spanning86710838/001/rt_105050.dat size:105050 with EC context 10+4 (total: 14) --- PASS: TestEcReadRoundTrip (0.96s) --- PASS: TestEcReadRoundTrip/1_large_row_exact (0.06s) --- PASS: TestEcReadRoundTrip/2_large_rows_exact (0.08s) --- PASS: TestEcReadRoundTrip/3_large_rows_exact (0.25s) --- PASS: TestEcReadRoundTrip/1_large_row_plus_1 (0.04s) --- PASS: TestEcReadRoundTrip/2_large_rows_plus_small (0.09s) --- PASS: TestEcReadRoundTrip/1_large_row_plus_half_small (0.07s) --- PASS: TestEcReadRoundTrip/just_under_1_large_row (0.09s) --- PASS: TestEcReadRoundTrip/just_under_2_large_rows (0.15s) --- PASS: TestEcReadRoundTrip/small_only (0.01s) --- PASS: TestEcReadRoundTrip/small_single_row (0.01s) --- PASS: TestEcReadRoundTrip/boundary_spanning (0.09s) === RUN TestEcOffByOneBug_Issue8947 I0824 22:18:44.355337 ec_encoder.go:138 encodeDatFile /tmp/TestEcOffByOneBug_Issue89472792193102/001/bug_200000.dat size:200000 with EC context 10+4 (total: 14) ec_roundtrip_test.go:269: Buggy formula returned wrong data: expected first bytes 3607ad0bfd3a773a, got fb970c1ef24e5060 ec_roundtrip_test.go:279: Fixed: nLargeBlockRows=2, interval={BlockIndex:11 InnerBlockOffset:50 Size:50 IsLargeBlock:true LargeBlockRowsCount:2} ec_roundtrip_test.go:280: Buggy: nLargeBlockRows=1, interval={BlockIndex:100 InnerBlockOffset:50 Size:50 IsLargeBlock:false LargeBlockRowsCount:1} --- PASS: TestEcOffByOneBug_Issue8947 (0.14s) === RUN TestEcDecodeDatRoundTrip === RUN TestEcDecodeDatRoundTrip/size_1000 I0824 22:18:44.491306 ec_encoder.go:138 encodeDatFile /tmp/TestEcDecodeDatRoundTripsize_10003775512772/001/dec_1000.dat size:1000 with EC context 10+4 (total: 14) === RUN TestEcDecodeDatRoundTrip/size_10485760 I0824 22:18:44.846973 ec_encoder.go:138 encodeDatFile /tmp/TestEcDecodeDatRoundTripsize_10485760891808064/001/dec_10485760.dat size:10485760 with EC context 10+4 (total: 14) === RUN TestEcDecodeDatRoundTrip/size_10486260 I0824 22:18:47.077913 ec_encoder.go:138 encodeDatFile /tmp/TestEcDecodeDatRoundTripsize_104862602826111346/001/dec_10486260.dat size:10486260 with EC context 10+4 (total: 14) --- PASS: TestEcDecodeDatRoundTrip (3.68s) --- PASS: TestEcDecodeDatRoundTrip/size_1000 (0.15s) --- PASS: TestEcDecodeDatRoundTrip/size_10485760 (2.17s) --- PASS: TestEcDecodeDatRoundTrip/size_10486260 (1.34s) === RUN TestWriteDatFileAfterTailDeletion I0824 22:18:48.170708 ec_encoder.go:138 encodeDatFile /tmp/TestWriteDatFileAfterTailDeletion3727207367/001/tail_del.dat size:102530 with EC context 10+4 (total: 14) --- PASS: TestWriteDatFileAfterTailDeletion (0.17s) === RUN TestWriteDatFileFallbackLayout I0824 22:18:48.338006 ec_encoder.go:138 encodeDatFile /tmp/TestWriteDatFileFallbackLayout1775221077/001/plain.dat size:102530 with EC context 10+4 (total: 14) I0824 22:18:48.441209 ec_encoder.go:138 encodeDatFile /tmp/TestWriteDatFileFallbackLayout1775221077/001/ambiguous1.dat size:99999 with EC context 10+4 (total: 14) I0824 22:18:48.516247 ec_encoder.go:138 encodeDatFile /tmp/TestWriteDatFileFallbackLayout1775221077/001/ambiguous2.dat size:199999 with EC context 10+4 (total: 14) --- PASS: TestWriteDatFileFallbackLayout (0.54s) === RUN TestShardsInfo_SetAndGet --- PASS: TestShardsInfo_SetAndGet (0.00s) === RUN TestShardsInfo_SortedOrder --- PASS: TestShardsInfo_SortedOrder (0.00s) === RUN TestShardsInfo_Delete --- PASS: TestShardsInfo_Delete (0.00s) === RUN TestShardsInfo_Update --- PASS: TestShardsInfo_Update (0.00s) === RUN TestShardsInfo_TotalSize --- PASS: TestShardsInfo_TotalSize (0.00s) === RUN TestShardsInfo_Sizes --- PASS: TestShardsInfo_Sizes (0.00s) === RUN TestShardsInfo_Copy --- PASS: TestShardsInfo_Copy (0.00s) === RUN TestShardsInfo_AddSubtract --- PASS: TestShardsInfo_AddSubtract (0.00s) === RUN TestShardsInfo_PlusMinus --- PASS: TestShardsInfo_PlusMinus (0.00s) === RUN TestShardsInfo_DeleteParityShards --- PASS: TestShardsInfo_DeleteParityShards (0.00s) === RUN TestShardsInfo_FromVolumeEcShardInformationMessage === RUN TestShardsInfo_FromVolumeEcShardInformationMessage/nil_message === RUN TestShardsInfo_FromVolumeEcShardInformationMessage/single_shard === RUN TestShardsInfo_FromVolumeEcShardInformationMessage/multiple_shards === RUN TestShardsInfo_FromVolumeEcShardInformationMessage/missing_sizes --- PASS: TestShardsInfo_FromVolumeEcShardInformationMessage (0.01s) --- PASS: TestShardsInfo_FromVolumeEcShardInformationMessage/nil_message (0.00s) --- PASS: TestShardsInfo_FromVolumeEcShardInformationMessage/single_shard (0.00s) --- PASS: TestShardsInfo_FromVolumeEcShardInformationMessage/multiple_shards (0.00s) --- PASS: TestShardsInfo_FromVolumeEcShardInformationMessage/missing_sizes (0.00s) === RUN TestEcShardsTotalSize === RUN TestEcShardsTotalSize/nil_message === RUN TestEcShardsTotalSize/all_shards_present === RUN TestEcShardsTotalSize/only_parity_shards --- PASS: TestEcShardsTotalSize (0.00s) --- PASS: TestEcShardsTotalSize/nil_message (0.00s) --- PASS: TestEcShardsTotalSize/all_shards_present (0.00s) --- PASS: TestEcShardsTotalSize/only_parity_shards (0.00s) === RUN TestEcShardsDataSize === RUN TestEcShardsDataSize/nil_message === RUN TestEcShardsDataSize/data_shards_only === RUN TestEcShardsDataSize/mixed_data_and_parity_(default_10+4) === RUN TestEcShardsDataSize/only_parity_shards_(excluded) === RUN TestEcShardsDataSize/missing_sizes_tolerated === RUN TestEcShardsDataSize/custom_6+3_ratio:_shards_0..5_are_data,_6..8_are_parity === RUN TestEcShardsDataSize/custom_16+6_ratio:_shard_id_12_is_data,_not_parity === RUN TestEcShardsDataSize/negative_dataShards_falls_back_to_default --- PASS: TestEcShardsDataSize (0.00s) --- PASS: TestEcShardsDataSize/nil_message (0.00s) --- PASS: TestEcShardsDataSize/data_shards_only (0.00s) --- PASS: TestEcShardsDataSize/mixed_data_and_parity_(default_10+4) (0.00s) --- PASS: TestEcShardsDataSize/only_parity_shards_(excluded) (0.00s) --- PASS: TestEcShardsDataSize/missing_sizes_tolerated (0.00s) --- PASS: TestEcShardsDataSize/custom_6+3_ratio:_shards_0..5_are_data,_6..8_are_parity (0.00s) --- PASS: TestEcShardsDataSize/custom_16+6_ratio:_shard_id_12_is_data,_not_parity (0.00s) --- PASS: TestEcShardsDataSize/negative_dataShards_falls_back_to_default (0.00s) === RUN TestShardsInfo_String --- PASS: TestShardsInfo_String (0.00s) === RUN TestShardsInfo_ConcurrentAccess --- PASS: TestShardsInfo_ConcurrentAccess (0.02s) === RUN TestEncodingDecoding I0824 22:18:48.926169 ec_encoder.go:138 encodeDatFile 1.dat size:2590912 with EC context 10+4 (total: 14) --- PASS: TestEncodingDecoding (2.14s) === RUN TestLocateData [{BlockIndex:5 InnerBlockOffset:100 Size:9900 IsLargeBlock:true LargeBlockRowsCount:1} {BlockIndex:6 InnerBlockOffset:0 Size:10000 IsLargeBlock:true LargeBlockRowsCount:1} {BlockIndex:7 InnerBlockOffset:0 Size:10000 IsLargeBlock:true LargeBlockRowsCount:1} {BlockIndex:8 InnerBlockOffset:0 Size:10000 IsLargeBlock:true LargeBlockRowsCount:1} {BlockIndex:9 InnerBlockOffset:0 Size:10000 IsLargeBlock:true LargeBlockRowsCount:1} {BlockIndex:0 InnerBlockOffset:0 Size:1 IsLargeBlock:false LargeBlockRowsCount:1}] --- PASS: TestLocateData (0.00s) === RUN TestLocateData2 --- PASS: TestLocateData2 (0.00s) === RUN TestLocateData3 {BlockIndex:8876 InnerBlockOffset:912752 Size:112568 IsLargeBlock:false LargeBlockRowsCount:2} --- PASS: TestLocateData3 (0.00s) === RUN TestLocateData_ExactMultiple_Issue8947 --- PASS: TestLocateData_ExactMultiple_Issue8947 (0.00s) === RUN TestLocateData_Issue8179 --- PASS: TestLocateData_Issue8179 (0.00s) === RUN TestPositioning offset: 31300679656 size: 1167 offset: 11513014944 size: 66044 offset: 26311863528 size: 26823 interval: {BlockIndex:14852 InnerBlockOffset:994536 Size:26856 IsLargeBlock:false LargeBlockRowsCount:1}, shardId: 2, shardOffset: 2631871720 --- PASS: TestPositioning (0.00s) === RUN TestNewEcVolumeLoadsEncodeTsNs --- PASS: TestNewEcVolumeLoadsEncodeTsNs (0.00s) === RUN TestNewEcVolumeDoesNotWriteStubVif W0824 22:18:51.080577 ec_volume.go:206 vif file not found, using defaults, volumeId:124, filename:/tmp/TestNewEcVolumeDoesNotWriteStubVif802629606/001/124.vif --- PASS: TestNewEcVolumeDoesNotWriteStubVif (0.00s) === RUN TestNewEcVolumeLoadsCustomRatio --- PASS: TestNewEcVolumeLoadsCustomRatio (0.00s) === RUN TestNewEcVolumePrefersLocalEcx --- PASS: TestNewEcVolumePrefersLocalEcx (0.00s) === RUN TestNewEcVolumeZeroByteLocalYieldsToSharedEcx --- PASS: TestNewEcVolumeZeroByteLocalYieldsToSharedEcx (0.00s) === RUN TestEcVolumeDestroySweepsBothDirs --- PASS: TestEcVolumeDestroySweepsBothDirs (0.00s) === RUN TestRequireRecoverableShardSet_AllPresent --- PASS: TestRequireRecoverableShardSet_AllPresent (0.00s) === RUN TestRequireRecoverableShardSet_DegradedButRecoverable --- PASS: TestRequireRecoverableShardSet_DegradedButRecoverable (0.00s) === RUN TestRequireRecoverableShardSet_BelowDataShards --- PASS: TestRequireRecoverableShardSet_BelowDataShards (0.00s) === RUN TestRequireRecoverableShardSet_EmptyBitmap --- PASS: TestRequireRecoverableShardSet_EmptyBitmap (0.00s) === RUN TestRequireRecoverableShardSet_CustomRatio --- PASS: TestRequireRecoverableShardSet_CustomRatio (0.00s) === RUN TestRequireRecoverableShardSet_RejectsInvalidParams --- PASS: TestRequireRecoverableShardSet_RejectsInvalidParams (0.00s) === RUN TestSummarizeShardInventory_Deterministic --- PASS: TestSummarizeShardInventory_Deterministic (0.00s) === RUN TestSummarizeShardInventory_IncludesError --- PASS: TestSummarizeShardInventory_IncludesError (0.00s) === RUN TestHasLiveNeedles_AllDeletedIsFalse --- PASS: TestHasLiveNeedles_AllDeletedIsFalse (0.00s) === RUN TestHasLiveNeedles_WithLiveEntryIsTrue --- PASS: TestHasLiveNeedles_WithLiveEntryIsTrue (0.00s) === RUN TestHasLiveNeedles_EmptyFileIsFalse --- PASS: TestHasLiveNeedles_EmptyFileIsFalse (0.00s) === RUN TestHasLiveNeedles_TruncatedEcxErrors --- PASS: TestHasLiveNeedles_TruncatedEcxErrors (0.00s) === RUN TestWriteIdxFileFromEcIndex_PreservesDeletedNeedles --- PASS: TestWriteIdxFileFromEcIndex_PreservesDeletedNeedles (0.00s) === RUN TestWriteIdxFileFromEcIndex_ProcessesEcjJournal --- PASS: TestWriteIdxFileFromEcIndex_ProcessesEcjJournal (0.00s) === RUN TestDecodeAtomicPublish --- PASS: TestDecodeAtomicPublish (0.00s) === RUN TestDecodeWithNonEmptyEcj_AllDeleted --- PASS: TestDecodeWithNonEmptyEcj_AllDeleted (0.00s) === RUN TestDecodeWithNonEmptyEcj_PartiallyDeleted --- PASS: TestDecodeWithNonEmptyEcj_PartiallyDeleted (0.00s) === RUN TestDecodeWithEmptyEcj --- PASS: TestDecodeWithEmptyEcj (0.00s) === RUN TestDecodeWithNoEcjFile --- PASS: TestDecodeWithNoEcjFile (0.00s) === RUN TestEcxFileDeletionVisibleAfterSync --- PASS: TestEcxFileDeletionVisibleAfterSync (0.00s) === RUN TestEcxFileDeletionWithSeparateHandles --- PASS: TestEcxFileDeletionWithSeparateHandles (0.00s) === RUN TestEcVolumeDeleteDurableToJournal W0824 22:18:51.121098 volume_info.go:46 empty volume info file /tmp/TestEcVolumeDeleteDurableToJournal4141204035/001/test_1.vif, treating as non-existent W0824 22:18:51.121273 ec_volume.go:206 vif file not found, using defaults, volumeId:1, filename:/tmp/TestEcVolumeDeleteDurableToJournal4141204035/001/test_1.vif --- PASS: TestEcVolumeDeleteDurableToJournal (0.00s) === RUN TestScrubLocal_SuppressesDeleteStateDisagreement W0824 22:18:51.124111 volume_info.go:46 empty volume info file /tmp/TestScrubLocal_SuppressesDeleteStateDisagreement2086486131/001/test_1.vif, treating as non-existent W0824 22:18:51.124289 ec_volume.go:206 vif file not found, using defaults, volumeId:1, filename:/tmp/TestScrubLocal_SuppressesDeleteStateDisagreement2086486131/001/test_1.vif --- PASS: TestScrubLocal_SuppressesDeleteStateDisagreement (0.00s) === RUN TestScrubLocal_ReportsGenuineSizeCorruption W0824 22:18:51.126974 volume_info.go:46 empty volume info file /tmp/TestScrubLocal_ReportsGenuineSizeCorruption2110826651/001/test_1.vif, treating as non-existent W0824 22:18:51.127196 ec_volume.go:206 vif file not found, using defaults, volumeId:1, filename:/tmp/TestScrubLocal_ReportsGenuineSizeCorruption2110826651/001/test_1.vif --- PASS: TestScrubLocal_ReportsGenuineSizeCorruption (0.00s) === RUN TestEcVolumeFileAndDeleteCountInitial W0824 22:18:51.130824 volume_info.go:46 empty volume info file /tmp/TestEcVolumeFileAndDeleteCountInitial4093363322/001/test_1.vif, treating as non-existent W0824 22:18:51.130979 ec_volume.go:206 vif file not found, using defaults, volumeId:1, filename:/tmp/TestEcVolumeFileAndDeleteCountInitial4093363322/001/test_1.vif --- PASS: TestEcVolumeFileAndDeleteCountInitial (0.00s) === RUN TestEcVolumeFileAndDeleteCountAfterDelete W0824 22:18:51.133249 volume_info.go:46 empty volume info file /tmp/TestEcVolumeFileAndDeleteCountAfterDelete755122890/001/test_1.vif, treating as non-existent W0824 22:18:51.133368 ec_volume.go:206 vif file not found, using defaults, volumeId:1, filename:/tmp/TestEcVolumeFileAndDeleteCountAfterDelete755122890/001/test_1.vif --- PASS: TestEcVolumeFileAndDeleteCountAfterDelete (0.00s) === RUN TestShardsInfoDeleteParityShards --- PASS: TestShardsInfoDeleteParityShards (0.00s) === RUN TestShardsInfoAsSlice --- PASS: TestShardsInfoAsSlice (0.00s) === RUN TestShardsInfoSerialize === RUN TestShardsInfoSerialize/no_bits === RUN TestShardsInfoSerialize/single_shard,_first === RUN TestShardsInfoSerialize/single_shard,_5th === RUN TestShardsInfoSerialize/multiple_shards --- PASS: TestShardsInfoSerialize (0.00s) --- PASS: TestShardsInfoSerialize/no_bits (0.00s) --- PASS: TestShardsInfoSerialize/single_shard,_first (0.00s) --- PASS: TestShardsInfoSerialize/single_shard,_5th (0.00s) --- PASS: TestShardsInfoSerialize/multiple_shards (0.00s) === RUN TestShardsInfoFromVolumeEcShardInformationMessage === RUN TestShardsInfoFromVolumeEcShardInformationMessage/no_msg === RUN TestShardsInfoFromVolumeEcShardInformationMessage/no_shards === RUN TestShardsInfoFromVolumeEcShardInformationMessage/single_shard === RUN TestShardsInfoFromVolumeEcShardInformationMessage/multiple_shards === RUN TestShardsInfoFromVolumeEcShardInformationMessage/multiple_shards_with_missing_sizes --- PASS: TestShardsInfoFromVolumeEcShardInformationMessage (0.00s) --- PASS: TestShardsInfoFromVolumeEcShardInformationMessage/no_msg (0.00s) --- PASS: TestShardsInfoFromVolumeEcShardInformationMessage/no_shards (0.00s) --- PASS: TestShardsInfoFromVolumeEcShardInformationMessage/single_shard (0.00s) --- PASS: TestShardsInfoFromVolumeEcShardInformationMessage/multiple_shards (0.00s) --- PASS: TestShardsInfoFromVolumeEcShardInformationMessage/multiple_shards_with_missing_sizes (0.00s) === RUN TestShardsInfoCombine --- PASS: TestShardsInfoCombine (0.00s) PASS ok github.com/seaweedfs/seaweedfs/weed/storage/erasure_coding 41.818s ? github.com/seaweedfs/seaweedfs/weed/storage/erasure_coding/distribution [no test files] === RUN TestPickBestDiskOnNode === RUN TestPickBestDiskOnNode/skips_disks_with_no_free_slots === RUN TestPickBestDiskOnNode/spreads_a_volume's_shards_across_disks === RUN TestPickBestDiskOnNode/data_shard_avoids_disk_holding_parity === RUN TestPickBestDiskOnNode/anti-affinity_follows_the_supplied_ratio_boundary === RUN TestPickBestDiskOnNode/only_matching_disk_type_when_set --- PASS: TestPickBestDiskOnNode (0.15s) --- PASS: TestPickBestDiskOnNode/skips_disks_with_no_free_slots (0.14s) --- PASS: TestPickBestDiskOnNode/spreads_a_volume's_shards_across_disks (0.00s) --- PASS: TestPickBestDiskOnNode/data_shard_avoids_disk_holding_parity (0.00s) --- PASS: TestPickBestDiskOnNode/anti-affinity_follows_the_supplied_ratio_boundary (0.00s) --- PASS: TestPickBestDiskOnNode/only_matching_disk_type_when_set (0.00s) === RUN TestPlanSourceDiskAttribution --- PASS: TestPlanSourceDiskAttribution (0.00s) === RUN TestPlanSpreadsAcrossDestinationDisks --- PASS: TestPlanSpreadsAcrossDestinationDisks (0.00s) === RUN TestPlanCrossRackParityAntiAffinity --- PASS: TestPlanCrossRackParityAntiAffinity (0.00s) === RUN TestWithinRackParityAntiAffinity --- PASS: TestWithinRackParityAntiAffinity (0.00s) === RUN TestPlanReplicaPlacementCapsPerRack --- PASS: TestPlanReplicaPlacementCapsPerRack (0.00s) === RUN TestPlanDedup --- PASS: TestPlanDedup (0.04s) === RUN TestCeilDivide --- PASS: TestCeilDivide (0.00s) === RUN TestPlanConvergesToBalancedLoadOverRounds balancer_test.go:335: converged after 13 rounds: map[10.0.0.1:140 10.0.0.2:140 10.0.0.3:140 10.0.0.4:140 10.0.0.5:140 10.0.0.6:140] --- PASS: TestPlanConvergesToBalancedLoadOverRounds (0.17s) === RUN TestGlobalImbalanceMovesFromFullToEmpty --- PASS: TestGlobalImbalanceMovesFromFullToEmpty (0.00s) === RUN TestGlobalImbalanceHeterogeneousCapacity --- PASS: TestGlobalImbalanceHeterogeneousCapacity (0.00s) === RUN TestGlobalImbalanceSkipsFullNodes --- PASS: TestGlobalImbalanceSkipsFullNodes (0.00s) === RUN TestPlanBalancesSkewedDataParityWithEvenTotals --- PASS: TestPlanBalancesSkewedDataParityWithEvenTotals (0.04s) === RUN TestPlanVolumeRatioOverridesCollection --- PASS: TestPlanVolumeRatioOverridesCollection (0.00s) === RUN TestGlobalPrefersVolumeAbsentFromDestination --- PASS: TestGlobalPrefersVolumeAbsentFromDestination (0.00s) === RUN TestPlanKeepsCollectionsWithSameVolumeIdDistinct --- PASS: TestPlanKeepsCollectionsWithSameVolumeIdDistinct (0.00s) === RUN TestDedupFreesCapacityForLaterPhases --- PASS: TestDedupFreesCapacityForLaterPhases (0.00s) === RUN TestWithinRackSpreadsAcrossMachines --- PASS: TestWithinRackSpreadsAcrossMachines (0.00s) === RUN TestWithinRackMachineSpreadBalancesCombinedOccupancy --- PASS: TestWithinRackMachineSpreadBalancesCombinedOccupancy (0.00s) === RUN TestWithinRackMachineSpreadActsOnExactlyHalfSkew --- PASS: TestWithinRackMachineSpreadActsOnExactlyHalfSkew (0.00s) === RUN TestWithinRackSpreadUsesRackLocalShardCount --- PASS: TestWithinRackSpreadUsesRackLocalShardCount (0.00s) === RUN TestWithinRackNodeFallbackHonorsThreshold --- PASS: TestWithinRackNodeFallbackHonorsThreshold (0.00s) === RUN TestWithinRackSpreadDefaultsToNodes --- PASS: TestWithinRackSpreadDefaultsToNodes (0.00s) === RUN TestGlobalDoesNotConcentrateVolumeAcrossMachines --- PASS: TestGlobalDoesNotConcentrateVolumeAcrossMachines (0.00s) === RUN TestWithinRackMachineSkipsCappedMachine --- PASS: TestWithinRackMachineSkipsCappedMachine (0.00s) === RUN TestGlobalPrefersVolumeAbsentFromDestinationMachine --- PASS: TestGlobalPrefersVolumeAbsentFromDestinationMachine (0.00s) === RUN TestPlaceStrictSpreadAndCaps --- PASS: TestPlaceStrictSpreadAndCaps (0.00s) === RUN TestPlaceDistributesEvenlyAcrossManyVolumes --- PASS: TestPlaceDistributesEvenlyAcrossManyVolumes (0.15s) === RUN TestPlaceSpreadsAcrossMachines --- PASS: TestPlaceSpreadsAcrossMachines (0.00s) === RUN TestPlaceStrictFailsAndRollsBack --- PASS: TestPlaceStrictFailsAndRollsBack (0.00s) === RUN TestPlaceDurabilityFirstRelaxesRP --- PASS: TestPlaceDurabilityFirstRelaxesRP (0.00s) === RUN TestPlaceSameRackCountIsDirectPerNodeCap --- PASS: TestPlaceSameRackCountIsDirectPerNodeCap (0.00s) === RUN TestPlaceDiskTypeHardFilter --- PASS: TestPlaceDiskTypeHardFilter (0.00s) === RUN TestPlaceDiskTypeUnavailableFails --- PASS: TestPlaceDiskTypeUnavailableFails (0.01s) === RUN TestPlaceHDDRequestMatchesEmptyTypeDisks --- PASS: TestPlaceHDDRequestMatchesEmptyTypeDisks (0.00s) === RUN TestPlaceDurabilityCapRejectsSkewed --- PASS: TestPlaceDurabilityCapRejectsSkewed (0.00s) === RUN TestReleaseVolumeShards --- PASS: TestReleaseVolumeShards (0.00s) === RUN TestClearShardAccounting --- PASS: TestClearShardAccounting (0.00s) === RUN TestPlaceDiskTypePreferSpills --- PASS: TestPlaceDiskTypePreferSpills (0.00s) === RUN TestPlacePreferredTagsUseTaggedDisks --- PASS: TestPlacePreferredTagsUseTaggedDisks (0.03s) === RUN TestPlacePreferredTagsSpillWhenInsufficient --- PASS: TestPlacePreferredTagsSpillWhenInsufficient (0.00s) === RUN TestPlaceStrictCapsCountEligibleRacks --- PASS: TestPlaceStrictCapsCountEligibleRacks (0.00s) === RUN TestFromActiveTopology --- PASS: TestFromActiveTopology (0.00s) === RUN TestFromActiveTopologySkipsPhysicallyFullDisk --- PASS: TestFromActiveTopologySkipsPhysicallyFullDisk (0.00s) === RUN TestFromActiveTopologyGroupsByAddressHost --- PASS: TestFromActiveTopologyGroupsByAddressHost (0.00s) === RUN TestEcShardSlotsOnDiskRoundsUp --- PASS: TestEcShardSlotsOnDiskRoundsUp (0.00s) PASS ok github.com/seaweedfs/seaweedfs/weed/storage/erasure_coding/ecbalancer 15.808s === RUN TestCheckIndexFile_IgnoresOffset0Tombstone --- PASS: TestCheckIndexFile_IgnoresOffset0Tombstone (0.05s) === RUN TestCheckIndexFile === RUN TestCheckIndexFile/healthy_index === RUN TestCheckIndexFile/healthy_index_with_deleted_files === RUN TestCheckIndexFile/damaged_index_(bitrot) === RUN TestCheckIndexFile/damaged_index_(truncated) === RUN TestCheckIndexFile/healthy_EC_index_with_deleted_files === RUN TestCheckIndexFile/damaged_EC_index_(bitrot) --- PASS: TestCheckIndexFile (0.02s) --- PASS: TestCheckIndexFile/healthy_index (0.00s) --- PASS: TestCheckIndexFile/healthy_index_with_deleted_files (0.00s) --- PASS: TestCheckIndexFile/damaged_index_(bitrot) (0.00s) --- PASS: TestCheckIndexFile/damaged_index_(truncated) (0.00s) --- PASS: TestCheckIndexFile/healthy_EC_index_with_deleted_files (0.00s) --- PASS: TestCheckIndexFile/damaged_EC_index_(bitrot) (0.00s) PASS ok github.com/seaweedfs/seaweedfs/weed/storage/idx 11.532s === RUN TestParseFileIdFromString --- PASS: TestParseFileIdFromString (0.00s) === RUN TestNeedleIdFileId --- PASS: TestNeedleIdFileId (0.00s) === RUN TestEagerPreGrow === RUN TestEagerPreGrow/small_content-length_grows_exactly === RUN TestEagerPreGrow/content-length_at_cap_grows_to_cap === RUN TestEagerPreGrow/content-length_above_cap_is_clamped_to_cap === RUN TestEagerPreGrow/content-length_above_sizeLimit_is_rejected === RUN TestEagerPreGrow/zero_content-length_is_a_no-op === RUN TestEagerPreGrow/negative_content-length_(chunked_encoding)_is_a_no-op === RUN TestEagerPreGrow/buffer_with_sufficient_cap_is_left_alone --- PASS: TestEagerPreGrow (0.25s) --- PASS: TestEagerPreGrow/small_content-length_grows_exactly (0.01s) --- PASS: TestEagerPreGrow/content-length_at_cap_grows_to_cap (0.02s) --- PASS: TestEagerPreGrow/content-length_above_cap_is_clamped_to_cap (0.06s) --- PASS: TestEagerPreGrow/content-length_above_sizeLimit_is_rejected (0.00s) --- PASS: TestEagerPreGrow/zero_content-length_is_a_no-op (0.00s) --- PASS: TestEagerPreGrow/negative_content-length_(chunked_encoding)_is_a_no-op (0.00s) --- PASS: TestEagerPreGrow/buffer_with_sufficient_cap_is_left_alone (0.06s) === RUN TestParseUpload_GzipStreamCount === RUN TestParseUpload_GzipStreamCount/no_MD5:_stream-count needle_parse_upload_stream_test.go:73: allocated=111184 bytes (gzipped=16826, uncompressed=4194304) === RUN TestParseUpload_GzipStreamCount/MD5_on_part:_materialize needle_parse_upload_stream_test.go:73: allocated=16839760 bytes (gzipped=16826, uncompressed=4194304) === RUN TestParseUpload_GzipStreamCount/MD5_on_request:_materialize needle_parse_upload_stream_test.go:73: allocated=16886544 bytes (gzipped=16826, uncompressed=4194304) --- PASS: TestParseUpload_GzipStreamCount (1.64s) --- PASS: TestParseUpload_GzipStreamCount/no_MD5:_stream-count (0.15s) --- PASS: TestParseUpload_GzipStreamCount/MD5_on_part:_materialize (0.55s) --- PASS: TestParseUpload_GzipStreamCount/MD5_on_request:_materialize (0.39s) === RUN TestParseUploadUsesDiscoveredFilePartContentType --- PASS: TestParseUploadUsesDiscoveredFilePartContentType (0.01s) === RUN TestReadNeedleTailLegacyChecksum --- PASS: TestReadNeedleTailLegacyChecksum (0.00s) === RUN TestReadNeedleTailRawChecksum --- PASS: TestReadNeedleTailRawChecksum (0.00s) === RUN TestReadNeedleTailCorrupted --- PASS: TestReadNeedleTailCorrupted (0.00s) === RUN TestParseKeyHash --- PASS: TestParseKeyHash (0.00s) === RUN TestNeedleParsePathRejectsDeltaOverflow --- PASS: TestNeedleParsePathRejectsDeltaOverflow (0.00s) === RUN TestAppend --- PASS: TestAppend (0.00s) === RUN TestWriteNeedle_CompatibilityWithLegacy === RUN TestWriteNeedle_CompatibilityWithLegacy/Version1 === RUN TestWriteNeedle_CompatibilityWithLegacy/Version2 === RUN TestWriteNeedle_CompatibilityWithLegacy/Version3 --- PASS: TestWriteNeedle_CompatibilityWithLegacy (0.00s) --- PASS: TestWriteNeedle_CompatibilityWithLegacy/Version1 (0.00s) --- PASS: TestWriteNeedle_CompatibilityWithLegacy/Version2 (0.00s) --- PASS: TestWriteNeedle_CompatibilityWithLegacy/Version3 (0.00s) === RUN TestNewVolumeId volume_id_test.go:11: a is not legal volume id, strconv.ParseUint: parsing "a": invalid syntax --- PASS: TestNewVolumeId (0.00s) === RUN TestVolumeId_String --- PASS: TestVolumeId_String (0.00s) === RUN TestVolumeId_Next --- PASS: TestVolumeId_Next (0.00s) === RUN TestTTLReadWrite --- PASS: TestTTLReadWrite (0.00s) === RUN TestLoadTTLIsInterned --- PASS: TestLoadTTLIsInterned (0.01s) PASS ok github.com/seaweedfs/seaweedfs/weed/storage/needle 12.071s === RUN TestMemoryUsage Each 14.16 Bytes Alloc = 22 MiB TotalAlloc = 48 MiB Sys = 47 MiB NumGC = 8 CompactMap = 1539351/1749659 elements on 2198 segments, 87.98% efficiency Taken = 15.635542089s Each 13.33 Bytes Alloc = 43 MiB TotalAlloc = 94 MiB Sys = 85 MiB NumGC = 10 CompactMap = 1539351/1749659 elements on 2198 segments, 87.98% efficiency Taken = 14.673568685s Each 13.06 Bytes Alloc = 63 MiB TotalAlloc = 139 MiB Sys = 113 MiB NumGC = 12 CompactMap = 1539351/1749659 elements on 2198 segments, 87.98% efficiency Taken = 9.462410609s Each 12.92 Bytes Alloc = 83 MiB TotalAlloc = 185 MiB Sys = 137 MiB NumGC = 13 CompactMap = 1539351/1749659 elements on 2198 segments, 87.98% efficiency Taken = 7.481488942s Each 12.83 Bytes Alloc = 103 MiB TotalAlloc = 231 MiB Sys = 161 MiB NumGC = 14 CompactMap = 1539351/1749659 elements on 2198 segments, 87.98% efficiency Taken = 7.111382451s Each 12.77 Bytes Alloc = 123 MiB TotalAlloc = 276 MiB Sys = 177 MiB NumGC = 15 CompactMap = 1539351/1749659 elements on 2198 segments, 87.98% efficiency Taken = 6.867196057s Each 12.73 Bytes Alloc = 144 MiB TotalAlloc = 322 MiB Sys = 198 MiB NumGC = 16 CompactMap = 1539351/1749659 elements on 2198 segments, 87.98% efficiency Taken = 6.763536953s Each 12.70 Bytes Alloc = 164 MiB TotalAlloc = 367 MiB Sys = 218 MiB NumGC = 17 CompactMap = 1539351/1749659 elements on 2198 segments, 87.98% efficiency Taken = 6.436792008s Each 12.68 Bytes Alloc = 184 MiB TotalAlloc = 413 MiB Sys = 250 MiB NumGC = 18 CompactMap = 1539351/1749659 elements on 2198 segments, 87.98% efficiency Taken = 7.198999341s Each 12.66 Bytes Alloc = 204 MiB TotalAlloc = 458 MiB Sys = 266 MiB NumGC = 19 CompactMap = 1539351/1749659 elements on 2198 segments, 87.98% efficiency Taken = 6.738183062s --- PASS: TestMemoryUsage (88.37s) === RUN TestSegmentBsearchKey === RUN TestSegmentBsearchKey/empty_segment === RUN TestSegmentBsearchKey/new_key,_insert_at_beginning === RUN TestSegmentBsearchKey/new_key,_insert_at_end === RUN TestSegmentBsearchKey/new_key,_insert_second === RUN TestSegmentBsearchKey/new_key,_insert_in_middle === RUN TestSegmentBsearchKey/key_#1 === RUN TestSegmentBsearchKey/key_#2 === RUN TestSegmentBsearchKey/key_#3 === RUN TestSegmentBsearchKey/key_#4 === RUN TestSegmentBsearchKey/key_#5 --- PASS: TestSegmentBsearchKey (0.00s) --- PASS: TestSegmentBsearchKey/empty_segment (0.00s) --- PASS: TestSegmentBsearchKey/new_key,_insert_at_beginning (0.00s) --- PASS: TestSegmentBsearchKey/new_key,_insert_at_end (0.00s) --- PASS: TestSegmentBsearchKey/new_key,_insert_second (0.00s) --- PASS: TestSegmentBsearchKey/new_key,_insert_in_middle (0.00s) --- PASS: TestSegmentBsearchKey/key_#1 (0.00s) --- PASS: TestSegmentBsearchKey/key_#2 (0.00s) --- PASS: TestSegmentBsearchKey/key_#3 (0.00s) --- PASS: TestSegmentBsearchKey/key_#4 (0.00s) --- PASS: TestSegmentBsearchKey/key_#5 (0.00s) === RUN TestSegmentSet --- PASS: TestSegmentSet (0.00s) === RUN TestSegmentSetOrdering --- PASS: TestSegmentSetOrdering (6.78s) === RUN TestSegmentGet === RUN TestSegmentGet/invalid_key === RUN TestSegmentGet/key_#1 === RUN TestSegmentGet/key_#2 === RUN TestSegmentGet/key_#3 --- PASS: TestSegmentGet (0.00s) --- PASS: TestSegmentGet/invalid_key (0.00s) --- PASS: TestSegmentGet/key_#1 (0.00s) --- PASS: TestSegmentGet/key_#2 (0.00s) --- PASS: TestSegmentGet/key_#3 (0.00s) === RUN TestSegmentDelete --- PASS: TestSegmentDelete (0.00s) === RUN TestSegmentForKey === RUN TestSegmentForKey/first_segment === RUN TestSegmentForKey/second_segment,_gapless === RUN TestSegmentForKey/gapped_segment --- PASS: TestSegmentForKey (0.00s) --- PASS: TestSegmentForKey/first_segment (0.00s) --- PASS: TestSegmentForKey/second_segment,_gapless (0.00s) --- PASS: TestSegmentForKey/gapped_segment (0.00s) === RUN TestAscendingVisit --- PASS: TestAscendingVisit (0.00s) === RUN TestRandomInsert --- PASS: TestRandomInsert (131.46s) PASS ok github.com/seaweedfs/seaweedfs/weed/storage/needle_map 237.744s === RUN TestMemoryUsage Each 16.35 Bytes Alloc = 26 MiB TotalAlloc = 116 MiB Sys = 61 MiB NumGC = 14 Taken = 14.538928052s Each 15.51 Bytes Alloc = 50 MiB TotalAlloc = 230 MiB Sys = 101 MiB NumGC = 18 Taken = 13.947160019s Each 15.23 Bytes Alloc = 73 MiB TotalAlloc = 343 MiB Sys = 145 MiB NumGC = 21 Taken = 8.962845764s Each 15.09 Bytes Alloc = 97 MiB TotalAlloc = 457 MiB Sys = 173 MiB NumGC = 23 Taken = 6.883294191s Each 15.01 Bytes Alloc = 121 MiB TotalAlloc = 570 MiB Sys = 217 MiB NumGC = 25 Taken = 6.937790433s Each 14.96 Bytes Alloc = 145 MiB TotalAlloc = 683 MiB Sys = 262 MiB NumGC = 26 Taken = 6.291537358s Each 14.92 Bytes Alloc = 168 MiB TotalAlloc = 797 MiB Sys = 290 MiB NumGC = 27 Taken = 6.080628212s Each 14.89 Bytes Alloc = 192 MiB TotalAlloc = 910 MiB Sys = 319 MiB NumGC = 28 Taken = 5.869016067s Each 14.87 Bytes Alloc = 216 MiB TotalAlloc = 1024 MiB Sys = 352 MiB NumGC = 29 Taken = 5.6561267s Each 14.85 Bytes Alloc = 240 MiB TotalAlloc = 1137 MiB Sys = 380 MiB NumGC = 30 Taken = 5.942770081s --- PASS: TestMemoryUsage (81.11s) === RUN TestSnowflakeSequencer I0824 22:19:42.301509 snowflake_sequencer.go:24 use snowflake seq id generator, nodeid:for_test hex_of_nodeid: 1 --- PASS: TestSnowflakeSequencer (0.29s) === RUN TestOverflow2 needle key: 150073 needle key: 150076 needle key: 150088 needle key: 150089 needle key: 150124 needle key: 150137 needle key: 150145 needle key: 150147 needle key: 150158 needle key: 150162 --- PASS: TestOverflow2 (0.00s) === RUN TestIssue52 key 10002 ok true 10002, 1250, 10002 key 10002 ok true 10002, 1250, 10002 --- PASS: TestIssue52 (0.00s) === RUN TestCompactMap --- PASS: TestCompactMap (0.56s) === RUN TestOverflow overflow[ 0 ]: 1 overflow[ 1 ]: 2 overflow[ 2 ]: 3 overflow[ 3 ]: 4 overflow[ 4 ]: 5 overflow[ 0 ]: 1 size -12 overflow[ 1 ]: 2 size 12 overflow[ 2 ]: 3 size 24 overflow[ 3 ]: 4 size -12 overflow[ 4 ]: 5 size 12 overflow[ 0 ]: 1 overflow[ 1 ]: 2 overflow[ 2 ]: 3 overflow[ 3 ]: 4 overflow[ 4 ]: 5 overflow[ 0 ]: 1 overflow[ 1 ]: 2 overflow[ 2 ]: 3 overflow[ 3 ]: 4 overflow[ 4 ]: 5 --- PASS: TestOverflow (0.00s) === RUN TestCompactSection_Get compact_map_test.go:204: 1574318345753513987 compact_map_test.go:215: 1574318350048481283 --- PASS: TestCompactSection_Get (5.79s) === RUN TestCompactSection_PutOutOfOrderItemBeyondLookBackWindow --- PASS: TestCompactSection_PutOutOfOrderItemBeyondLookBackWindow (0.00s) PASS ok github.com/seaweedfs/seaweedfs/weed/storage/needle_map/old 99.643s === RUN TestReplicaPlacementFromByteMatchesString --- PASS: TestReplicaPlacementFromByteMatchesString (0.04s) === RUN TestReplicaPlacementSerialDeserial --- PASS: TestReplicaPlacementSerialDeserial (0.00s) === RUN TestReplicaPlacementHasReplication === RUN TestReplicaPlacementHasReplication/empty_replica_placement === RUN TestReplicaPlacementHasReplication/no_replication === RUN TestReplicaPlacementHasReplication/same_rack_replication === RUN TestReplicaPlacementHasReplication/diff_rack_replication === RUN TestReplicaPlacementHasReplication/DC_replication === RUN TestReplicaPlacementHasReplication/full_replication --- PASS: TestReplicaPlacementHasReplication (0.03s) --- PASS: TestReplicaPlacementHasReplication/empty_replica_placement (0.00s) --- PASS: TestReplicaPlacementHasReplication/no_replication (0.00s) --- PASS: TestReplicaPlacementHasReplication/same_rack_replication (0.00s) --- PASS: TestReplicaPlacementHasReplication/diff_rack_replication (0.00s) --- PASS: TestReplicaPlacementHasReplication/DC_replication (0.00s) --- PASS: TestReplicaPlacementHasReplication/full_replication (0.00s) === RUN TestSuperBlockReadWrite --- PASS: TestSuperBlockReadWrite (0.00s) PASS ok github.com/seaweedfs/seaweedfs/weed/storage/super_block 12.009s ? github.com/seaweedfs/seaweedfs/weed/storage/types [no test files] ? github.com/seaweedfs/seaweedfs/weed/storage/volume_info [no test files] === RUN TestBuildUnionFromMultipleIndexDatabases --- PASS: TestBuildUnionFromMultipleIndexDatabases (0.19s) === RUN TestFindLargestReplica === RUN TestFindLargestReplica/single_replica === RUN TestFindLargestReplica/first_is_largest === RUN TestFindLargestReplica/last_is_largest === RUN TestFindLargestReplica/middle_is_largest === RUN TestFindLargestReplica/all_equal_-_pick_first --- PASS: TestFindLargestReplica (0.01s) --- PASS: TestFindLargestReplica/single_replica (0.00s) --- PASS: TestFindLargestReplica/first_is_largest (0.00s) --- PASS: TestFindLargestReplica/last_is_largest (0.00s) --- PASS: TestFindLargestReplica/middle_is_largest (0.00s) --- PASS: TestFindLargestReplica/all_equal_-_pick_first (0.00s) === RUN TestDeletedEntriesAreSkipped --- PASS: TestDeletedEntriesAreSkipped (0.02s) === RUN TestReplicaUnionBuilder_EmptyLocations --- PASS: TestReplicaUnionBuilder_EmptyLocations (0.00s) === RUN TestAvoidDuplicateCopies --- PASS: TestAvoidDuplicateCopies (0.01s) PASS ok github.com/seaweedfs/seaweedfs/weed/storage/volume_replica 15.362s === RUN TestCollectVolumeStatsCountsEcShards I0824 22:18:27.172813 node.go:471 weedfs adds child dc1 I0824 22:18:27.219082 node.go:471 weedfs:dc1 adds child rack1 I0824 22:18:27.219838 node.go:471 weedfs:dc1:rack1 adds child 127.0.0.1:34534 I0824 22:18:27.220428 node.go:471 weedfs:dc1:rack1:127.0.0.1:34534 adds child I0824 22:18:27.220671 node.go:471 weedfs:dc1:rack1 adds child 127.0.0.2:34534 I0824 22:18:27.220849 node.go:471 weedfs:dc1:rack1:127.0.0.2:34534 adds child --- PASS: TestCollectVolumeStatsCountsEcShards (0.10s) PASS ok github.com/seaweedfs/seaweedfs/weed/telemetry 17.821s === RUN TestCollectionStatisticsMatchesAFullListing I0824 22:18:24.634764 node.go:471 stats adds child dc1 I0824 22:18:24.642213 node.go:471 stats:dc1 adds child rack1 I0824 22:18:24.642452 node.go:471 stats:dc1:rack1 adds child 10.0.0.1:8080 I0824 22:18:24.642647 node.go:471 stats:dc1:rack1:10.0.0.1:8080 adds child I0824 22:18:24.642748 node.go:471 stats:dc1:rack1:10.0.0.1:8080 adds child ssd I0824 22:18:24.642852 node.go:471 stats:dc1:rack1 adds child 10.0.0.2:8080 I0824 22:18:24.642898 node.go:471 stats:dc1:rack1:10.0.0.2:8080 adds child I0824 22:18:24.642945 node.go:471 stats:dc1:rack1:10.0.0.2:8080 adds child ssd I0824 22:18:24.642987 node.go:471 stats:dc1:rack1 adds child 10.0.0.3:8080 I0824 22:18:24.643021 node.go:471 stats:dc1:rack1:10.0.0.3:8080 adds child I0824 22:18:24.645747 node.go:471 stats:dc1:rack1:10.0.0.3:8080 adds child ssd --- PASS: TestCollectionStatisticsMatchesAFullListing (0.31s) === RUN TestCollectionStatisticsPicksTheLargestReplica I0824 22:18:24.680931 node.go:471 stats adds child dc1 I0824 22:18:24.681169 node.go:471 stats:dc1 adds child rack1 I0824 22:18:24.681367 node.go:471 stats:dc1:rack1 adds child a I0824 22:18:24.681504 node.go:471 stats:dc1:rack1:a adds child I0824 22:18:24.681551 node.go:471 stats:dc1:rack1 adds child b I0824 22:18:24.681590 node.go:471 stats:dc1:rack1:b adds child --- PASS: TestCollectionStatisticsPicksTheLargestReplica (0.04s) === RUN TestCollectionStatisticsTakesTheLargestEcFileCount I0824 22:18:24.684275 node.go:471 stats adds child dc1 I0824 22:18:24.684421 node.go:471 stats:dc1 adds child rack1 I0824 22:18:24.684481 node.go:471 stats:dc1:rack1 adds child a I0824 22:18:24.684587 node.go:471 stats:dc1:rack1:a adds child I0824 22:18:24.684631 node.go:471 stats:dc1:rack1 adds child b I0824 22:18:24.684687 node.go:471 stats:dc1:rack1:b adds child --- PASS: TestCollectionStatisticsTakesTheLargestEcFileCount (0.00s) === RUN TestCollectionStatisticsCountsRegularAndEcTogether I0824 22:18:24.685237 node.go:471 stats adds child dc1 I0824 22:18:24.685277 node.go:471 stats:dc1 adds child rack1 I0824 22:18:24.685342 node.go:471 stats:dc1:rack1 adds child a I0824 22:18:24.685431 node.go:471 stats:dc1:rack1:a adds child --- PASS: TestCollectionStatisticsCountsRegularAndEcTogether (0.00s) === RUN TestCollectionStatisticsPicksTheReplicaHoldingTheMostLiveData I0824 22:18:24.687758 node.go:471 stats adds child dc1 I0824 22:18:24.687908 node.go:471 stats:dc1 adds child rack1 I0824 22:18:24.688018 node.go:471 stats:dc1:rack1 adds child a I0824 22:18:24.688102 node.go:471 stats:dc1:rack1:a adds child I0824 22:18:24.688174 node.go:471 stats:dc1:rack1 adds child b I0824 22:18:24.688295 node.go:471 stats:dc1:rack1:b adds child --- PASS: TestCollectionStatisticsPicksTheReplicaHoldingTheMostLiveData (0.00s) === RUN TestCollectionStatisticsDeletionsNeverExceedTheVolume I0824 22:18:24.720469 node.go:471 stats adds child dc1 I0824 22:18:24.720739 node.go:471 stats:dc1 adds child rack1 I0824 22:18:24.720896 node.go:471 stats:dc1:rack1 adds child a I0824 22:18:24.721167 node.go:471 stats:dc1:rack1:a adds child --- PASS: TestCollectionStatisticsDeletionsNeverExceedTheVolume (0.03s) === RUN TestCrowdedVolumesAreOnesThatCanTakeWrites I0824 22:18:24.721744 node.go:471 crowd adds child dc1 I0824 22:18:24.721774 node.go:471 crowd:dc1 adds child rack1 I0824 22:18:24.721812 node.go:471 crowd:dc1:rack1 adds child 127.0.0.1:8080 I0824 22:18:24.721875 node.go:471 crowd:dc1:rack1:127.0.0.1:8080 adds child I0824 22:18:24.722005 volume_layout.go:959 Volume 1 becomes crowded --- PASS: TestCrowdedVolumesAreOnesThatCanTakeWrites (0.00s) === RUN TestCrowdedCountStillMatchesWritables I0824 22:18:24.724273 node.go:471 crowd adds child dc1 I0824 22:18:24.724325 node.go:471 crowd:dc1 adds child rack1 I0824 22:18:24.724378 node.go:471 crowd:dc1:rack1 adds child 127.0.0.1:8080 I0824 22:18:24.724416 node.go:471 crowd:dc1:rack1:127.0.0.1:8080 adds child I0824 22:18:24.724490 volume_layout.go:959 Volume 1 becomes crowded --- PASS: TestCrowdedCountStillMatchesWritables (0.00s) === RUN TestEcShardsAcrossMultipleDisksOnSameNode I0824 22:18:24.724800 node.go:471 weedfs adds child dc1 I0824 22:18:24.724934 node.go:471 weedfs:dc1 adds child rack1 I0824 22:18:24.725018 node.go:471 weedfs:dc1:rack1 adds child 127.0.0.1:34534 I0824 22:18:24.725073 node.go:471 weedfs:dc1:rack1:127.0.0.1:34534 adds child --- PASS: TestEcShardsAcrossMultipleDisksOnSameNode (0.00s) === RUN TestEcShardsAfterRestartHeartbeat I0824 22:18:24.725851 node.go:471 weedfs adds child dc1 I0824 22:18:24.725898 node.go:471 weedfs:dc1 adds child rack1 I0824 22:18:24.726018 node.go:471 weedfs:dc1:rack1 adds child 127.0.0.1:34534 I0824 22:18:24.726110 node.go:471 weedfs:dc1:rack1:127.0.0.1:34534 adds child --- PASS: TestEcShardsAfterRestartHeartbeat (0.00s) === RUN TestToDataNodeInfoReportsEmptyPhysicalDisks I0824 22:18:24.730467 node.go:471 weedfs adds child dc1 I0824 22:18:24.730517 node.go:471 weedfs:dc1 adds child rack1 I0824 22:18:24.730572 node.go:471 weedfs:dc1:rack1 adds child 127.0.0.1:34534 I0824 22:18:24.730792 node.go:471 weedfs:dc1:rack1:127.0.0.1:34534 adds child --- PASS: TestToDataNodeInfoReportsEmptyPhysicalDisks (0.00s) === RUN TestToDataNodeInfoKeepsZeroCapacityDisk I0824 22:18:24.731066 node.go:471 weedfs adds child dc1 I0824 22:18:24.731098 node.go:471 weedfs:dc1 adds child rack1 I0824 22:18:24.731130 node.go:471 weedfs:dc1:rack1 adds child 127.0.0.1:34534 I0824 22:18:24.731165 node.go:471 weedfs:dc1:rack1:127.0.0.1:34534 adds child --- PASS: TestToDataNodeInfoKeepsZeroCapacityDisk (0.00s) === RUN TestToDataNodeInfoFallsBackWhenNoCapacityReported I0824 22:18:24.731315 node.go:471 weedfs adds child dc1 I0824 22:18:24.731343 node.go:471 weedfs:dc1 adds child rack1 I0824 22:18:24.731374 node.go:471 weedfs:dc1:rack1 adds child 127.0.0.1:34534 I0824 22:18:24.731431 node.go:471 weedfs:dc1:rack1:127.0.0.1:34534 adds child --- PASS: TestToDataNodeInfoFallsBackWhenNoCapacityReported (0.00s) === RUN TestDiskInfoReportsEveryVolume I0824 22:18:24.731575 node.go:471 diskinfo adds child dc1 I0824 22:18:24.731690 node.go:471 diskinfo:dc1 adds child rack1 I0824 22:18:24.731742 node.go:471 diskinfo:dc1:rack1 adds child 10.0.0.1:8080 I0824 22:18:24.731821 node.go:471 diskinfo:dc1:rack1:10.0.0.1:8080 adds child --- PASS: TestDiskInfoReportsEveryVolume (0.00s) === RUN TestDiskInfoReportsTheDiskIdOfEcOnlyDisks I0824 22:18:24.735017 node.go:471 diskinfo adds child dc1 I0824 22:18:24.735072 node.go:471 diskinfo:dc1 adds child rack1 I0824 22:18:24.735219 node.go:471 diskinfo:dc1:rack1 adds child 10.0.0.1:8080 I0824 22:18:24.735387 node.go:471 diskinfo:dc1:rack1:10.0.0.1:8080 adds child --- PASS: TestDiskInfoReportsTheDiskIdOfEcOnlyDisks (0.00s) === RUN TestProvisionalUpdateKeepsReportedDiskId --- PASS: TestProvisionalUpdateKeepsReportedDiskId (0.03s) === RUN TestRaceConditionStress I0824 22:18:24.765654 node.go:471 weedfs adds child dc1 I0824 22:18:24.765809 node.go:471 weedfs:dc1 adds child rack1 I0824 22:18:24.765865 node.go:471 weedfs:dc1:rack1 adds child server1 I0824 22:18:24.765906 node.go:471 weedfs:dc1:rack1:server1 adds child I0824 22:18:24.765978 node.go:471 weedfs:dc1:rack1 adds child server2 I0824 22:18:24.766056 node.go:471 weedfs:dc1:rack1:server2 adds child I0824 22:18:24.766104 node.go:471 weedfs:dc1:rack1 adds child server3 I0824 22:18:24.766189 node.go:471 weedfs:dc1:rack1:server3 adds child race_condition_stress_test.go:108: Test completed in 92.037548ms race_condition_stress_test.go:109: Successful allocations: 50 race_condition_stress_test.go:110: Failed allocations: 0 race_condition_stress_test.go:111: Total volumes created: 50 race_condition_stress_test.go:120: Server 1: 15 volumes (max: 40) race_condition_stress_test.go:120: Server 2: 12 volumes (max: 40) race_condition_stress_test.go:120: Server 3: 23 volumes (max: 40) race_condition_stress_test.go:148: Race condition test passed: Capacity limits respected with 50 concurrent requests --- PASS: TestRaceConditionStress (0.09s) === RUN TestCapacityJudgmentAccuracy I0824 22:18:24.861961 node.go:471 weedfs adds child dc1 I0824 22:18:24.862022 node.go:471 weedfs:dc1 adds child rack1 I0824 22:18:24.862210 node.go:471 weedfs:dc1:rack1 adds child server1 I0824 22:18:24.862393 node.go:471 weedfs:dc1:rack1:server1 adds child race_condition_stress_test.go:226: Step 0: Volume count after update: 1 race_condition_stress_test.go:226: Step 1: Volume count after update: 2 race_condition_stress_test.go:226: Step 2: Volume count after update: 3 race_condition_stress_test.go:226: Step 3: Volume count after update: 4 race_condition_stress_test.go:226: Step 4: Volume count after update: 5 race_condition_stress_test.go:226: Step 5: Volume count after update: 6 race_condition_stress_test.go:226: Step 6: Volume count after update: 7 race_condition_stress_test.go:226: Step 7: Volume count after update: 8 race_condition_stress_test.go:226: Step 8: Volume count after update: 9 race_condition_stress_test.go:226: Step 9: Volume count after update: 10 I0824 22:18:24.869354 node.go:220 weedfs failed to pick 1 from 0 node candidates race_condition_stress_test.go:252: Capacity judgment accuracy test passed --- PASS: TestCapacityJudgmentAccuracy (0.01s) === RUN TestReservationSystemPerformance I0824 22:18:24.896788 node.go:471 weedfs adds child dc1 I0824 22:18:24.897020 node.go:471 weedfs:dc1 adds child rack1 I0824 22:18:24.897082 node.go:471 weedfs:dc1:rack1 adds child server1 I0824 22:18:24.897147 node.go:471 weedfs:dc1:rack1:server1 adds child race_condition_stress_test.go:299: Performance: 1000 reservations in 46.829243ms (avg: 46.829µs per reservation) race_condition_stress_test.go:306: Performance test passed: 46.829µs per reservation --- PASS: TestReservationSystemPerformance (0.07s) === RUN TestDisk_GetEcShards_Race --- PASS: TestDisk_GetEcShards_Race (4.79s) === RUN TestClassifyReplicationError === RUN TestClassifyReplicationError/deadline_exceeded === RUN TestClassifyReplicationError/context_cancelled === RUN TestClassifyReplicationError/connection_refused === RUN TestClassifyReplicationError/generic_error === RUN TestClassifyReplicationError/nil_error --- PASS: TestClassifyReplicationError (0.00s) --- PASS: TestClassifyReplicationError/deadline_exceeded (0.00s) --- PASS: TestClassifyReplicationError/context_cancelled (0.00s) --- PASS: TestClassifyReplicationError/connection_refused (0.00s) --- PASS: TestClassifyReplicationError/generic_error (0.00s) --- PASS: TestClassifyReplicationError/nil_error (0.00s) === RUN TestDistributedOperationCancelsSiblingsOnFirstError --- PASS: TestDistributedOperationCancelsSiblingsOnFirstError (0.00s) === RUN TestDistributedOperationEmpty --- PASS: TestDistributedOperationEmpty (0.00s) === RUN TestReplicatedWriteForwardsFsyncToReplicas === RUN TestReplicatedWriteForwardsFsyncToReplicas/fsync_requested === RUN TestReplicatedWriteForwardsFsyncToReplicas/no_fsync_requested --- PASS: TestReplicatedWriteForwardsFsyncToReplicas (0.22s) --- PASS: TestReplicatedWriteForwardsFsyncToReplicas/fsync_requested (0.20s) --- PASS: TestReplicatedWriteForwardsFsyncToReplicas/no_fsync_requested (0.00s) === RUN TestRegisterEcShardsAdvancesMaxVolumeId I0824 22:18:29.968596 node.go:471 weedfs adds child dc1 I0824 22:18:29.968722 node.go:471 weedfs:dc1 adds child rack1 I0824 22:18:29.968895 node.go:471 weedfs:dc1:rack1 adds child 127.0.0.1:34534 I0824 22:18:29.969014 node.go:471 weedfs:dc1:rack1:127.0.0.1:34534 adds child --- PASS: TestRegisterEcShardsAdvancesMaxVolumeId (0.00s) === RUN TestIncrementalSyncDataNodeEcShardsAdvancesMaxVolumeId I0824 22:18:29.969338 node.go:471 weedfs adds child dc1 I0824 22:18:29.969378 node.go:471 weedfs:dc1 adds child rack1 I0824 22:18:29.969417 node.go:471 weedfs:dc1:rack1 adds child 127.0.0.1:34534 I0824 22:18:29.969458 node.go:471 weedfs:dc1:rack1:127.0.0.1:34534 adds child --- PASS: TestIncrementalSyncDataNodeEcShardsAdvancesMaxVolumeId (0.00s) === RUN TestCollectionVolumeStats I0824 22:18:29.969626 node.go:471 weedfs adds child dc1 I0824 22:18:29.969695 node.go:471 weedfs:dc1 adds child rack1 I0824 22:18:29.969732 node.go:471 weedfs:dc1:rack1 adds child 127.0.0.1:34534 I0824 22:18:29.969838 node.go:471 weedfs:dc1:rack1:127.0.0.1:34534 adds child I0824 22:18:29.969876 node.go:471 weedfs:dc1:rack1:127.0.0.1:34534 adds child ssd I0824 22:18:29.970108 node.go:471 weedfs:dc1:rack1 adds child 127.0.0.2:34534 I0824 22:18:29.970290 node.go:471 weedfs:dc1:rack1:127.0.0.2:34534 adds child I0824 22:18:29.970329 node.go:471 weedfs:dc1:rack1:127.0.0.2:34534 adds child ssd --- PASS: TestCollectionVolumeStats (0.00s) === RUN TestCollectionVolumeStatsWithEcVolumes I0824 22:18:29.970850 node.go:471 weedfs adds child dc1 I0824 22:18:29.970926 node.go:471 weedfs:dc1 adds child rack1 I0824 22:18:29.970981 node.go:471 weedfs:dc1:rack1 adds child 127.0.0.1:34534 I0824 22:18:29.971047 node.go:471 weedfs:dc1:rack1:127.0.0.1:34534 adds child I0824 22:18:29.971090 node.go:471 weedfs:dc1:rack1 adds child 127.0.0.2:34534 I0824 22:18:29.971133 node.go:471 weedfs:dc1:rack1:127.0.0.2:34534 adds child --- PASS: TestCollectionVolumeStatsWithEcVolumes (0.00s) === RUN TestCollectionVolumeStatsClampsDeletedOverTotals I0824 22:18:29.971521 node.go:471 weedfs adds child dc1 I0824 22:18:29.971656 node.go:471 weedfs:dc1 adds child rack1 I0824 22:18:29.971757 node.go:471 weedfs:dc1:rack1 adds child 127.0.0.1:34534 I0824 22:18:29.971804 node.go:471 weedfs:dc1:rack1:127.0.0.1:34534 adds child --- PASS: TestCollectionVolumeStatsClampsDeletedOverTotals (0.00s) === RUN TestRemoveDataCenter data: map[dc1:map[rack1:map[server111:map[limit:3 volumes:[map[id:1 size:12312] map[id:2 size:12312] map[id:3 size:12312]]] server112:map[limit:10 volumes:[map[id:4 size:12312] map[id:5 size:12312] map[id:6 size:12312]]]] rack2:map[server121:map[limit:4 volumes:[map[id:4 size:12312] map[id:5 size:12312] map[id:6 size:12312]]] server122:map[limit:4 volumes:[]] server123:map[limit:5 volumes:[map[id:2 size:12312] map[id:3 size:12312] map[id:4 size:12312]]]]] dc2:map[] dc3:map[rack2:map[server321:map[limit:4 volumes:[map[id:1 size:12312] map[id:3 size:12312] map[id:5 size:12312]]]]]] I0824 22:18:29.975232 node.go:471 weedfs adds child dc1 I0824 22:18:29.975294 node.go:471 weedfs:dc1 adds child rack1 I0824 22:18:29.975331 node.go:471 weedfs:dc1:rack1 adds child server111 I0824 22:18:29.975381 node.go:471 weedfs:dc1:rack1:server111 adds child I0824 22:18:29.975451 node.go:471 weedfs:dc1:rack1 adds child server112 I0824 22:18:29.975497 node.go:471 weedfs:dc1:rack1:server112 adds child I0824 22:18:29.975546 node.go:471 weedfs:dc1 adds child rack2 I0824 22:18:29.975573 node.go:471 weedfs:dc1:rack2 adds child server121 I0824 22:18:29.975598 node.go:471 weedfs:dc1:rack2:server121 adds child I0824 22:18:29.975655 node.go:471 weedfs:dc1:rack2 adds child server122 I0824 22:18:29.975693 node.go:471 weedfs:dc1:rack2:server122 adds child I0824 22:18:29.975727 node.go:471 weedfs:dc1:rack2 adds child server123 I0824 22:18:29.975753 node.go:471 weedfs:dc1:rack2:server123 adds child I0824 22:18:29.975806 node.go:471 weedfs adds child dc2 I0824 22:18:29.975830 node.go:471 weedfs adds child dc3 I0824 22:18:29.975853 node.go:471 weedfs:dc3 adds child rack2 I0824 22:18:29.975878 node.go:471 weedfs:dc3:rack2 adds child server321 I0824 22:18:29.975903 node.go:471 weedfs:dc3:rack2:server321 adds child I0824 22:18:29.975956 node.go:492 weedfs removes dc2 I0824 22:18:29.976039 node.go:492 weedfs removes dc3 --- PASS: TestRemoveDataCenter (0.00s) === RUN TestHandlingVolumeServerHeartbeat I0824 22:18:29.976422 node.go:471 weedfs adds child dc1 I0824 22:18:29.976451 node.go:471 weedfs:dc1 adds child rack1 I0824 22:18:29.976488 node.go:471 weedfs:dc1:rack1 adds child 127.0.0.1:34534 I0824 22:18:29.976570 node.go:471 weedfs:dc1:rack1:127.0.0.1:34534 adds child I0824 22:18:29.976629 node.go:471 weedfs:dc1:rack1:127.0.0.1:34534 adds child ssd I0824 22:18:29.977133 data_node.go:114 Deleting volume id: 7 I0824 22:18:29.977223 data_node.go:114 Deleting volume id: 8 I0824 22:18:29.977250 data_node.go:114 Deleting volume id: 9 I0824 22:18:29.977274 data_node.go:114 Deleting volume id: 10 I0824 22:18:29.977293 data_node.go:114 Deleting volume id: 11 I0824 22:18:29.977316 data_node.go:114 Deleting volume id: 12 I0824 22:18:29.977392 data_node.go:114 Deleting volume id: 13 I0824 22:18:29.977415 data_node.go:114 Deleting volume id: 14 I0824 22:18:29.977486 topology.go:534 removing volume info: Id:7, Size:25432, ReplicaPlacement:000, Collection:, Version:3, Ttl:, FileCount:2343, DeleteCount:345, DeletedByteCount:34524, ReadOnly:false, ModifiedAtSecond:0 from 127.0.0.1:34534 I0824 22:18:29.977638 volume_layout.go:815 Volume 7 becomes unwritable I0824 22:18:29.977663 volume_layout.go:335 volume 7 does not have enough copies I0824 22:18:29.977687 topology.go:534 removing volume info: Id:8, Size:25432, ReplicaPlacement:000, Collection:, Version:3, Ttl:, FileCount:2343, DeleteCount:345, DeletedByteCount:34524, ReadOnly:false, ModifiedAtSecond:0 from 127.0.0.1:34534 I0824 22:18:29.977720 volume_layout.go:815 Volume 8 becomes unwritable I0824 22:18:29.977739 volume_layout.go:335 volume 8 does not have enough copies I0824 22:18:29.977758 topology.go:534 removing volume info: Id:9, Size:25432, ReplicaPlacement:000, Collection:, Version:3, Ttl:, FileCount:2343, DeleteCount:345, DeletedByteCount:34524, ReadOnly:false, ModifiedAtSecond:0 from 127.0.0.1:34534 I0824 22:18:29.977791 volume_layout.go:815 Volume 9 becomes unwritable I0824 22:18:29.977811 volume_layout.go:335 volume 9 does not have enough copies I0824 22:18:29.977939 topology.go:534 removing volume info: Id:10, Size:25432, ReplicaPlacement:000, Collection:, Version:3, Ttl:, FileCount:2343, DeleteCount:345, DeletedByteCount:34524, ReadOnly:false, ModifiedAtSecond:0 from 127.0.0.1:34534 I0824 22:18:29.977989 volume_layout.go:815 Volume 10 becomes unwritable I0824 22:18:29.978010 volume_layout.go:335 volume 10 does not have enough copies I0824 22:18:29.978028 topology.go:534 removing volume info: Id:11, Size:25432, ReplicaPlacement:000, Collection:, Version:3, Ttl:, FileCount:2343, DeleteCount:345, DeletedByteCount:34524, ReadOnly:false, ModifiedAtSecond:0 from 127.0.0.1:34534 I0824 22:18:29.978074 volume_layout.go:815 Volume 11 becomes unwritable I0824 22:18:29.978096 volume_layout.go:335 volume 11 does not have enough copies I0824 22:18:29.978114 topology.go:534 removing volume info: Id:12, Size:25432, ReplicaPlacement:000, Collection:, Version:3, Ttl:, FileCount:2343, DeleteCount:345, DeletedByteCount:34524, ReadOnly:false, ModifiedAtSecond:0 from 127.0.0.1:34534 I0824 22:18:29.978155 volume_layout.go:815 Volume 12 becomes unwritable I0824 22:18:29.978174 volume_layout.go:335 volume 12 does not have enough copies I0824 22:18:29.978192 topology.go:534 removing volume info: Id:13, Size:25432, ReplicaPlacement:000, Collection:, Version:3, Ttl:, FileCount:2343, DeleteCount:345, DeletedByteCount:34524, ReadOnly:false, ModifiedAtSecond:0 from 127.0.0.1:34534 I0824 22:18:29.978230 volume_layout.go:815 Volume 13 becomes unwritable I0824 22:18:29.978248 volume_layout.go:335 volume 13 does not have enough copies I0824 22:18:29.978267 topology.go:534 removing volume info: Id:14, Size:25432, ReplicaPlacement:000, Collection:, Version:3, Ttl:, FileCount:2343, DeleteCount:345, DeletedByteCount:34524, ReadOnly:false, ModifiedAtSecond:0 from 127.0.0.1:34534 I0824 22:18:29.978294 volume_layout.go:815 Volume 14 becomes unwritable I0824 22:18:29.978312 volume_layout.go:335 volume 14 does not have enough copies I0824 22:18:29.978429 topology.go:534 removing volume info: Id:3, Size:0, ReplicaPlacement:000, Collection:, Version:3, Ttl:, FileCount:0, DeleteCount:0, DeletedByteCount:0, ReadOnly:false, ModifiedAtSecond:0 from 127.0.0.1:34534 I0824 22:18:29.978571 volume_layout.go:815 Volume 3 becomes unwritable I0824 22:18:29.978593 volume_layout.go:335 volume 3 does not have enough copies after add volume id 6 after add volume id 1 after add volume id 2 after add volume id 3 after add volume id 4 after add volume id 5 after add writable volume id 1 after add writable volume id 2 after add writable volume id 4 after add writable volume id 5 after add writable volume id 6 after add writable volume id 3 I0824 22:18:29.978766 topology_event_handling.go:94 Removing Volume 3 from the dead volume server 127.0.0.1:34534 I0824 22:18:29.978800 volume_layout.go:866 Volume 3 has 0 replica, less than required 1 I0824 22:18:29.978821 volume_layout.go:815 Volume 3 becomes unwritable I0824 22:18:29.978841 topology_event_handling.go:94 Removing Volume 4 from the dead volume server 127.0.0.1:34534 I0824 22:18:29.978867 volume_layout.go:866 Volume 4 has 0 replica, less than required 1 I0824 22:18:29.978885 volume_layout.go:815 Volume 4 becomes unwritable I0824 22:18:29.978902 topology_event_handling.go:94 Removing Volume 5 from the dead volume server 127.0.0.1:34534 I0824 22:18:29.978943 volume_layout.go:866 Volume 5 has 0 replica, less than required 1 I0824 22:18:29.978962 volume_layout.go:815 Volume 5 becomes unwritable I0824 22:18:29.978979 topology_event_handling.go:94 Removing Volume 6 from the dead volume server 127.0.0.1:34534 I0824 22:18:29.978997 volume_layout.go:866 Volume 6 has 0 replica, less than required 1 I0824 22:18:29.979014 volume_layout.go:815 Volume 6 becomes unwritable I0824 22:18:29.979031 topology_event_handling.go:94 Removing Volume 1 from the dead volume server 127.0.0.1:34534 I0824 22:18:29.979062 volume_layout.go:866 Volume 1 has 0 replica, less than required 1 I0824 22:18:29.979094 volume_layout.go:815 Volume 1 becomes unwritable I0824 22:18:29.979111 topology_event_handling.go:94 Removing Volume 2 from the dead volume server 127.0.0.1:34534 I0824 22:18:29.979129 volume_layout.go:866 Volume 2 has 0 replica, less than required 1 I0824 22:18:29.979144 volume_layout.go:815 Volume 2 becomes unwritable I0824 22:18:29.979325 node.go:492 weedfs:dc1:rack1 removes 127.0.0.1:34534 --- PASS: TestHandlingVolumeServerHeartbeat (0.00s) === RUN TestDataNodeToDataNodeInfo_IncludeEmptyDiskFromUsage --- PASS: TestDataNodeToDataNodeInfo_IncludeEmptyDiskFromUsage (0.00s) === RUN TestAddRemoveVolume I0824 22:18:29.979950 node.go:471 weedfs adds child dc1 I0824 22:18:29.979985 node.go:471 weedfs:dc1 adds child rack1 I0824 22:18:29.980028 node.go:471 weedfs:dc1:rack1 adds child 127.0.0.1:34534 I0824 22:18:29.980104 node.go:471 weedfs:dc1:rack1:127.0.0.1:34534 adds child I0824 22:18:29.980138 node.go:471 weedfs:dc1:rack1:127.0.0.1:34534 adds child ssd I0824 22:18:29.980209 topology.go:534 removing volume info: Id:1, Size:100, ReplicaPlacement:000, Collection:xcollection, Version:3, Ttl:, FileCount:123, DeleteCount:23, DeletedByteCount:45, ReadOnly:false, ModifiedAtSecond:0 from 127.0.0.1:34534 I0824 22:18:29.980366 volume_layout.go:815 Volume 1 becomes unwritable I0824 22:18:29.980388 volume_layout.go:335 volume 1 does not have enough copies --- PASS: TestAddRemoveVolume (0.00s) === RUN TestVolumeReadOnlyStatusChange I0824 22:18:29.980602 node.go:471 weedfs adds child dc1 I0824 22:18:29.980630 node.go:471 weedfs:dc1 adds child rack1 I0824 22:18:29.980661 node.go:471 weedfs:dc1:rack1 adds child 127.0.0.1:34534 I0824 22:18:29.980767 node.go:471 weedfs:dc1:rack1:127.0.0.1:34534 adds child --- PASS: TestVolumeReadOnlyStatusChange (0.00s) === RUN TestVolumeReadOnlyAndRemoteStatusChange I0824 22:18:29.981208 node.go:471 weedfs adds child dc1 I0824 22:18:29.981250 node.go:471 weedfs:dc1 adds child rack1 I0824 22:18:29.981303 node.go:471 weedfs:dc1:rack1 adds child 127.0.0.1:34534 I0824 22:18:29.981335 node.go:471 weedfs:dc1:rack1:127.0.0.1:34534 adds child --- PASS: TestVolumeReadOnlyAndRemoteStatusChange (0.00s) === RUN TestListCollections I0824 22:18:29.981663 node.go:471 weedfs adds child dc1 I0824 22:18:29.981691 node.go:471 weedfs:dc1 adds child rack1 I0824 22:18:29.981722 node.go:471 weedfs:dc1:rack1 adds child 127.0.0.1:34534 === RUN TestListCollections/no_volume_types_selected === RUN TestListCollections/normal_volumes === RUN TestListCollections/EC_volumes === RUN TestListCollections/normal_+_EC_volumes --- PASS: TestListCollections (0.00s) --- PASS: TestListCollections/no_volume_types_selected (0.00s) --- PASS: TestListCollections/normal_volumes (0.00s) --- PASS: TestListCollections/EC_volumes (0.00s) --- PASS: TestListCollections/normal_+_EC_volumes (0.00s) === RUN TestDataNodeIdBasedIdentification I0824 22:18:29.982740 node.go:471 weedfs adds child dc1 I0824 22:18:29.982792 node.go:471 weedfs:dc1 adds child rack1 I0824 22:18:29.982831 node.go:471 weedfs:dc1:rack1 adds child node-1 I0824 22:18:29.982882 node.go:471 weedfs:dc1:rack1:node-1 adds child I0824 22:18:29.982914 rack.go:68 DataNode node-1 address changed from 10.0.0.1:8080 to 10.0.0.2:8080 I0824 22:18:29.982956 node.go:471 weedfs:dc1:rack1 adds child node-2 I0824 22:18:29.982995 node.go:471 weedfs:dc1:rack1:node-2 adds child I0824 22:18:29.983028 node.go:471 weedfs:dc1:rack1 adds child 10.0.0.4:8080 I0824 22:18:29.983121 node.go:471 weedfs:dc1:rack1:10.0.0.4:8080 adds child I0824 22:18:29.983191 rack.go:100 Volume server 10.0.0.4:8080 transitioning id from 10.0.0.4:8080 to node-4-explicit I0824 22:18:29.983384 node.go:471 weedfs:dc1:rack1 adds child node-99 I0824 22:18:29.983437 node.go:471 weedfs:dc1:rack1:node-99 adds child --- PASS: TestDataNodeIdBasedIdentification (0.00s) === RUN TestLookupDataNodeByAddress I0824 22:18:29.983675 node.go:471 weedfs adds child dc1 I0824 22:18:29.983716 node.go:471 weedfs:dc1 adds child rack1 I0824 22:18:29.983949 node.go:471 weedfs:dc1:rack1 adds child n1 I0824 22:18:29.984001 node.go:471 weedfs:dc1:rack1:n1 adds child I0824 22:18:29.984041 rack.go:68 DataNode n1 address changed from 10.1.2.3:8080 to 10.9.9.9:8080 I0824 22:18:29.984115 node.go:492 weedfs:dc1:rack1 removes n1 --- PASS: TestLookupDataNodeByAddress (0.00s) === RUN TestSyncDataNodeRegistrationReRegistersMissingVolume I0824 22:18:29.985650 node.go:471 weedfs adds child dc1 I0824 22:18:29.985723 node.go:471 weedfs:dc1 adds child rack1 I0824 22:18:29.985775 node.go:471 weedfs:dc1:rack1 adds child 127.0.0.1:34534 I0824 22:18:29.985858 node.go:471 weedfs:dc1:rack1:127.0.0.1:34534 adds child I0824 22:18:29.985992 volume_layout.go:866 Volume 18994 has 0 replica, less than required 1 I0824 22:18:29.986023 volume_layout.go:815 Volume 18994 becomes unwritable --- PASS: TestSyncDataNodeRegistrationReRegistersMissingVolume (0.00s) === RUN TestSetVolumeAvailableRepairsMissingVolume I0824 22:18:29.986364 node.go:471 weedfs adds child dc1 I0824 22:18:29.986465 node.go:471 weedfs:dc1 adds child rack1 I0824 22:18:29.986499 node.go:471 weedfs:dc1:rack1 adds child 127.0.0.1:34534 I0824 22:18:29.986531 node.go:471 weedfs:dc1:rack1:127.0.0.1:34534 adds child I0824 22:18:29.986686 volume_layout.go:866 Volume 2640 has 0 replica, less than required 1 I0824 22:18:29.986711 volume_layout.go:815 Volume 2640 becomes unwritable --- PASS: TestSetVolumeAvailableRepairsMissingVolume (0.00s) === RUN TestVolumeDigestIsStableAcrossRepeatedHeartbeats I0824 22:18:29.988134 node.go:471 digest adds child dc1 I0824 22:18:29.988209 node.go:471 digest:dc1 adds child rack1 I0824 22:18:29.988270 node.go:471 digest:dc1:rack1 adds child 127.0.0.1:8080 I0824 22:18:29.988339 node.go:471 digest:dc1:rack1:127.0.0.1:8080 adds child --- PASS: TestVolumeDigestIsStableAcrossRepeatedHeartbeats (0.00s) === RUN TestVolumeDigestIsIndependentOfReportOrder I0824 22:18:29.988776 node.go:471 digest adds child dc1 I0824 22:18:29.988861 node.go:471 digest:dc1 adds child rack1 I0824 22:18:29.988907 node.go:471 digest:dc1:rack1 adds child 127.0.0.1:8080 I0824 22:18:29.988954 node.go:471 digest:dc1:rack1:127.0.0.1:8080 adds child I0824 22:18:29.989131 node.go:471 digest adds child dc1 I0824 22:18:29.989277 node.go:471 digest:dc1 adds child rack1 I0824 22:18:29.989319 node.go:471 digest:dc1:rack1 adds child 127.0.0.1:8080 I0824 22:18:29.989351 node.go:471 digest:dc1:rack1:127.0.0.1:8080 adds child --- PASS: TestVolumeDigestIsIndependentOfReportOrder (0.00s) === RUN TestVolumeDigestTracksEveryReportedField === RUN TestVolumeDigestTracksEveryReportedField/id === RUN TestVolumeDigestTracksEveryReportedField/size === RUN TestVolumeDigestTracksEveryReportedField/collection === RUN TestVolumeDigestTracksEveryReportedField/file_count === RUN TestVolumeDigestTracksEveryReportedField/delete_count === RUN TestVolumeDigestTracksEveryReportedField/deleted_byte_count === RUN TestVolumeDigestTracksEveryReportedField/read_only === RUN TestVolumeDigestTracksEveryReportedField/replica_placement === RUN TestVolumeDigestTracksEveryReportedField/version === RUN TestVolumeDigestTracksEveryReportedField/ttl === RUN TestVolumeDigestTracksEveryReportedField/compact_revision === RUN TestVolumeDigestTracksEveryReportedField/modified_at_second === RUN TestVolumeDigestTracksEveryReportedField/remote_storage_name === RUN TestVolumeDigestTracksEveryReportedField/remote_storage_key === RUN TestVolumeDigestTracksEveryReportedField/disk_type === RUN TestVolumeDigestTracksEveryReportedField/disk_id --- PASS: TestVolumeDigestTracksEveryReportedField (0.01s) --- PASS: TestVolumeDigestTracksEveryReportedField/id (0.00s) --- PASS: TestVolumeDigestTracksEveryReportedField/size (0.00s) --- PASS: TestVolumeDigestTracksEveryReportedField/collection (0.00s) --- PASS: TestVolumeDigestTracksEveryReportedField/file_count (0.00s) --- PASS: TestVolumeDigestTracksEveryReportedField/delete_count (0.00s) --- PASS: TestVolumeDigestTracksEveryReportedField/deleted_byte_count (0.00s) --- PASS: TestVolumeDigestTracksEveryReportedField/read_only (0.00s) --- PASS: TestVolumeDigestTracksEveryReportedField/replica_placement (0.00s) --- PASS: TestVolumeDigestTracksEveryReportedField/version (0.00s) --- PASS: TestVolumeDigestTracksEveryReportedField/ttl (0.00s) --- PASS: TestVolumeDigestTracksEveryReportedField/compact_revision (0.00s) --- PASS: TestVolumeDigestTracksEveryReportedField/modified_at_second (0.00s) --- PASS: TestVolumeDigestTracksEveryReportedField/remote_storage_name (0.00s) --- PASS: TestVolumeDigestTracksEveryReportedField/remote_storage_key (0.00s) --- PASS: TestVolumeDigestTracksEveryReportedField/disk_type (0.00s) --- PASS: TestVolumeDigestTracksEveryReportedField/disk_id (0.00s) === RUN TestVolumeDigestFollowsVolumeChanges I0824 22:18:30.003466 node.go:471 digest adds child dc1 I0824 22:18:30.003598 node.go:471 digest:dc1 adds child rack1 I0824 22:18:30.003682 node.go:471 digest:dc1:rack1 adds child 127.0.0.1:8080 I0824 22:18:30.003982 node.go:471 digest:dc1:rack1:127.0.0.1:8080 adds child I0824 22:18:30.005011 data_node.go:114 Deleting volume id: 2 I0824 22:18:30.005225 topology.go:534 removing volume info: Id:2, Size:1048576, ReplicaPlacement:000, Collection:c, Version:3, Ttl:, FileCount:10, DeleteCount:1, DeletedByteCount:128, ReadOnly:false, ModifiedAtSecond:1700000000 from 127.0.0.1:8080 I0824 22:18:30.008442 volume_layout.go:815 Volume 2 becomes unwritable I0824 22:18:30.008537 volume_layout.go:335 volume 2 does not have enough copies --- PASS: TestVolumeDigestFollowsVolumeChanges (0.02s) === RUN TestVolumeDigestEmptiesWithTheNode I0824 22:18:30.019683 node.go:471 digest adds child dc1 I0824 22:18:30.019758 node.go:471 digest:dc1 adds child rack1 I0824 22:18:30.019841 node.go:471 digest:dc1:rack1 adds child 127.0.0.1:8080 I0824 22:18:30.019912 node.go:471 digest:dc1:rack1:127.0.0.1:8080 adds child I0824 22:18:30.020124 data_node.go:114 Deleting volume id: 1 I0824 22:18:30.020157 data_node.go:114 Deleting volume id: 2 I0824 22:18:30.020188 topology.go:534 removing volume info: Id:1, Size:1048576, ReplicaPlacement:000, Collection:c, Version:3, Ttl:, FileCount:10, DeleteCount:1, DeletedByteCount:128, ReadOnly:false, ModifiedAtSecond:1700000000 from 127.0.0.1:8080 I0824 22:18:30.020251 volume_layout.go:815 Volume 1 becomes unwritable I0824 22:18:30.020299 volume_layout.go:335 volume 1 does not have enough copies I0824 22:18:30.020377 topology.go:534 removing volume info: Id:2, Size:1048576, ReplicaPlacement:000, Collection:c, Version:3, Ttl:, FileCount:10, DeleteCount:1, DeletedByteCount:128, ReadOnly:false, ModifiedAtSecond:1700000000 from 127.0.0.1:8080 I0824 22:18:30.020471 volume_layout.go:815 Volume 2 becomes unwritable I0824 22:18:30.020497 volume_layout.go:335 volume 2 does not have enough copies --- PASS: TestVolumeDigestEmptiesWithTheNode (0.00s) === RUN TestVolumeDigestFollowsDeltaRegistration I0824 22:18:30.020714 node.go:471 digest adds child dc1 I0824 22:18:30.020750 node.go:471 digest:dc1 adds child rack1 I0824 22:18:30.020789 node.go:471 digest:dc1:rack1 adds child 127.0.0.1:8080 I0824 22:18:30.020831 node.go:471 digest:dc1:rack1:127.0.0.1:8080 adds child I0824 22:18:30.021002 topology.go:534 removing volume info: Id:2, Size:0, ReplicaPlacement:000, Collection:, Version:0, Ttl:, FileCount:0, DeleteCount:0, DeletedByteCount:0, ReadOnly:false, ModifiedAtSecond:0 from 127.0.0.1:8080 --- PASS: TestVolumeDigestFollowsDeltaRegistration (0.00s) === RUN TestVolumeDigestMatchesAfterApplyingOnlyChangedVolumes I0824 22:18:30.021435 node.go:471 digest adds child dc1 I0824 22:18:30.021493 node.go:471 digest:dc1 adds child rack1 I0824 22:18:30.021611 node.go:471 digest:dc1:rack1 adds child 127.0.0.1:8080 I0824 22:18:30.021733 node.go:471 digest:dc1:rack1:127.0.0.1:8080 adds child I0824 22:18:30.023017 node.go:471 digest adds child dc1 I0824 22:18:30.023114 node.go:471 digest:dc1 adds child rack1 I0824 22:18:30.023205 node.go:471 digest:dc1:rack1 adds child 127.0.0.1:8080 I0824 22:18:30.023256 node.go:471 digest:dc1:rack1:127.0.0.1:8080 adds child --- PASS: TestVolumeDigestMatchesAfterApplyingOnlyChangedVolumes (0.01s) === RUN TestVolumeDigestCatchesASilentlyLostVolume I0824 22:18:30.028233 node.go:471 digest adds child dc1 I0824 22:18:30.028330 node.go:471 digest:dc1 adds child rack1 I0824 22:18:30.031798 node.go:471 digest:dc1:rack1 adds child 127.0.0.1:8080 I0824 22:18:30.032205 node.go:471 digest:dc1:rack1:127.0.0.1:8080 adds child I0824 22:18:30.032722 node.go:471 digest adds child dc1 I0824 22:18:30.032935 node.go:471 digest:dc1 adds child rack1 I0824 22:18:30.033053 node.go:471 digest:dc1:rack1 adds child 127.0.0.1:8080 I0824 22:18:30.033139 node.go:471 digest:dc1:rack1:127.0.0.1:8080 adds child --- PASS: TestVolumeDigestCatchesASilentlyLostVolume (0.01s) === RUN TestVolumeIndexDigestSeesLookupDivergence I0824 22:18:30.034552 node.go:471 digest adds child dc1 I0824 22:18:30.034747 node.go:471 digest:dc1 adds child rack1 I0824 22:18:30.034832 node.go:471 digest:dc1:rack1 adds child 127.0.0.1:8080 I0824 22:18:30.034922 node.go:471 digest:dc1:rack1:127.0.0.1:8080 adds child I0824 22:18:30.035137 volume_layout.go:866 Volume 1 has 0 replica, less than required 1 I0824 22:18:30.035173 volume_layout.go:815 Volume 1 becomes unwritable --- PASS: TestVolumeIndexDigestSeesLookupDivergence (0.00s) === RUN TestVolumeIndexDigestFollowsNodeLifecycle I0824 22:18:30.035389 node.go:471 digest adds child dc1 I0824 22:18:30.035419 node.go:471 digest:dc1 adds child rack1 I0824 22:18:30.035472 node.go:471 digest:dc1:rack1 adds child 127.0.0.1:8080 I0824 22:18:30.035540 node.go:471 digest:dc1:rack1:127.0.0.1:8080 adds child I0824 22:18:30.035670 data_node.go:114 Deleting volume id: 3 I0824 22:18:30.035743 topology.go:534 removing volume info: Id:3, Size:1048576, ReplicaPlacement:000, Collection:c, Version:3, Ttl:, FileCount:10, DeleteCount:1, DeletedByteCount:128, ReadOnly:false, ModifiedAtSecond:1700000000 from 127.0.0.1:8080 I0824 22:18:30.035830 volume_layout.go:815 Volume 3 becomes unwritable I0824 22:18:30.035853 volume_layout.go:335 volume 3 does not have enough copies I0824 22:18:30.037530 topology.go:534 removing volume info: Id:9, Size:0, ReplicaPlacement:000, Collection:, Version:0, Ttl:, FileCount:0, DeleteCount:0, DeletedByteCount:0, ReadOnly:false, ModifiedAtSecond:0 from 127.0.0.1:8080 I0824 22:18:30.037675 volume_layout.go:815 Volume 9 becomes unwritable I0824 22:18:30.037791 volume_layout.go:335 volume 9 does not have enough copies I0824 22:18:30.037839 topology_event_handling.go:94 Removing Volume 1 from the dead volume server 127.0.0.1:8080 I0824 22:18:30.037882 volume_layout.go:866 Volume 1 has 0 replica, less than required 1 I0824 22:18:30.037905 volume_layout.go:815 Volume 1 becomes unwritable I0824 22:18:30.037924 topology_event_handling.go:94 Removing Volume 2 from the dead volume server 127.0.0.1:8080 I0824 22:18:30.037947 volume_layout.go:866 Volume 2 has 0 replica, less than required 1 I0824 22:18:30.037967 volume_layout.go:815 Volume 2 becomes unwritable I0824 22:18:30.038056 node.go:492 digest:dc1:rack1 removes 127.0.0.1:8080 --- PASS: TestVolumeIndexDigestFollowsNodeLifecycle (0.00s) === RUN TestVolumeDigestRefusesDuplicateVolumeIds I0824 22:18:30.038561 node.go:471 digest adds child dc1 I0824 22:18:30.038616 node.go:471 digest:dc1 adds child rack1 I0824 22:18:30.038672 node.go:471 digest:dc1:rack1 adds child 127.0.0.1:8080 I0824 22:18:30.038796 node.go:471 digest:dc1:rack1:127.0.0.1:8080 adds child --- PASS: TestVolumeDigestRefusesDuplicateVolumeIds (0.00s) === RUN TestVolumeIndexDigestFollowsDisplacedLookupEntry I0824 22:18:30.039126 node.go:471 digest adds child dc1 I0824 22:18:30.039323 node.go:471 digest:dc1 adds child rack1 I0824 22:18:30.039366 node.go:471 digest:dc1:rack1 adds child n1 I0824 22:18:30.039405 node.go:471 digest:dc1:rack1:n1 adds child W0824 22:18:30.039546 rack.go:96 Volume server with id n2 has ip:port 10.1.2.3:8080 which is used by node n1 I0824 22:18:30.041330 node.go:471 digest:dc1:rack1 adds child n2 I0824 22:18:30.041449 node.go:471 digest:dc1:rack1:n2 adds child --- PASS: TestVolumeIndexDigestFollowsDisplacedLookupEntry (0.00s) === RUN TestVolumeIndexDigestFollowsRemovedLookupEntry I0824 22:18:30.041840 node.go:471 digest adds child dc1 I0824 22:18:30.041915 node.go:471 digest:dc1 adds child rack1 I0824 22:18:30.041953 node.go:471 digest:dc1:rack1 adds child n1 I0824 22:18:30.041999 node.go:471 digest:dc1:rack1:n1 adds child W0824 22:18:30.042029 rack.go:96 Volume server with id n2 has ip:port 10.1.2.3:8080 which is used by node n1 I0824 22:18:30.042074 node.go:471 digest:dc1:rack1 adds child n2 I0824 22:18:30.042109 node.go:471 digest:dc1:rack1:n2 adds child I0824 22:18:30.042196 volume_layout.go:866 Volume 1 has 0 replica, less than required 1 I0824 22:18:30.042223 volume_layout.go:815 Volume 1 becomes unwritable --- PASS: TestVolumeIndexDigestFollowsRemovedLookupEntry (0.00s) === RUN TestMasterDigestMatchesWhatAVolumeServerReports I0824 22:18:30.054082 node.go:471 digest adds child dc1 I0824 22:18:30.054165 node.go:471 digest:dc1 adds child rack1 I0824 22:18:30.054235 node.go:471 digest:dc1:rack1 adds child 127.0.0.1:8080 I0824 22:18:30.054284 node.go:471 digest:dc1:rack1:127.0.0.1:8080 adds child --- PASS: TestMasterDigestMatchesWhatAVolumeServerReports (0.01s) === RUN TestVolumeMovedBetweenDisks === RUN TestVolumeMovedBetweenDisks/SameDiskType I0824 22:18:30.055515 node.go:471 move adds child dc1 I0824 22:18:30.055553 node.go:471 move:dc1 adds child rack1 I0824 22:18:30.055595 node.go:471 move:dc1:rack1 adds child 127.0.0.1:8080 I0824 22:18:30.055666 node.go:471 move:dc1:rack1:127.0.0.1:8080 adds child I0824 22:18:30.055733 node.go:471 move:dc1:rack1:127.0.0.1:8080 adds child ssd === RUN TestVolumeMovedBetweenDisks/DifferentDiskType I0824 22:18:30.056032 node.go:471 move adds child dc1 I0824 22:18:30.056085 node.go:471 move:dc1 adds child rack1 I0824 22:18:30.056127 node.go:471 move:dc1:rack1 adds child 127.0.0.1:8080 I0824 22:18:30.056263 node.go:471 move:dc1:rack1:127.0.0.1:8080 adds child I0824 22:18:30.056320 node.go:471 move:dc1:rack1:127.0.0.1:8080 adds child ssd I0824 22:18:30.056431 data_node.go:114 Deleting volume id: 1 I0824 22:18:30.056615 topology.go:534 removing volume info: Id:1, Size:1024, ReplicaPlacement:000, Collection:c, Version:3, Ttl:, FileCount:0, DeleteCount:0, DeletedByteCount:0, ReadOnly:false, ModifiedAtSecond:0 from 127.0.0.1:8080 I0824 22:18:30.056722 volume_layout.go:815 Volume 1 becomes unwritable I0824 22:18:30.056745 volume_layout.go:335 volume 1 does not have enough copies === RUN TestVolumeMovedBetweenDisks/BackAgain I0824 22:18:30.056984 node.go:471 move adds child dc1 I0824 22:18:30.057045 node.go:471 move:dc1 adds child rack1 I0824 22:18:30.057080 node.go:471 move:dc1:rack1 adds child 127.0.0.1:8080 I0824 22:18:30.057132 node.go:471 move:dc1:rack1:127.0.0.1:8080 adds child I0824 22:18:30.057178 node.go:471 move:dc1:rack1:127.0.0.1:8080 adds child ssd I0824 22:18:30.057261 data_node.go:114 Deleting volume id: 1 I0824 22:18:30.057359 topology.go:534 removing volume info: Id:1, Size:1024, ReplicaPlacement:000, Collection:c, Version:3, Ttl:, FileCount:0, DeleteCount:0, DeletedByteCount:0, ReadOnly:false, ModifiedAtSecond:0 from 127.0.0.1:8080 I0824 22:18:30.057435 volume_layout.go:815 Volume 1 becomes unwritable I0824 22:18:30.057456 volume_layout.go:335 volume 1 does not have enough copies --- PASS: TestVolumeMovedBetweenDisks (0.00s) --- PASS: TestVolumeMovedBetweenDisks/SameDiskType (0.00s) --- PASS: TestVolumeMovedBetweenDisks/DifferentDiskType (0.00s) --- PASS: TestVolumeMovedBetweenDisks/BackAgain (0.00s) === RUN TestVolumeReportedOnTwoDiskTypesIsKept I0824 22:18:30.057779 node.go:471 move adds child dc1 I0824 22:18:30.057891 node.go:471 move:dc1 adds child rack1 I0824 22:18:30.057998 node.go:471 move:dc1:rack1 adds child 127.0.0.1:8080 I0824 22:18:30.058078 node.go:471 move:dc1:rack1:127.0.0.1:8080 adds child ssd I0824 22:18:30.058117 node.go:471 move:dc1:rack1:127.0.0.1:8080 adds child I0824 22:18:30.058226 data_node.go:114 Deleting volume id: 1 I0824 22:18:30.058259 topology.go:534 removing volume info: Id:1, Size:1024, ReplicaPlacement:000, Collection:c, Version:3, Ttl:, FileCount:0, DeleteCount:0, DeletedByteCount:0, ReadOnly:false, ModifiedAtSecond:0 from 127.0.0.1:8080 I0824 22:18:30.058294 volume_layout.go:815 Volume 1 becomes unwritable I0824 22:18:30.058314 volume_layout.go:335 volume 1 does not have enough copies --- PASS: TestVolumeReportedOnTwoDiskTypesIsKept (0.00s) === RUN TestVolumeReportedTwiceOnOneDiskTypeIsFlagged I0824 22:18:30.058625 node.go:471 move adds child dc1 I0824 22:18:30.058686 node.go:471 move:dc1 adds child rack1 I0824 22:18:30.058721 node.go:471 move:dc1:rack1 adds child 127.0.0.1:8080 I0824 22:18:30.058797 node.go:471 move:dc1:rack1:127.0.0.1:8080 adds child I0824 22:18:30.058830 node.go:471 move:dc1:rack1:127.0.0.1:8080 adds child ssd --- PASS: TestVolumeReportedTwiceOnOneDiskTypeIsFlagged (0.00s) === RUN TestVolumeDigestSurvivesADiskMove I0824 22:18:30.059230 node.go:471 move adds child dc1 I0824 22:18:30.059275 node.go:471 move:dc1 adds child rack1 I0824 22:18:30.059323 node.go:471 move:dc1:rack1 adds child 127.0.0.1:8080 I0824 22:18:30.059371 node.go:471 move:dc1:rack1:127.0.0.1:8080 adds child I0824 22:18:30.059431 node.go:471 move:dc1:rack1:127.0.0.1:8080 adds child ssd I0824 22:18:30.059534 data_node.go:114 Deleting volume id: 1 I0824 22:18:30.059654 topology.go:534 removing volume info: Id:1, Size:1024, ReplicaPlacement:000, Collection:c, Version:3, Ttl:, FileCount:0, DeleteCount:0, DeletedByteCount:0, ReadOnly:false, ModifiedAtSecond:0 from 127.0.0.1:8080 I0824 22:18:30.059722 volume_layout.go:815 Volume 1 becomes unwritable I0824 22:18:30.059744 volume_layout.go:335 volume 1 does not have enough copies I0824 22:18:30.059801 node.go:471 move adds child dc1 I0824 22:18:30.059865 node.go:471 move:dc1 adds child rack1 I0824 22:18:30.059909 node.go:471 move:dc1:rack1 adds child 127.0.0.1:8080 I0824 22:18:30.059949 node.go:471 move:dc1:rack1:127.0.0.1:8080 adds child I0824 22:18:30.059986 node.go:471 move:dc1:rack1:127.0.0.1:8080 adds child ssd --- PASS: TestVolumeDigestSurvivesADiskMove (0.00s) === RUN TestVolumeFilterSelects I0824 22:18:30.060523 node.go:471 filter adds child dc1 I0824 22:18:30.060593 node.go:471 filter:dc1 adds child rack1 I0824 22:18:30.060629 node.go:471 filter:dc1:rack1 adds child 10.0.0.1:8080 I0824 22:18:30.060698 node.go:471 filter:dc1:rack1:10.0.0.1:8080 adds child === RUN TestVolumeFilterSelects/no_filter_takes_everything === RUN TestVolumeFilterSelects/one_collection === RUN TestVolumeFilterSelects/the_default_collection === RUN TestVolumeFilterSelects/a_collection_nothing_is_in === RUN TestVolumeFilterSelects/one_volume === RUN TestVolumeFilterSelects/one_ec_volume === RUN TestVolumeFilterSelects/both,_agreeing === RUN TestVolumeFilterSelects/both,_disagreeing --- PASS: TestVolumeFilterSelects (0.00s) --- PASS: TestVolumeFilterSelects/no_filter_takes_everything (0.00s) --- PASS: TestVolumeFilterSelects/one_collection (0.00s) --- PASS: TestVolumeFilterSelects/the_default_collection (0.00s) --- PASS: TestVolumeFilterSelects/a_collection_nothing_is_in (0.00s) --- PASS: TestVolumeFilterSelects/one_volume (0.00s) --- PASS: TestVolumeFilterSelects/one_ec_volume (0.00s) --- PASS: TestVolumeFilterSelects/both,_agreeing (0.00s) --- PASS: TestVolumeFilterSelects/both,_disagreeing (0.00s) === RUN TestVolumeFilterKeepsTheTopology I0824 22:18:30.065465 node.go:471 filter adds child dc1 I0824 22:18:30.065584 node.go:471 filter:dc1 adds child rack1 I0824 22:18:30.065644 node.go:471 filter:dc1:rack1 adds child 10.0.0.1:8080 I0824 22:18:30.065689 node.go:471 filter:dc1:rack1:10.0.0.1:8080 adds child --- PASS: TestVolumeFilterKeepsTheTopology (0.00s) === RUN TestVolumeFilterKeepsTheDiskIdWhenNothingMatches I0824 22:18:30.067291 node.go:471 filter adds child dc1 I0824 22:18:30.067423 node.go:471 filter:dc1 adds child rack1 I0824 22:18:30.067467 node.go:471 filter:dc1:rack1 adds child 10.0.0.1:8080 I0824 22:18:30.067505 node.go:471 filter:dc1:rack1:10.0.0.1:8080 adds child --- PASS: TestVolumeFilterKeepsTheDiskIdWhenNothingMatches (0.00s) === RUN TestNewVolumeFilterReadsTheRequest === RUN TestNewVolumeFilterReadsTheRequest/an_empty_request === RUN TestNewVolumeFilterReadsTheRequest/an_empty_collection === RUN TestNewVolumeFilterReadsTheRequest/a_named_collection === RUN TestNewVolumeFilterReadsTheRequest/the_default_collection === RUN TestNewVolumeFilterReadsTheRequest/both --- PASS: TestNewVolumeFilterReadsTheRequest (0.00s) --- PASS: TestNewVolumeFilterReadsTheRequest/an_empty_request (0.00s) --- PASS: TestNewVolumeFilterReadsTheRequest/an_empty_collection (0.00s) --- PASS: TestNewVolumeFilterReadsTheRequest/a_named_collection (0.00s) --- PASS: TestNewVolumeFilterReadsTheRequest/the_default_collection (0.00s) --- PASS: TestNewVolumeFilterReadsTheRequest/both (0.00s) === RUN TestVolumeFilterListsWhileVolumesChange I0824 22:18:30.070800 node.go:471 filter adds child dc1 I0824 22:18:30.070884 node.go:471 filter:dc1 adds child rack1 I0824 22:18:30.070924 node.go:471 filter:dc1:rack1 adds child 10.0.0.1:8080 I0824 22:18:30.070969 node.go:471 filter:dc1:rack1:10.0.0.1:8080 adds child I0824 22:18:30.073742 data_node.go:114 Deleting volume id: 2 I0824 22:18:30.073856 data_node.go:114 Deleting volume id: 3 I0824 22:18:30.073886 data_node.go:114 Deleting volume id: 1 I0824 22:18:30.075455 topology.go:534 removing volume info: Id:2, Size:2000, ReplicaPlacement:000, Collection:c, Version:3, Ttl:, FileCount:0, DeleteCount:0, DeletedByteCount:0, ReadOnly:false, ModifiedAtSecond:0 from 10.0.0.1:8080 I0824 22:18:30.075682 volume_layout.go:815 Volume 2 becomes unwritable I0824 22:18:30.075715 volume_layout.go:335 volume 2 does not have enough copies I0824 22:18:30.075746 topology.go:534 removing volume info: Id:3, Size:3000, ReplicaPlacement:000, Collection:other, Version:3, Ttl:, FileCount:0, DeleteCount:0, DeletedByteCount:0, ReadOnly:false, ModifiedAtSecond:0 from 10.0.0.1:8080 I0824 22:18:30.075813 volume_layout.go:815 Volume 3 becomes unwritable I0824 22:18:30.075890 volume_layout.go:335 volume 3 does not have enough copies I0824 22:18:30.075964 topology.go:534 removing volume info: Id:1, Size:1000, ReplicaPlacement:000, Collection:, Version:3, Ttl:, FileCount:0, DeleteCount:0, DeletedByteCount:0, ReadOnly:false, ModifiedAtSecond:0 from 10.0.0.1:8080 I0824 22:18:30.076036 volume_layout.go:815 Volume 1 becomes unwritable I0824 22:18:30.076065 volume_layout.go:335 volume 1 does not have enough copies I0824 22:18:30.076269 data_node.go:114 Deleting volume id: 100 I0824 22:18:30.076380 topology.go:534 removing volume info: Id:100, Size:1, ReplicaPlacement:000, Collection:c, Version:3, Ttl:, FileCount:0, DeleteCount:0, DeletedByteCount:0, ReadOnly:false, ModifiedAtSecond:0 from 10.0.0.1:8080 I0824 22:18:30.076417 volume_layout.go:815 Volume 100 becomes unwritable I0824 22:18:30.076437 volume_layout.go:335 volume 100 does not have enough copies I0824 22:18:30.079741 data_node.go:114 Deleting volume id: 101 I0824 22:18:30.079858 topology.go:534 removing volume info: Id:101, Size:1, ReplicaPlacement:000, Collection:c, Version:3, Ttl:, FileCount:0, DeleteCount:0, DeletedByteCount:0, ReadOnly:false, ModifiedAtSecond:0 from 10.0.0.1:8080 I0824 22:18:30.079904 volume_layout.go:815 Volume 101 becomes unwritable I0824 22:18:30.079927 volume_layout.go:335 volume 101 does not have enough copies I0824 22:18:30.080009 data_node.go:114 Deleting volume id: 102 I0824 22:18:30.080086 topology.go:534 removing volume info: Id:102, Size:1, ReplicaPlacement:000, Collection:c, Version:3, Ttl:, FileCount:0, DeleteCount:0, DeletedByteCount:0, ReadOnly:false, ModifiedAtSecond:0 from 10.0.0.1:8080 I0824 22:18:30.080164 volume_layout.go:815 Volume 102 becomes unwritable I0824 22:18:30.082428 volume_layout.go:335 volume 102 does not have enough copies I0824 22:18:30.082571 data_node.go:114 Deleting volume id: 103 I0824 22:18:30.082642 topology.go:534 removing volume info: Id:103, Size:1, ReplicaPlacement:000, Collection:c, Version:3, Ttl:, FileCount:0, DeleteCount:0, DeletedByteCount:0, ReadOnly:false, ModifiedAtSecond:0 from 10.0.0.1:8080 I0824 22:18:30.082712 volume_layout.go:815 Volume 103 becomes unwritable I0824 22:18:30.082739 volume_layout.go:335 volume 103 does not have enough copies I0824 22:18:30.082805 data_node.go:114 Deleting volume id: 104 I0824 22:18:30.082848 topology.go:534 removing volume info: Id:104, Size:1, ReplicaPlacement:000, Collection:c, Version:3, Ttl:, FileCount:0, DeleteCount:0, DeletedByteCount:0, ReadOnly:false, ModifiedAtSecond:0 from 10.0.0.1:8080 I0824 22:18:30.082881 volume_layout.go:815 Volume 104 becomes unwritable I0824 22:18:30.082900 volume_layout.go:335 volume 104 does not have enough copies I0824 22:18:30.082980 data_node.go:114 Deleting volume id: 105 I0824 22:18:30.083235 topology.go:534 removing volume info: Id:105, Size:1, ReplicaPlacement:000, Collection:c, Version:3, Ttl:, FileCount:0, DeleteCount:0, DeletedByteCount:0, ReadOnly:false, ModifiedAtSecond:0 from 10.0.0.1:8080 I0824 22:18:30.083309 volume_layout.go:815 Volume 105 becomes unwritable I0824 22:18:30.083336 volume_layout.go:335 volume 105 does not have enough copies I0824 22:18:30.083447 data_node.go:114 Deleting volume id: 106 I0824 22:18:30.083556 topology.go:534 removing volume info: Id:106, Size:1, ReplicaPlacement:000, Collection:c, Version:3, Ttl:, FileCount:0, DeleteCount:0, DeletedByteCount:0, ReadOnly:false, ModifiedAtSecond:0 from 10.0.0.1:8080 I0824 22:18:30.083653 volume_layout.go:815 Volume 106 becomes unwritable I0824 22:18:30.083678 volume_layout.go:335 volume 106 does not have enough copies I0824 22:18:30.083775 data_node.go:114 Deleting volume id: 107 I0824 22:18:30.085899 topology.go:534 removing volume info: Id:107, Size:1, ReplicaPlacement:000, Collection:c, Version:3, Ttl:, FileCount:0, DeleteCount:0, DeletedByteCount:0, ReadOnly:false, ModifiedAtSecond:0 from 10.0.0.1:8080 I0824 22:18:30.085984 volume_layout.go:815 Volume 107 becomes unwritable I0824 22:18:30.086011 volume_layout.go:335 volume 107 does not have enough copies I0824 22:18:30.086094 data_node.go:114 Deleting volume id: 108 I0824 22:18:30.086171 topology.go:534 removing volume info: Id:108, Size:1, ReplicaPlacement:000, Collection:c, Version:3, Ttl:, FileCount:0, DeleteCount:0, DeletedByteCount:0, ReadOnly:false, ModifiedAtSecond:0 from 10.0.0.1:8080 I0824 22:18:30.086229 volume_layout.go:815 Volume 108 becomes unwritable I0824 22:18:30.086284 volume_layout.go:335 volume 108 does not have enough copies I0824 22:18:30.086425 data_node.go:114 Deleting volume id: 109 I0824 22:18:30.086500 topology.go:534 removing volume info: Id:109, Size:1, ReplicaPlacement:000, Collection:c, Version:3, Ttl:, FileCount:0, DeleteCount:0, DeletedByteCount:0, ReadOnly:false, ModifiedAtSecond:0 from 10.0.0.1:8080 I0824 22:18:30.086552 volume_layout.go:815 Volume 109 becomes unwritable I0824 22:18:30.086574 volume_layout.go:335 volume 109 does not have enough copies I0824 22:18:30.086629 data_node.go:114 Deleting volume id: 110 I0824 22:18:30.086686 topology.go:534 removing volume info: Id:110, Size:1, ReplicaPlacement:000, Collection:c, Version:3, Ttl:, FileCount:0, DeleteCount:0, DeletedByteCount:0, ReadOnly:false, ModifiedAtSecond:0 from 10.0.0.1:8080 I0824 22:18:30.086719 volume_layout.go:815 Volume 110 becomes unwritable I0824 22:18:30.086737 volume_layout.go:335 volume 110 does not have enough copies I0824 22:18:30.086816 data_node.go:114 Deleting volume id: 111 I0824 22:18:30.086887 topology.go:534 removing volume info: Id:111, Size:1, ReplicaPlacement:000, Collection:c, Version:3, Ttl:, FileCount:0, DeleteCount:0, DeletedByteCount:0, ReadOnly:false, ModifiedAtSecond:0 from 10.0.0.1:8080 I0824 22:18:30.086950 volume_layout.go:815 Volume 111 becomes unwritable I0824 22:18:30.086970 volume_layout.go:335 volume 111 does not have enough copies I0824 22:18:30.087123 data_node.go:114 Deleting volume id: 112 I0824 22:18:30.087348 topology.go:534 removing volume info: Id:112, Size:1, ReplicaPlacement:000, Collection:c, Version:3, Ttl:, FileCount:0, DeleteCount:0, DeletedByteCount:0, ReadOnly:false, ModifiedAtSecond:0 from 10.0.0.1:8080 I0824 22:18:30.087430 volume_layout.go:815 Volume 112 becomes unwritable I0824 22:18:30.087467 volume_layout.go:335 volume 112 does not have enough copies I0824 22:18:30.087552 data_node.go:114 Deleting volume id: 113 I0824 22:18:30.087595 topology.go:534 removing volume info: Id:113, Size:1, ReplicaPlacement:000, Collection:c, Version:3, Ttl:, FileCount:0, DeleteCount:0, DeletedByteCount:0, ReadOnly:false, ModifiedAtSecond:0 from 10.0.0.1:8080 I0824 22:18:30.087628 volume_layout.go:815 Volume 113 becomes unwritable I0824 22:18:30.087646 volume_layout.go:335 volume 113 does not have enough copies I0824 22:18:30.087701 data_node.go:114 Deleting volume id: 114 I0824 22:18:30.087761 topology.go:534 removing volume info: Id:114, Size:1, ReplicaPlacement:000, Collection:c, Version:3, Ttl:, FileCount:0, DeleteCount:0, DeletedByteCount:0, ReadOnly:false, ModifiedAtSecond:0 from 10.0.0.1:8080 I0824 22:18:30.087792 volume_layout.go:815 Volume 114 becomes unwritable I0824 22:18:30.087832 volume_layout.go:335 volume 114 does not have enough copies I0824 22:18:30.087912 data_node.go:114 Deleting volume id: 115 I0824 22:18:30.087956 topology.go:534 removing volume info: Id:115, Size:1, ReplicaPlacement:000, Collection:c, Version:3, Ttl:, FileCount:0, DeleteCount:0, DeletedByteCount:0, ReadOnly:false, ModifiedAtSecond:0 from 10.0.0.1:8080 I0824 22:18:30.087997 volume_layout.go:815 Volume 115 becomes unwritable I0824 22:18:30.088019 volume_layout.go:335 volume 115 does not have enough copies I0824 22:18:30.088107 data_node.go:114 Deleting volume id: 116 I0824 22:18:30.088214 topology.go:534 removing volume info: Id:116, Size:1, ReplicaPlacement:000, Collection:c, Version:3, Ttl:, FileCount:0, DeleteCount:0, DeletedByteCount:0, ReadOnly:false, ModifiedAtSecond:0 from 10.0.0.1:8080 I0824 22:18:30.088283 volume_layout.go:815 Volume 116 becomes unwritable I0824 22:18:30.088309 volume_layout.go:335 volume 116 does not have enough copies I0824 22:18:30.088417 data_node.go:114 Deleting volume id: 117 I0824 22:18:30.088484 topology.go:534 removing volume info: Id:117, Size:1, ReplicaPlacement:000, Collection:c, Version:3, Ttl:, FileCount:0, DeleteCount:0, DeletedByteCount:0, ReadOnly:false, ModifiedAtSecond:0 from 10.0.0.1:8080 I0824 22:18:30.088644 volume_layout.go:815 Volume 117 becomes unwritable I0824 22:18:30.088680 volume_layout.go:335 volume 117 does not have enough copies I0824 22:18:30.088881 data_node.go:114 Deleting volume id: 118 I0824 22:18:30.089043 topology.go:534 removing volume info: Id:118, Size:1, ReplicaPlacement:000, Collection:c, Version:3, Ttl:, FileCount:0, DeleteCount:0, DeletedByteCount:0, ReadOnly:false, ModifiedAtSecond:0 from 10.0.0.1:8080 I0824 22:18:30.089086 volume_layout.go:815 Volume 118 becomes unwritable I0824 22:18:30.089108 volume_layout.go:335 volume 118 does not have enough copies I0824 22:18:30.089234 data_node.go:114 Deleting volume id: 119 I0824 22:18:30.089337 topology.go:534 removing volume info: Id:119, Size:1, ReplicaPlacement:000, Collection:c, Version:3, Ttl:, FileCount:0, DeleteCount:0, DeletedByteCount:0, ReadOnly:false, ModifiedAtSecond:0 from 10.0.0.1:8080 I0824 22:18:30.089406 volume_layout.go:815 Volume 119 becomes unwritable I0824 22:18:30.089447 volume_layout.go:335 volume 119 does not have enough copies I0824 22:18:30.089514 data_node.go:114 Deleting volume id: 120 I0824 22:18:30.089608 topology.go:534 removing volume info: Id:120, Size:1, ReplicaPlacement:000, Collection:c, Version:3, Ttl:, FileCount:0, DeleteCount:0, DeletedByteCount:0, ReadOnly:false, ModifiedAtSecond:0 from 10.0.0.1:8080 I0824 22:18:30.089642 volume_layout.go:815 Volume 120 becomes unwritable I0824 22:18:30.089680 volume_layout.go:335 volume 120 does not have enough copies I0824 22:18:30.089839 data_node.go:114 Deleting volume id: 121 I0824 22:18:30.089883 topology.go:534 removing volume info: Id:121, Size:1, ReplicaPlacement:000, Collection:c, Version:3, Ttl:, FileCount:0, DeleteCount:0, DeletedByteCount:0, ReadOnly:false, ModifiedAtSecond:0 from 10.0.0.1:8080 I0824 22:18:30.089953 volume_layout.go:815 Volume 121 becomes unwritable I0824 22:18:30.090010 volume_layout.go:335 volume 121 does not have enough copies I0824 22:18:30.090067 data_node.go:114 Deleting volume id: 122 I0824 22:18:30.090113 topology.go:534 removing volume info: Id:122, Size:1, ReplicaPlacement:000, Collection:c, Version:3, Ttl:, FileCount:0, DeleteCount:0, DeletedByteCount:0, ReadOnly:false, ModifiedAtSecond:0 from 10.0.0.1:8080 I0824 22:18:30.090144 volume_layout.go:815 Volume 122 becomes unwritable I0824 22:18:30.090162 volume_layout.go:335 volume 122 does not have enough copies I0824 22:18:30.090233 data_node.go:114 Deleting volume id: 123 I0824 22:18:30.090319 topology.go:534 removing volume info: Id:123, Size:1, ReplicaPlacement:000, Collection:c, Version:3, Ttl:, FileCount:0, DeleteCount:0, DeletedByteCount:0, ReadOnly:false, ModifiedAtSecond:0 from 10.0.0.1:8080 I0824 22:18:30.090432 volume_layout.go:815 Volume 123 becomes unwritable I0824 22:18:30.090455 volume_layout.go:335 volume 123 does not have enough copies I0824 22:18:30.090531 data_node.go:114 Deleting volume id: 124 I0824 22:18:30.090611 topology.go:534 removing volume info: Id:124, Size:1, ReplicaPlacement:000, Collection:c, Version:3, Ttl:, FileCount:0, DeleteCount:0, DeletedByteCount:0, ReadOnly:false, ModifiedAtSecond:0 from 10.0.0.1:8080 I0824 22:18:30.090684 volume_layout.go:815 Volume 124 becomes unwritable I0824 22:18:30.090716 volume_layout.go:335 volume 124 does not have enough copies I0824 22:18:30.090803 data_node.go:114 Deleting volume id: 125 I0824 22:18:30.090849 topology.go:534 removing volume info: Id:125, Size:1, ReplicaPlacement:000, Collection:c, Version:3, Ttl:, FileCount:0, DeleteCount:0, DeletedByteCount:0, ReadOnly:false, ModifiedAtSecond:0 from 10.0.0.1:8080 I0824 22:18:30.090882 volume_layout.go:815 Volume 125 becomes unwritable I0824 22:18:30.090918 volume_layout.go:335 volume 125 does not have enough copies I0824 22:18:30.090975 data_node.go:114 Deleting volume id: 126 I0824 22:18:30.091016 topology.go:534 removing volume info: Id:126, Size:1, ReplicaPlacement:000, Collection:c, Version:3, Ttl:, FileCount:0, DeleteCount:0, DeletedByteCount:0, ReadOnly:false, ModifiedAtSecond:0 from 10.0.0.1:8080 I0824 22:18:30.091046 volume_layout.go:815 Volume 126 becomes unwritable I0824 22:18:30.091167 volume_layout.go:335 volume 126 does not have enough copies I0824 22:18:30.091268 data_node.go:114 Deleting volume id: 127 I0824 22:18:30.091349 topology.go:534 removing volume info: Id:127, Size:1, ReplicaPlacement:000, Collection:c, Version:3, Ttl:, FileCount:0, DeleteCount:0, DeletedByteCount:0, ReadOnly:false, ModifiedAtSecond:0 from 10.0.0.1:8080 I0824 22:18:30.091439 volume_layout.go:815 Volume 127 becomes unwritable I0824 22:18:30.091463 volume_layout.go:335 volume 127 does not have enough copies I0824 22:18:30.091518 data_node.go:114 Deleting volume id: 128 I0824 22:18:30.091562 topology.go:534 removing volume info: Id:128, Size:1, ReplicaPlacement:000, Collection:c, Version:3, Ttl:, FileCount:0, DeleteCount:0, DeletedByteCount:0, ReadOnly:false, ModifiedAtSecond:0 from 10.0.0.1:8080 I0824 22:18:30.091594 volume_layout.go:815 Volume 128 becomes unwritable I0824 22:18:30.091613 volume_layout.go:335 volume 128 does not have enough copies I0824 22:18:30.091671 data_node.go:114 Deleting volume id: 129 I0824 22:18:30.091712 topology.go:534 removing volume info: Id:129, Size:1, ReplicaPlacement:000, Collection:c, Version:3, Ttl:, FileCount:0, DeleteCount:0, DeletedByteCount:0, ReadOnly:false, ModifiedAtSecond:0 from 10.0.0.1:8080 I0824 22:18:30.091778 volume_layout.go:815 Volume 129 becomes unwritable I0824 22:18:30.091800 volume_layout.go:335 volume 129 does not have enough copies I0824 22:18:30.091866 data_node.go:114 Deleting volume id: 130 I0824 22:18:30.091944 topology.go:534 removing volume info: Id:130, Size:1, ReplicaPlacement:000, Collection:c, Version:3, Ttl:, FileCount:0, DeleteCount:0, DeletedByteCount:0, ReadOnly:false, ModifiedAtSecond:0 from 10.0.0.1:8080 I0824 22:18:30.091989 volume_layout.go:815 Volume 130 becomes unwritable I0824 22:18:30.092007 volume_layout.go:335 volume 130 does not have enough copies I0824 22:18:30.092091 data_node.go:114 Deleting volume id: 131 I0824 22:18:30.092131 topology.go:534 removing volume info: Id:131, Size:1, ReplicaPlacement:000, Collection:c, Version:3, Ttl:, FileCount:0, DeleteCount:0, DeletedByteCount:0, ReadOnly:false, ModifiedAtSecond:0 from 10.0.0.1:8080 I0824 22:18:30.092162 volume_layout.go:815 Volume 131 becomes unwritable I0824 22:18:30.092195 volume_layout.go:335 volume 131 does not have enough copies I0824 22:18:30.092251 data_node.go:114 Deleting volume id: 132 I0824 22:18:30.092477 topology.go:534 removing volume info: Id:132, Size:1, ReplicaPlacement:000, Collection:c, Version:3, Ttl:, FileCount:0, DeleteCount:0, DeletedByteCount:0, ReadOnly:false, ModifiedAtSecond:0 from 10.0.0.1:8080 I0824 22:18:30.092543 volume_layout.go:815 Volume 132 becomes unwritable I0824 22:18:30.092563 volume_layout.go:335 volume 132 does not have enough copies I0824 22:18:30.092630 data_node.go:114 Deleting volume id: 133 I0824 22:18:30.092705 topology.go:534 removing volume info: Id:133, Size:1, ReplicaPlacement:000, Collection:c, Version:3, Ttl:, FileCount:0, DeleteCount:0, DeletedByteCount:0, ReadOnly:false, ModifiedAtSecond:0 from 10.0.0.1:8080 I0824 22:18:30.092739 volume_layout.go:815 Volume 133 becomes unwritable I0824 22:18:30.092758 volume_layout.go:335 volume 133 does not have enough copies I0824 22:18:30.092838 data_node.go:114 Deleting volume id: 134 I0824 22:18:30.092907 topology.go:534 removing volume info: Id:134, Size:1, ReplicaPlacement:000, Collection:c, Version:3, Ttl:, FileCount:0, DeleteCount:0, DeletedByteCount:0, ReadOnly:false, ModifiedAtSecond:0 from 10.0.0.1:8080 I0824 22:18:30.092954 volume_layout.go:815 Volume 134 becomes unwritable I0824 22:18:30.092977 volume_layout.go:335 volume 134 does not have enough copies I0824 22:18:30.093097 data_node.go:114 Deleting volume id: 135 I0824 22:18:30.093233 topology.go:534 removing volume info: Id:135, Size:1, ReplicaPlacement:000, Collection:c, Version:3, Ttl:, FileCount:0, DeleteCount:0, DeletedByteCount:0, ReadOnly:false, ModifiedAtSecond:0 from 10.0.0.1:8080 I0824 22:18:30.093303 volume_layout.go:815 Volume 135 becomes unwritable I0824 22:18:30.093423 volume_layout.go:335 volume 135 does not have enough copies I0824 22:18:30.093529 data_node.go:114 Deleting volume id: 136 I0824 22:18:30.093598 topology.go:534 removing volume info: Id:136, Size:1, ReplicaPlacement:000, Collection:c, Version:3, Ttl:, FileCount:0, DeleteCount:0, DeletedByteCount:0, ReadOnly:false, ModifiedAtSecond:0 from 10.0.0.1:8080 I0824 22:18:30.093634 volume_layout.go:815 Volume 136 becomes unwritable I0824 22:18:30.093651 volume_layout.go:335 volume 136 does not have enough copies I0824 22:18:30.093695 data_node.go:114 Deleting volume id: 137 I0824 22:18:30.094041 topology.go:534 removing volume info: Id:137, Size:1, ReplicaPlacement:000, Collection:c, Version:3, Ttl:, FileCount:0, DeleteCount:0, DeletedByteCount:0, ReadOnly:false, ModifiedAtSecond:0 from 10.0.0.1:8080 I0824 22:18:30.094098 volume_layout.go:815 Volume 137 becomes unwritable I0824 22:18:30.094141 volume_layout.go:335 volume 137 does not have enough copies I0824 22:18:30.094216 data_node.go:114 Deleting volume id: 138 I0824 22:18:30.094289 topology.go:534 removing volume info: Id:138, Size:1, ReplicaPlacement:000, Collection:c, Version:3, Ttl:, FileCount:0, DeleteCount:0, DeletedByteCount:0, ReadOnly:false, ModifiedAtSecond:0 from 10.0.0.1:8080 I0824 22:18:30.094320 volume_layout.go:815 Volume 138 becomes unwritable I0824 22:18:30.094339 volume_layout.go:335 volume 138 does not have enough copies I0824 22:18:30.094412 data_node.go:114 Deleting volume id: 139 I0824 22:18:30.094486 topology.go:534 removing volume info: Id:139, Size:1, ReplicaPlacement:000, Collection:c, Version:3, Ttl:, FileCount:0, DeleteCount:0, DeletedByteCount:0, ReadOnly:false, ModifiedAtSecond:0 from 10.0.0.1:8080 I0824 22:18:30.094520 volume_layout.go:815 Volume 139 becomes unwritable I0824 22:18:30.094540 volume_layout.go:335 volume 139 does not have enough copies I0824 22:18:30.094602 data_node.go:114 Deleting volume id: 140 I0824 22:18:30.094685 topology.go:534 removing volume info: Id:140, Size:1, ReplicaPlacement:000, Collection:c, Version:3, Ttl:, FileCount:0, DeleteCount:0, DeletedByteCount:0, ReadOnly:false, ModifiedAtSecond:0 from 10.0.0.1:8080 I0824 22:18:30.094717 volume_layout.go:815 Volume 140 becomes unwritable I0824 22:18:30.094735 volume_layout.go:335 volume 140 does not have enough copies I0824 22:18:30.094819 data_node.go:114 Deleting volume id: 141 I0824 22:18:30.094883 topology.go:534 removing volume info: Id:141, Size:1, ReplicaPlacement:000, Collection:c, Version:3, Ttl:, FileCount:0, DeleteCount:0, DeletedByteCount:0, ReadOnly:false, ModifiedAtSecond:0 from 10.0.0.1:8080 I0824 22:18:30.094917 volume_layout.go:815 Volume 141 becomes unwritable I0824 22:18:30.094936 volume_layout.go:335 volume 141 does not have enough copies I0824 22:18:30.095025 data_node.go:114 Deleting volume id: 142 I0824 22:18:30.095127 topology.go:534 removing volume info: Id:142, Size:1, ReplicaPlacement:000, Collection:c, Version:3, Ttl:, FileCount:0, DeleteCount:0, DeletedByteCount:0, ReadOnly:false, ModifiedAtSecond:0 from 10.0.0.1:8080 I0824 22:18:30.095277 volume_layout.go:815 Volume 142 becomes unwritable I0824 22:18:30.095298 volume_layout.go:335 volume 142 does not have enough copies I0824 22:18:30.095339 data_node.go:114 Deleting volume id: 143 I0824 22:18:30.096016 topology.go:534 removing volume info: Id:143, Size:1, ReplicaPlacement:000, Collection:c, Version:3, Ttl:, FileCount:0, DeleteCount:0, DeletedByteCount:0, ReadOnly:false, ModifiedAtSecond:0 from 10.0.0.1:8080 I0824 22:18:30.096683 volume_layout.go:815 Volume 143 becomes unwritable I0824 22:18:30.096873 volume_layout.go:335 volume 143 does not have enough copies I0824 22:18:30.097273 data_node.go:114 Deleting volume id: 144 I0824 22:18:30.097621 topology.go:534 removing volume info: Id:144, Size:1, ReplicaPlacement:000, Collection:c, Version:3, Ttl:, FileCount:0, DeleteCount:0, DeletedByteCount:0, ReadOnly:false, ModifiedAtSecond:0 from 10.0.0.1:8080 I0824 22:18:30.097842 volume_layout.go:815 Volume 144 becomes unwritable I0824 22:18:30.097947 volume_layout.go:335 volume 144 does not have enough copies I0824 22:18:30.098248 data_node.go:114 Deleting volume id: 145 I0824 22:18:30.098494 topology.go:534 removing volume info: Id:145, Size:1, ReplicaPlacement:000, Collection:c, Version:3, Ttl:, FileCount:0, DeleteCount:0, DeletedByteCount:0, ReadOnly:false, ModifiedAtSecond:0 from 10.0.0.1:8080 I0824 22:18:30.098765 volume_layout.go:815 Volume 145 becomes unwritable I0824 22:18:30.098841 volume_layout.go:335 volume 145 does not have enough copies I0824 22:18:30.098958 data_node.go:114 Deleting volume id: 146 I0824 22:18:30.099256 topology.go:534 removing volume info: Id:146, Size:1, ReplicaPlacement:000, Collection:c, Version:3, Ttl:, FileCount:0, DeleteCount:0, DeletedByteCount:0, ReadOnly:false, ModifiedAtSecond:0 from 10.0.0.1:8080 I0824 22:18:30.099423 volume_layout.go:815 Volume 146 becomes unwritable I0824 22:18:30.099461 volume_layout.go:335 volume 146 does not have enough copies I0824 22:18:30.099720 data_node.go:114 Deleting volume id: 147 I0824 22:18:30.099985 topology.go:534 removing volume info: Id:147, Size:1, ReplicaPlacement:000, Collection:c, Version:3, Ttl:, FileCount:0, DeleteCount:0, DeletedByteCount:0, ReadOnly:false, ModifiedAtSecond:0 from 10.0.0.1:8080 I0824 22:18:30.100212 volume_layout.go:815 Volume 147 becomes unwritable I0824 22:18:30.100291 volume_layout.go:335 volume 147 does not have enough copies I0824 22:18:30.100490 data_node.go:114 Deleting volume id: 148 I0824 22:18:30.100575 topology.go:534 removing volume info: Id:148, Size:1, ReplicaPlacement:000, Collection:c, Version:3, Ttl:, FileCount:0, DeleteCount:0, DeletedByteCount:0, ReadOnly:false, ModifiedAtSecond:0 from 10.0.0.1:8080 I0824 22:18:30.100755 volume_layout.go:815 Volume 148 becomes unwritable I0824 22:18:30.100796 volume_layout.go:335 volume 148 does not have enough copies I0824 22:18:30.100934 data_node.go:114 Deleting volume id: 149 I0824 22:18:30.101496 topology.go:534 removing volume info: Id:149, Size:1, ReplicaPlacement:000, Collection:c, Version:3, Ttl:, FileCount:0, DeleteCount:0, DeletedByteCount:0, ReadOnly:false, ModifiedAtSecond:0 from 10.0.0.1:8080 I0824 22:18:30.101602 volume_layout.go:815 Volume 149 becomes unwritable I0824 22:18:30.101647 volume_layout.go:335 volume 149 does not have enough copies I0824 22:18:30.101935 data_node.go:114 Deleting volume id: 100 I0824 22:18:30.102335 topology.go:534 removing volume info: Id:100, Size:1, ReplicaPlacement:000, Collection:c, Version:3, Ttl:, FileCount:0, DeleteCount:0, DeletedByteCount:0, ReadOnly:false, ModifiedAtSecond:0 from 10.0.0.1:8080 I0824 22:18:30.102720 volume_layout.go:815 Volume 100 becomes unwritable I0824 22:18:30.102838 volume_layout.go:335 volume 100 does not have enough copies I0824 22:18:30.103606 data_node.go:114 Deleting volume id: 101 I0824 22:18:30.103939 topology.go:534 removing volume info: Id:101, Size:1, ReplicaPlacement:000, Collection:c, Version:3, Ttl:, FileCount:0, DeleteCount:0, DeletedByteCount:0, ReadOnly:false, ModifiedAtSecond:0 from 10.0.0.1:8080 I0824 22:18:30.104211 volume_layout.go:815 Volume 101 becomes unwritable I0824 22:18:30.104283 volume_layout.go:335 volume 101 does not have enough copies I0824 22:18:30.104759 data_node.go:114 Deleting volume id: 102 I0824 22:18:30.107982 topology.go:534 removing volume info: Id:102, Size:1, ReplicaPlacement:000, Collection:c, Version:3, Ttl:, FileCount:0, DeleteCount:0, DeletedByteCount:0, ReadOnly:false, ModifiedAtSecond:0 from 10.0.0.1:8080 I0824 22:18:30.108498 volume_layout.go:815 Volume 102 becomes unwritable I0824 22:18:30.108696 volume_layout.go:335 volume 102 does not have enough copies I0824 22:18:30.109286 data_node.go:114 Deleting volume id: 103 I0824 22:18:30.109582 topology.go:534 removing volume info: Id:103, Size:1, ReplicaPlacement:000, Collection:c, Version:3, Ttl:, FileCount:0, DeleteCount:0, DeletedByteCount:0, ReadOnly:false, ModifiedAtSecond:0 from 10.0.0.1:8080 I0824 22:18:30.109841 volume_layout.go:815 Volume 103 becomes unwritable I0824 22:18:30.110008 volume_layout.go:335 volume 103 does not have enough copies I0824 22:18:30.110273 data_node.go:114 Deleting volume id: 104 I0824 22:18:30.110748 topology.go:534 removing volume info: Id:104, Size:1, ReplicaPlacement:000, Collection:c, Version:3, Ttl:, FileCount:0, DeleteCount:0, DeletedByteCount:0, ReadOnly:false, ModifiedAtSecond:0 from 10.0.0.1:8080 I0824 22:18:30.112163 volume_layout.go:815 Volume 104 becomes unwritable I0824 22:18:30.112564 volume_layout.go:335 volume 104 does not have enough copies I0824 22:18:30.113265 data_node.go:114 Deleting volume id: 105 I0824 22:18:30.113593 topology.go:534 removing volume info: Id:105, Size:1, ReplicaPlacement:000, Collection:c, Version:3, Ttl:, FileCount:0, DeleteCount:0, DeletedByteCount:0, ReadOnly:false, ModifiedAtSecond:0 from 10.0.0.1:8080 I0824 22:18:30.114201 volume_layout.go:815 Volume 105 becomes unwritable I0824 22:18:30.114275 volume_layout.go:335 volume 105 does not have enough copies I0824 22:18:30.114380 data_node.go:114 Deleting volume id: 106 I0824 22:18:30.114542 topology.go:534 removing volume info: Id:106, Size:1, ReplicaPlacement:000, Collection:c, Version:3, Ttl:, FileCount:0, DeleteCount:0, DeletedByteCount:0, ReadOnly:false, ModifiedAtSecond:0 from 10.0.0.1:8080 I0824 22:18:30.114626 volume_layout.go:815 Volume 106 becomes unwritable I0824 22:18:30.114649 volume_layout.go:335 volume 106 does not have enough copies I0824 22:18:30.114795 data_node.go:114 Deleting volume id: 107 I0824 22:18:30.114884 topology.go:534 removing volume info: Id:107, Size:1, ReplicaPlacement:000, Collection:c, Version:3, Ttl:, FileCount:0, DeleteCount:0, DeletedByteCount:0, ReadOnly:false, ModifiedAtSecond:0 from 10.0.0.1:8080 I0824 22:18:30.114979 volume_layout.go:815 Volume 107 becomes unwritable I0824 22:18:30.115005 volume_layout.go:335 volume 107 does not have enough copies I0824 22:18:30.115115 data_node.go:114 Deleting volume id: 108 I0824 22:18:30.115206 topology.go:534 removing volume info: Id:108, Size:1, ReplicaPlacement:000, Collection:c, Version:3, Ttl:, FileCount:0, DeleteCount:0, DeletedByteCount:0, ReadOnly:false, ModifiedAtSecond:0 from 10.0.0.1:8080 I0824 22:18:30.115251 volume_layout.go:815 Volume 108 becomes unwritable I0824 22:18:30.115296 volume_layout.go:335 volume 108 does not have enough copies I0824 22:18:30.115379 data_node.go:114 Deleting volume id: 109 I0824 22:18:30.115428 topology.go:534 removing volume info: Id:109, Size:1, ReplicaPlacement:000, Collection:c, Version:3, Ttl:, FileCount:0, DeleteCount:0, DeletedByteCount:0, ReadOnly:false, ModifiedAtSecond:0 from 10.0.0.1:8080 I0824 22:18:30.115501 volume_layout.go:815 Volume 109 becomes unwritable I0824 22:18:30.115526 volume_layout.go:335 volume 109 does not have enough copies I0824 22:18:30.115607 data_node.go:114 Deleting volume id: 110 I0824 22:18:30.115758 topology.go:534 removing volume info: Id:110, Size:1, ReplicaPlacement:000, Collection:c, Version:3, Ttl:, FileCount:0, DeleteCount:0, DeletedByteCount:0, ReadOnly:false, ModifiedAtSecond:0 from 10.0.0.1:8080 I0824 22:18:30.115856 volume_layout.go:815 Volume 110 becomes unwritable I0824 22:18:30.115878 volume_layout.go:335 volume 110 does not have enough copies I0824 22:18:30.116011 data_node.go:114 Deleting volume id: 111 I0824 22:18:30.117031 topology.go:534 removing volume info: Id:111, Size:1, ReplicaPlacement:000, Collection:c, Version:3, Ttl:, FileCount:0, DeleteCount:0, DeletedByteCount:0, ReadOnly:false, ModifiedAtSecond:0 from 10.0.0.1:8080 I0824 22:18:30.118275 volume_layout.go:815 Volume 111 becomes unwritable I0824 22:18:30.118372 volume_layout.go:335 volume 111 does not have enough copies I0824 22:18:30.118530 data_node.go:114 Deleting volume id: 112 I0824 22:18:30.118716 topology.go:534 removing volume info: Id:112, Size:1, ReplicaPlacement:000, Collection:c, Version:3, Ttl:, FileCount:0, DeleteCount:0, DeletedByteCount:0, ReadOnly:false, ModifiedAtSecond:0 from 10.0.0.1:8080 I0824 22:18:30.118809 volume_layout.go:815 Volume 112 becomes unwritable I0824 22:18:30.118862 volume_layout.go:335 volume 112 does not have enough copies I0824 22:18:30.119103 data_node.go:114 Deleting volume id: 113 I0824 22:18:30.119240 topology.go:534 removing volume info: Id:113, Size:1, ReplicaPlacement:000, Collection:c, Version:3, Ttl:, FileCount:0, DeleteCount:0, DeletedByteCount:0, ReadOnly:false, ModifiedAtSecond:0 from 10.0.0.1:8080 I0824 22:18:30.119311 volume_layout.go:815 Volume 113 becomes unwritable I0824 22:18:30.119334 volume_layout.go:335 volume 113 does not have enough copies I0824 22:18:30.119421 data_node.go:114 Deleting volume id: 114 I0824 22:18:30.119545 topology.go:534 removing volume info: Id:114, Size:1, ReplicaPlacement:000, Collection:c, Version:3, Ttl:, FileCount:0, DeleteCount:0, DeletedByteCount:0, ReadOnly:false, ModifiedAtSecond:0 from 10.0.0.1:8080 I0824 22:18:30.119653 volume_layout.go:815 Volume 114 becomes unwritable I0824 22:18:30.119677 volume_layout.go:335 volume 114 does not have enough copies I0824 22:18:30.119784 data_node.go:114 Deleting volume id: 115 I0824 22:18:30.119845 topology.go:534 removing volume info: Id:115, Size:1, ReplicaPlacement:000, Collection:c, Version:3, Ttl:, FileCount:0, DeleteCount:0, DeletedByteCount:0, ReadOnly:false, ModifiedAtSecond:0 from 10.0.0.1:8080 I0824 22:18:30.119918 volume_layout.go:815 Volume 115 becomes unwritable I0824 22:18:30.119939 volume_layout.go:335 volume 115 does not have enough copies I0824 22:18:30.120024 data_node.go:114 Deleting volume id: 116 I0824 22:18:30.120107 topology.go:534 removing volume info: Id:116, Size:1, ReplicaPlacement:000, Collection:c, Version:3, Ttl:, FileCount:0, DeleteCount:0, DeletedByteCount:0, ReadOnly:false, ModifiedAtSecond:0 from 10.0.0.1:8080 I0824 22:18:30.120165 volume_layout.go:815 Volume 116 becomes unwritable I0824 22:18:30.120207 volume_layout.go:335 volume 116 does not have enough copies I0824 22:18:30.120392 data_node.go:114 Deleting volume id: 117 I0824 22:18:30.120544 topology.go:534 removing volume info: Id:117, Size:1, ReplicaPlacement:000, Collection:c, Version:3, Ttl:, FileCount:0, DeleteCount:0, DeletedByteCount:0, ReadOnly:false, ModifiedAtSecond:0 from 10.0.0.1:8080 I0824 22:18:30.120586 volume_layout.go:815 Volume 117 becomes unwritable I0824 22:18:30.120608 volume_layout.go:335 volume 117 does not have enough copies I0824 22:18:30.120731 data_node.go:114 Deleting volume id: 118 I0824 22:18:30.120865 topology.go:534 removing volume info: Id:118, Size:1, ReplicaPlacement:000, Collection:c, Version:3, Ttl:, FileCount:0, DeleteCount:0, DeletedByteCount:0, ReadOnly:false, ModifiedAtSecond:0 from 10.0.0.1:8080 I0824 22:18:30.120908 volume_layout.go:815 Volume 118 becomes unwritable I0824 22:18:30.120931 volume_layout.go:335 volume 118 does not have enough copies I0824 22:18:30.121002 data_node.go:114 Deleting volume id: 119 I0824 22:18:30.121051 topology.go:534 removing volume info: Id:119, Size:1, ReplicaPlacement:000, Collection:c, Version:3, Ttl:, FileCount:0, DeleteCount:0, DeletedByteCount:0, ReadOnly:false, ModifiedAtSecond:0 from 10.0.0.1:8080 I0824 22:18:30.121087 volume_layout.go:815 Volume 119 becomes unwritable I0824 22:18:30.121107 volume_layout.go:335 volume 119 does not have enough copies I0824 22:18:30.121187 data_node.go:114 Deleting volume id: 120 I0824 22:18:30.121290 topology.go:534 removing volume info: Id:120, Size:1, ReplicaPlacement:000, Collection:c, Version:3, Ttl:, FileCount:0, DeleteCount:0, DeletedByteCount:0, ReadOnly:false, ModifiedAtSecond:0 from 10.0.0.1:8080 I0824 22:18:30.122604 volume_layout.go:815 Volume 120 becomes unwritable I0824 22:18:30.122703 volume_layout.go:335 volume 120 does not have enough copies I0824 22:18:30.122924 data_node.go:114 Deleting volume id: 121 I0824 22:18:30.123023 topology.go:534 removing volume info: Id:121, Size:1, ReplicaPlacement:000, Collection:c, Version:3, Ttl:, FileCount:0, DeleteCount:0, DeletedByteCount:0, ReadOnly:false, ModifiedAtSecond:0 from 10.0.0.1:8080 I0824 22:18:30.123086 volume_layout.go:815 Volume 121 becomes unwritable I0824 22:18:30.123126 volume_layout.go:335 volume 121 does not have enough copies I0824 22:18:30.123277 data_node.go:114 Deleting volume id: 122 I0824 22:18:30.123372 topology.go:534 removing volume info: Id:122, Size:1, ReplicaPlacement:000, Collection:c, Version:3, Ttl:, FileCount:0, DeleteCount:0, DeletedByteCount:0, ReadOnly:false, ModifiedAtSecond:0 from 10.0.0.1:8080 I0824 22:18:30.123456 volume_layout.go:815 Volume 122 becomes unwritable I0824 22:18:30.123481 volume_layout.go:335 volume 122 does not have enough copies I0824 22:18:30.123553 data_node.go:114 Deleting volume id: 123 I0824 22:18:30.123600 topology.go:534 removing volume info: Id:123, Size:1, ReplicaPlacement:000, Collection:c, Version:3, Ttl:, FileCount:0, DeleteCount:0, DeletedByteCount:0, ReadOnly:false, ModifiedAtSecond:0 from 10.0.0.1:8080 I0824 22:18:30.123648 volume_layout.go:815 Volume 123 becomes unwritable I0824 22:18:30.123669 volume_layout.go:335 volume 123 does not have enough copies I0824 22:18:30.123774 data_node.go:114 Deleting volume id: 124 I0824 22:18:30.123919 topology.go:534 removing volume info: Id:124, Size:1, ReplicaPlacement:000, Collection:c, Version:3, Ttl:, FileCount:0, DeleteCount:0, DeletedByteCount:0, ReadOnly:false, ModifiedAtSecond:0 from 10.0.0.1:8080 I0824 22:18:30.123958 volume_layout.go:815 Volume 124 becomes unwritable I0824 22:18:30.123980 volume_layout.go:335 volume 124 does not have enough copies I0824 22:18:30.124044 data_node.go:114 Deleting volume id: 125 I0824 22:18:30.124094 topology.go:534 removing volume info: Id:125, Size:1, ReplicaPlacement:000, Collection:c, Version:3, Ttl:, FileCount:0, DeleteCount:0, DeletedByteCount:0, ReadOnly:false, ModifiedAtSecond:0 from 10.0.0.1:8080 I0824 22:18:30.124147 volume_layout.go:815 Volume 125 becomes unwritable I0824 22:18:30.124187 volume_layout.go:335 volume 125 does not have enough copies I0824 22:18:30.124607 data_node.go:114 Deleting volume id: 126 I0824 22:18:30.124785 topology.go:534 removing volume info: Id:126, Size:1, ReplicaPlacement:000, Collection:c, Version:3, Ttl:, FileCount:0, DeleteCount:0, DeletedByteCount:0, ReadOnly:false, ModifiedAtSecond:0 from 10.0.0.1:8080 I0824 22:18:30.124957 volume_layout.go:815 Volume 126 becomes unwritable I0824 22:18:30.124991 volume_layout.go:335 volume 126 does not have enough copies I0824 22:18:30.125111 data_node.go:114 Deleting volume id: 127 I0824 22:18:30.125186 topology.go:534 removing volume info: Id:127, Size:1, ReplicaPlacement:000, Collection:c, Version:3, Ttl:, FileCount:0, DeleteCount:0, DeletedByteCount:0, ReadOnly:false, ModifiedAtSecond:0 from 10.0.0.1:8080 I0824 22:18:30.125264 volume_layout.go:815 Volume 127 becomes unwritable I0824 22:18:30.125336 volume_layout.go:335 volume 127 does not have enough copies I0824 22:18:30.125545 data_node.go:114 Deleting volume id: 128 I0824 22:18:30.125612 topology.go:534 removing volume info: Id:128, Size:1, ReplicaPlacement:000, Collection:c, Version:3, Ttl:, FileCount:0, DeleteCount:0, DeletedByteCount:0, ReadOnly:false, ModifiedAtSecond:0 from 10.0.0.1:8080 I0824 22:18:30.125650 volume_layout.go:815 Volume 128 becomes unwritable I0824 22:18:30.125673 volume_layout.go:335 volume 128 does not have enough copies I0824 22:18:30.125743 data_node.go:114 Deleting volume id: 129 I0824 22:18:30.125790 topology.go:534 removing volume info: Id:129, Size:1, ReplicaPlacement:000, Collection:c, Version:3, Ttl:, FileCount:0, DeleteCount:0, DeletedByteCount:0, ReadOnly:false, ModifiedAtSecond:0 from 10.0.0.1:8080 I0824 22:18:30.127319 volume_layout.go:815 Volume 129 becomes unwritable I0824 22:18:30.127382 volume_layout.go:335 volume 129 does not have enough copies I0824 22:18:30.127533 data_node.go:114 Deleting volume id: 130 I0824 22:18:30.127648 topology.go:534 removing volume info: Id:130, Size:1, ReplicaPlacement:000, Collection:c, Version:3, Ttl:, FileCount:0, DeleteCount:0, DeletedByteCount:0, ReadOnly:false, ModifiedAtSecond:0 from 10.0.0.1:8080 I0824 22:18:30.127732 volume_layout.go:815 Volume 130 becomes unwritable I0824 22:18:30.127755 volume_layout.go:335 volume 130 does not have enough copies I0824 22:18:30.127857 data_node.go:114 Deleting volume id: 131 I0824 22:18:30.127902 topology.go:534 removing volume info: Id:131, Size:1, ReplicaPlacement:000, Collection:c, Version:3, Ttl:, FileCount:0, DeleteCount:0, DeletedByteCount:0, ReadOnly:false, ModifiedAtSecond:0 from 10.0.0.1:8080 I0824 22:18:30.127936 volume_layout.go:815 Volume 131 becomes unwritable I0824 22:18:30.127954 volume_layout.go:335 volume 131 does not have enough copies I0824 22:18:30.128020 data_node.go:114 Deleting volume id: 132 I0824 22:18:30.128094 topology.go:534 removing volume info: Id:132, Size:1, ReplicaPlacement:000, Collection:c, Version:3, Ttl:, FileCount:0, DeleteCount:0, DeletedByteCount:0, ReadOnly:false, ModifiedAtSecond:0 from 10.0.0.1:8080 I0824 22:18:30.128128 volume_layout.go:815 Volume 132 becomes unwritable I0824 22:18:30.128146 volume_layout.go:335 volume 132 does not have enough copies I0824 22:18:30.128368 data_node.go:114 Deleting volume id: 133 I0824 22:18:30.128502 topology.go:534 removing volume info: Id:133, Size:1, ReplicaPlacement:000, Collection:c, Version:3, Ttl:, FileCount:0, DeleteCount:0, DeletedByteCount:0, ReadOnly:false, ModifiedAtSecond:0 from 10.0.0.1:8080 I0824 22:18:30.128546 volume_layout.go:815 Volume 133 becomes unwritable I0824 22:18:30.128565 volume_layout.go:335 volume 133 does not have enough copies I0824 22:18:30.128642 data_node.go:114 Deleting volume id: 134 I0824 22:18:30.128719 topology.go:534 removing volume info: Id:134, Size:1, ReplicaPlacement:000, Collection:c, Version:3, Ttl:, FileCount:0, DeleteCount:0, DeletedByteCount:0, ReadOnly:false, ModifiedAtSecond:0 from 10.0.0.1:8080 I0824 22:18:30.128751 volume_layout.go:815 Volume 134 becomes unwritable I0824 22:18:30.128771 volume_layout.go:335 volume 134 does not have enough copies I0824 22:18:30.128823 data_node.go:114 Deleting volume id: 135 I0824 22:18:30.128908 topology.go:534 removing volume info: Id:135, Size:1, ReplicaPlacement:000, Collection:c, Version:3, Ttl:, FileCount:0, DeleteCount:0, DeletedByteCount:0, ReadOnly:false, ModifiedAtSecond:0 from 10.0.0.1:8080 I0824 22:18:30.128939 volume_layout.go:815 Volume 135 becomes unwritable I0824 22:18:30.128958 volume_layout.go:335 volume 135 does not have enough copies I0824 22:18:30.129018 data_node.go:114 Deleting volume id: 136 I0824 22:18:30.129061 topology.go:534 removing volume info: Id:136, Size:1, ReplicaPlacement:000, Collection:c, Version:3, Ttl:, FileCount:0, DeleteCount:0, DeletedByteCount:0, ReadOnly:false, ModifiedAtSecond:0 from 10.0.0.1:8080 I0824 22:18:30.129091 volume_layout.go:815 Volume 136 becomes unwritable I0824 22:18:30.129109 volume_layout.go:335 volume 136 does not have enough copies I0824 22:18:30.129164 data_node.go:114 Deleting volume id: 137 I0824 22:18:30.129246 topology.go:534 removing volume info: Id:137, Size:1, ReplicaPlacement:000, Collection:c, Version:3, Ttl:, FileCount:0, DeleteCount:0, DeletedByteCount:0, ReadOnly:false, ModifiedAtSecond:0 from 10.0.0.1:8080 I0824 22:18:30.129280 volume_layout.go:815 Volume 137 becomes unwritable I0824 22:18:30.129324 volume_layout.go:335 volume 137 does not have enough copies I0824 22:18:30.129421 data_node.go:114 Deleting volume id: 138 I0824 22:18:30.129460 topology.go:534 removing volume info: Id:138, Size:1, ReplicaPlacement:000, Collection:c, Version:3, Ttl:, FileCount:0, DeleteCount:0, DeletedByteCount:0, ReadOnly:false, ModifiedAtSecond:0 from 10.0.0.1:8080 I0824 22:18:30.129504 volume_layout.go:815 Volume 138 becomes unwritable I0824 22:18:30.129524 volume_layout.go:335 volume 138 does not have enough copies I0824 22:18:30.129583 data_node.go:114 Deleting volume id: 139 I0824 22:18:30.129821 topology.go:534 removing volume info: Id:139, Size:1, ReplicaPlacement:000, Collection:c, Version:3, Ttl:, FileCount:0, DeleteCount:0, DeletedByteCount:0, ReadOnly:false, ModifiedAtSecond:0 from 10.0.0.1:8080 I0824 22:18:30.129925 volume_layout.go:815 Volume 139 becomes unwritable I0824 22:18:30.129946 volume_layout.go:335 volume 139 does not have enough copies I0824 22:18:30.130012 data_node.go:114 Deleting volume id: 140 I0824 22:18:30.130098 topology.go:534 removing volume info: Id:140, Size:1, ReplicaPlacement:000, Collection:c, Version:3, Ttl:, FileCount:0, DeleteCount:0, DeletedByteCount:0, ReadOnly:false, ModifiedAtSecond:0 from 10.0.0.1:8080 I0824 22:18:30.130148 volume_layout.go:815 Volume 140 becomes unwritable I0824 22:18:30.130186 volume_layout.go:335 volume 140 does not have enough copies I0824 22:18:30.130279 data_node.go:114 Deleting volume id: 141 I0824 22:18:30.130321 topology.go:534 removing volume info: Id:141, Size:1, ReplicaPlacement:000, Collection:c, Version:3, Ttl:, FileCount:0, DeleteCount:0, DeletedByteCount:0, ReadOnly:false, ModifiedAtSecond:0 from 10.0.0.1:8080 I0824 22:18:30.130351 volume_layout.go:815 Volume 141 becomes unwritable I0824 22:18:30.130370 volume_layout.go:335 volume 141 does not have enough copies I0824 22:18:30.130439 data_node.go:114 Deleting volume id: 142 I0824 22:18:30.130478 topology.go:534 removing volume info: Id:142, Size:1, ReplicaPlacement:000, Collection:c, Version:3, Ttl:, FileCount:0, DeleteCount:0, DeletedByteCount:0, ReadOnly:false, ModifiedAtSecond:0 from 10.0.0.1:8080 I0824 22:18:30.130528 volume_layout.go:815 Volume 142 becomes unwritable I0824 22:18:30.130547 volume_layout.go:335 volume 142 does not have enough copies I0824 22:18:30.130598 data_node.go:114 Deleting volume id: 143 I0824 22:18:30.130638 topology.go:534 removing volume info: Id:143, Size:1, ReplicaPlacement:000, Collection:c, Version:3, Ttl:, FileCount:0, DeleteCount:0, DeletedByteCount:0, ReadOnly:false, ModifiedAtSecond:0 from 10.0.0.1:8080 I0824 22:18:30.130667 volume_layout.go:815 Volume 143 becomes unwritable I0824 22:18:30.130685 volume_layout.go:335 volume 143 does not have enough copies I0824 22:18:30.130743 data_node.go:114 Deleting volume id: 144 I0824 22:18:30.130848 topology.go:534 removing volume info: Id:144, Size:1, ReplicaPlacement:000, Collection:c, Version:3, Ttl:, FileCount:0, DeleteCount:0, DeletedByteCount:0, ReadOnly:false, ModifiedAtSecond:0 from 10.0.0.1:8080 I0824 22:18:30.130917 volume_layout.go:815 Volume 144 becomes unwritable I0824 22:18:30.131001 volume_layout.go:335 volume 144 does not have enough copies I0824 22:18:30.131169 data_node.go:114 Deleting volume id: 145 I0824 22:18:30.131209 topology.go:534 removing volume info: Id:145, Size:1, ReplicaPlacement:000, Collection:c, Version:3, Ttl:, FileCount:0, DeleteCount:0, DeletedByteCount:0, ReadOnly:false, ModifiedAtSecond:0 from 10.0.0.1:8080 I0824 22:18:30.131298 volume_layout.go:815 Volume 145 becomes unwritable I0824 22:18:30.131320 volume_layout.go:335 volume 145 does not have enough copies I0824 22:18:30.131400 data_node.go:114 Deleting volume id: 146 I0824 22:18:30.131440 topology.go:534 removing volume info: Id:146, Size:1, ReplicaPlacement:000, Collection:c, Version:3, Ttl:, FileCount:0, DeleteCount:0, DeletedByteCount:0, ReadOnly:false, ModifiedAtSecond:0 from 10.0.0.1:8080 I0824 22:18:30.131469 volume_layout.go:815 Volume 146 becomes unwritable I0824 22:18:30.131488 volume_layout.go:335 volume 146 does not have enough copies I0824 22:18:30.131545 data_node.go:114 Deleting volume id: 147 I0824 22:18:30.131616 topology.go:534 removing volume info: Id:147, Size:1, ReplicaPlacement:000, Collection:c, Version:3, Ttl:, FileCount:0, DeleteCount:0, DeletedByteCount:0, ReadOnly:false, ModifiedAtSecond:0 from 10.0.0.1:8080 I0824 22:18:30.131645 volume_layout.go:815 Volume 147 becomes unwritable I0824 22:18:30.131672 volume_layout.go:335 volume 147 does not have enough copies I0824 22:18:30.131753 data_node.go:114 Deleting volume id: 148 I0824 22:18:30.131793 topology.go:534 removing volume info: Id:148, Size:1, ReplicaPlacement:000, Collection:c, Version:3, Ttl:, FileCount:0, DeleteCount:0, DeletedByteCount:0, ReadOnly:false, ModifiedAtSecond:0 from 10.0.0.1:8080 I0824 22:18:30.131824 volume_layout.go:815 Volume 148 becomes unwritable I0824 22:18:30.131842 volume_layout.go:335 volume 148 does not have enough copies I0824 22:18:30.131896 data_node.go:114 Deleting volume id: 149 I0824 22:18:30.131979 topology.go:534 removing volume info: Id:149, Size:1, ReplicaPlacement:000, Collection:c, Version:3, Ttl:, FileCount:0, DeleteCount:0, DeletedByteCount:0, ReadOnly:false, ModifiedAtSecond:0 from 10.0.0.1:8080 I0824 22:18:30.132010 volume_layout.go:815 Volume 149 becomes unwritable I0824 22:18:30.132028 volume_layout.go:335 volume 149 does not have enough copies I0824 22:18:30.132107 data_node.go:114 Deleting volume id: 100 I0824 22:18:30.132146 topology.go:534 removing volume info: Id:100, Size:1, ReplicaPlacement:000, Collection:c, Version:3, Ttl:, FileCount:0, DeleteCount:0, DeletedByteCount:0, ReadOnly:false, ModifiedAtSecond:0 from 10.0.0.1:8080 I0824 22:18:30.132208 volume_layout.go:815 Volume 100 becomes unwritable I0824 22:18:30.132315 volume_layout.go:335 volume 100 does not have enough copies I0824 22:18:30.132456 data_node.go:114 Deleting volume id: 101 I0824 22:18:30.132524 topology.go:534 removing volume info: Id:101, Size:1, ReplicaPlacement:000, Collection:c, Version:3, Ttl:, FileCount:0, DeleteCount:0, DeletedByteCount:0, ReadOnly:false, ModifiedAtSecond:0 from 10.0.0.1:8080 I0824 22:18:30.132554 volume_layout.go:815 Volume 101 becomes unwritable I0824 22:18:30.132573 volume_layout.go:335 volume 101 does not have enough copies I0824 22:18:30.132632 data_node.go:114 Deleting volume id: 102 I0824 22:18:30.132695 topology.go:534 removing volume info: Id:102, Size:1, ReplicaPlacement:000, Collection:c, Version:3, Ttl:, FileCount:0, DeleteCount:0, DeletedByteCount:0, ReadOnly:false, ModifiedAtSecond:0 from 10.0.0.1:8080 I0824 22:18:30.132726 volume_layout.go:815 Volume 102 becomes unwritable I0824 22:18:30.132744 volume_layout.go:335 volume 102 does not have enough copies I0824 22:18:30.132801 data_node.go:114 Deleting volume id: 103 I0824 22:18:30.132889 topology.go:534 removing volume info: Id:103, Size:1, ReplicaPlacement:000, Collection:c, Version:3, Ttl:, FileCount:0, DeleteCount:0, DeletedByteCount:0, ReadOnly:false, ModifiedAtSecond:0 from 10.0.0.1:8080 I0824 22:18:30.132934 volume_layout.go:815 Volume 103 becomes unwritable I0824 22:18:30.132974 volume_layout.go:335 volume 103 does not have enough copies I0824 22:18:30.133055 data_node.go:114 Deleting volume id: 104 I0824 22:18:30.133096 topology.go:534 removing volume info: Id:104, Size:1, ReplicaPlacement:000, Collection:c, Version:3, Ttl:, FileCount:0, DeleteCount:0, DeletedByteCount:0, ReadOnly:false, ModifiedAtSecond:0 from 10.0.0.1:8080 I0824 22:18:30.133126 volume_layout.go:815 Volume 104 becomes unwritable I0824 22:18:30.133145 volume_layout.go:335 volume 104 does not have enough copies I0824 22:18:30.133220 data_node.go:114 Deleting volume id: 105 I0824 22:18:30.133302 topology.go:534 removing volume info: Id:105, Size:1, ReplicaPlacement:000, Collection:c, Version:3, Ttl:, FileCount:0, DeleteCount:0, DeletedByteCount:0, ReadOnly:false, ModifiedAtSecond:0 from 10.0.0.1:8080 I0824 22:18:30.133333 volume_layout.go:815 Volume 105 becomes unwritable I0824 22:18:30.133352 volume_layout.go:335 volume 105 does not have enough copies I0824 22:18:30.133433 data_node.go:114 Deleting volume id: 106 I0824 22:18:30.133472 topology.go:534 removing volume info: Id:106, Size:1, ReplicaPlacement:000, Collection:c, Version:3, Ttl:, FileCount:0, DeleteCount:0, DeletedByteCount:0, ReadOnly:false, ModifiedAtSecond:0 from 10.0.0.1:8080 I0824 22:18:30.133520 volume_layout.go:815 Volume 106 becomes unwritable I0824 22:18:30.133691 volume_layout.go:335 volume 106 does not have enough copies I0824 22:18:30.133996 data_node.go:114 Deleting volume id: 107 I0824 22:18:30.134443 topology.go:534 removing volume info: Id:107, Size:1, ReplicaPlacement:000, Collection:c, Version:3, Ttl:, FileCount:0, DeleteCount:0, DeletedByteCount:0, ReadOnly:false, ModifiedAtSecond:0 from 10.0.0.1:8080 I0824 22:18:30.134725 volume_layout.go:815 Volume 107 becomes unwritable I0824 22:18:30.134902 volume_layout.go:335 volume 107 does not have enough copies I0824 22:18:30.135102 data_node.go:114 Deleting volume id: 108 I0824 22:18:30.135248 topology.go:534 removing volume info: Id:108, Size:1, ReplicaPlacement:000, Collection:c, Version:3, Ttl:, FileCount:0, DeleteCount:0, DeletedByteCount:0, ReadOnly:false, ModifiedAtSecond:0 from 10.0.0.1:8080 I0824 22:18:30.135369 volume_layout.go:815 Volume 108 becomes unwritable I0824 22:18:30.135409 volume_layout.go:335 volume 108 does not have enough copies I0824 22:18:30.135525 data_node.go:114 Deleting volume id: 109 I0824 22:18:30.135724 topology.go:534 removing volume info: Id:109, Size:1, ReplicaPlacement:000, Collection:c, Version:3, Ttl:, FileCount:0, DeleteCount:0, DeletedByteCount:0, ReadOnly:false, ModifiedAtSecond:0 from 10.0.0.1:8080 I0824 22:18:30.135800 volume_layout.go:815 Volume 109 becomes unwritable I0824 22:18:30.135871 volume_layout.go:335 volume 109 does not have enough copies I0824 22:18:30.135984 data_node.go:114 Deleting volume id: 110 I0824 22:18:30.136073 topology.go:534 removing volume info: Id:110, Size:1, ReplicaPlacement:000, Collection:c, Version:3, Ttl:, FileCount:0, DeleteCount:0, DeletedByteCount:0, ReadOnly:false, ModifiedAtSecond:0 from 10.0.0.1:8080 I0824 22:18:30.136130 volume_layout.go:815 Volume 110 becomes unwritable I0824 22:18:30.136187 volume_layout.go:335 volume 110 does not have enough copies I0824 22:18:30.136247 data_node.go:114 Deleting volume id: 111 I0824 22:18:30.136336 topology.go:534 removing volume info: Id:111, Size:1, ReplicaPlacement:000, Collection:c, Version:3, Ttl:, FileCount:0, DeleteCount:0, DeletedByteCount:0, ReadOnly:false, ModifiedAtSecond:0 from 10.0.0.1:8080 I0824 22:18:30.136571 volume_layout.go:815 Volume 111 becomes unwritable I0824 22:18:30.136626 volume_layout.go:335 volume 111 does not have enough copies I0824 22:18:30.136723 data_node.go:114 Deleting volume id: 112 I0824 22:18:30.136918 topology.go:534 removing volume info: Id:112, Size:1, ReplicaPlacement:000, Collection:c, Version:3, Ttl:, FileCount:0, DeleteCount:0, DeletedByteCount:0, ReadOnly:false, ModifiedAtSecond:0 from 10.0.0.1:8080 I0824 22:18:30.137126 volume_layout.go:815 Volume 112 becomes unwritable I0824 22:18:30.137175 volume_layout.go:335 volume 112 does not have enough copies I0824 22:18:30.137284 data_node.go:114 Deleting volume id: 113 I0824 22:18:30.137380 topology.go:534 removing volume info: Id:113, Size:1, ReplicaPlacement:000, Collection:c, Version:3, Ttl:, FileCount:0, DeleteCount:0, DeletedByteCount:0, ReadOnly:false, ModifiedAtSecond:0 from 10.0.0.1:8080 I0824 22:18:30.137472 volume_layout.go:815 Volume 113 becomes unwritable I0824 22:18:30.137500 volume_layout.go:335 volume 113 does not have enough copies I0824 22:18:30.137544 data_node.go:114 Deleting volume id: 114 I0824 22:18:30.137617 topology.go:534 removing volume info: Id:114, Size:1, ReplicaPlacement:000, Collection:c, Version:3, Ttl:, FileCount:0, DeleteCount:0, DeletedByteCount:0, ReadOnly:false, ModifiedAtSecond:0 from 10.0.0.1:8080 I0824 22:18:30.137652 volume_layout.go:815 Volume 114 becomes unwritable I0824 22:18:30.137672 volume_layout.go:335 volume 114 does not have enough copies I0824 22:18:30.137712 data_node.go:114 Deleting volume id: 115 I0824 22:18:30.137772 topology.go:534 removing volume info: Id:115, Size:1, ReplicaPlacement:000, Collection:c, Version:3, Ttl:, FileCount:0, DeleteCount:0, DeletedByteCount:0, ReadOnly:false, ModifiedAtSecond:0 from 10.0.0.1:8080 I0824 22:18:30.137806 volume_layout.go:815 Volume 115 becomes unwritable I0824 22:18:30.137883 volume_layout.go:335 volume 115 does not have enough copies I0824 22:18:30.138086 data_node.go:114 Deleting volume id: 116 I0824 22:18:30.138241 topology.go:534 removing volume info: Id:116, Size:1, ReplicaPlacement:000, Collection:c, Version:3, Ttl:, FileCount:0, DeleteCount:0, DeletedByteCount:0, ReadOnly:false, ModifiedAtSecond:0 from 10.0.0.1:8080 I0824 22:18:30.138288 volume_layout.go:815 Volume 116 becomes unwritable I0824 22:18:30.138310 volume_layout.go:335 volume 116 does not have enough copies I0824 22:18:30.138452 data_node.go:114 Deleting volume id: 117 I0824 22:18:30.138550 topology.go:534 removing volume info: Id:117, Size:1, ReplicaPlacement:000, Collection:c, Version:3, Ttl:, FileCount:0, DeleteCount:0, DeletedByteCount:0, ReadOnly:false, ModifiedAtSecond:0 from 10.0.0.1:8080 I0824 22:18:30.138608 volume_layout.go:815 Volume 117 becomes unwritable I0824 22:18:30.138631 volume_layout.go:335 volume 117 does not have enough copies I0824 22:18:30.138833 data_node.go:114 Deleting volume id: 118 I0824 22:18:30.138939 topology.go:534 removing volume info: Id:118, Size:1, ReplicaPlacement:000, Collection:c, Version:3, Ttl:, FileCount:0, DeleteCount:0, DeletedByteCount:0, ReadOnly:false, ModifiedAtSecond:0 from 10.0.0.1:8080 I0824 22:18:30.139065 volume_layout.go:815 Volume 118 becomes unwritable I0824 22:18:30.139092 volume_layout.go:335 volume 118 does not have enough copies I0824 22:18:30.139181 data_node.go:114 Deleting volume id: 119 I0824 22:18:30.139253 topology.go:534 removing volume info: Id:119, Size:1, ReplicaPlacement:000, Collection:c, Version:3, Ttl:, FileCount:0, DeleteCount:0, DeletedByteCount:0, ReadOnly:false, ModifiedAtSecond:0 from 10.0.0.1:8080 I0824 22:18:30.139319 volume_layout.go:815 Volume 119 becomes unwritable I0824 22:18:30.139342 volume_layout.go:335 volume 119 does not have enough copies I0824 22:18:30.139420 data_node.go:114 Deleting volume id: 120 I0824 22:18:30.139501 topology.go:534 removing volume info: Id:120, Size:1, ReplicaPlacement:000, Collection:c, Version:3, Ttl:, FileCount:0, DeleteCount:0, DeletedByteCount:0, ReadOnly:false, ModifiedAtSecond:0 from 10.0.0.1:8080 I0824 22:18:30.139536 volume_layout.go:815 Volume 120 becomes unwritable I0824 22:18:30.139590 volume_layout.go:335 volume 120 does not have enough copies I0824 22:18:30.139656 data_node.go:114 Deleting volume id: 121 I0824 22:18:30.139711 topology.go:534 removing volume info: Id:121, Size:1, ReplicaPlacement:000, Collection:c, Version:3, Ttl:, FileCount:0, DeleteCount:0, DeletedByteCount:0, ReadOnly:false, ModifiedAtSecond:0 from 10.0.0.1:8080 I0824 22:18:30.139862 volume_layout.go:815 Volume 121 becomes unwritable I0824 22:18:30.139894 volume_layout.go:335 volume 121 does not have enough copies I0824 22:18:30.140094 data_node.go:114 Deleting volume id: 122 I0824 22:18:30.140197 topology.go:534 removing volume info: Id:122, Size:1, ReplicaPlacement:000, Collection:c, Version:3, Ttl:, FileCount:0, DeleteCount:0, DeletedByteCount:0, ReadOnly:false, ModifiedAtSecond:0 from 10.0.0.1:8080 I0824 22:18:30.140242 volume_layout.go:815 Volume 122 becomes unwritable I0824 22:18:30.140266 volume_layout.go:335 volume 122 does not have enough copies I0824 22:18:30.140388 data_node.go:114 Deleting volume id: 123 I0824 22:18:30.140465 topology.go:534 removing volume info: Id:123, Size:1, ReplicaPlacement:000, Collection:c, Version:3, Ttl:, FileCount:0, DeleteCount:0, DeletedByteCount:0, ReadOnly:false, ModifiedAtSecond:0 from 10.0.0.1:8080 I0824 22:18:30.140503 volume_layout.go:815 Volume 123 becomes unwritable I0824 22:18:30.140527 volume_layout.go:335 volume 123 does not have enough copies I0824 22:18:30.140619 data_node.go:114 Deleting volume id: 124 I0824 22:18:30.140786 topology.go:534 removing volume info: Id:124, Size:1, ReplicaPlacement:000, Collection:c, Version:3, Ttl:, FileCount:0, DeleteCount:0, DeletedByteCount:0, ReadOnly:false, ModifiedAtSecond:0 from 10.0.0.1:8080 I0824 22:18:30.140869 volume_layout.go:815 Volume 124 becomes unwritable I0824 22:18:30.140891 volume_layout.go:335 volume 124 does not have enough copies I0824 22:18:30.140956 data_node.go:114 Deleting volume id: 125 I0824 22:18:30.141030 topology.go:534 removing volume info: Id:125, Size:1, ReplicaPlacement:000, Collection:c, Version:3, Ttl:, FileCount:0, DeleteCount:0, DeletedByteCount:0, ReadOnly:false, ModifiedAtSecond:0 from 10.0.0.1:8080 I0824 22:18:30.141110 volume_layout.go:815 Volume 125 becomes unwritable I0824 22:18:30.141180 volume_layout.go:335 volume 125 does not have enough copies I0824 22:18:30.141393 data_node.go:114 Deleting volume id: 126 I0824 22:18:30.141564 topology.go:534 removing volume info: Id:126, Size:1, ReplicaPlacement:000, Collection:c, Version:3, Ttl:, FileCount:0, DeleteCount:0, DeletedByteCount:0, ReadOnly:false, ModifiedAtSecond:0 from 10.0.0.1:8080 I0824 22:18:30.141640 volume_layout.go:815 Volume 126 becomes unwritable I0824 22:18:30.141688 volume_layout.go:335 volume 126 does not have enough copies I0824 22:18:30.141798 data_node.go:114 Deleting volume id: 127 I0824 22:18:30.141891 topology.go:534 removing volume info: Id:127, Size:1, ReplicaPlacement:000, Collection:c, Version:3, Ttl:, FileCount:0, DeleteCount:0, DeletedByteCount:0, ReadOnly:false, ModifiedAtSecond:0 from 10.0.0.1:8080 I0824 22:18:30.142086 volume_layout.go:815 Volume 127 becomes unwritable I0824 22:18:30.142130 volume_layout.go:335 volume 127 does not have enough copies I0824 22:18:30.142257 data_node.go:114 Deleting volume id: 128 I0824 22:18:30.142376 topology.go:534 removing volume info: Id:128, Size:1, ReplicaPlacement:000, Collection:c, Version:3, Ttl:, FileCount:0, DeleteCount:0, DeletedByteCount:0, ReadOnly:false, ModifiedAtSecond:0 from 10.0.0.1:8080 I0824 22:18:30.142425 volume_layout.go:815 Volume 128 becomes unwritable I0824 22:18:30.142452 volume_layout.go:335 volume 128 does not have enough copies I0824 22:18:30.142600 data_node.go:114 Deleting volume id: 129 I0824 22:18:30.142695 topology.go:534 removing volume info: Id:129, Size:1, ReplicaPlacement:000, Collection:c, Version:3, Ttl:, FileCount:0, DeleteCount:0, DeletedByteCount:0, ReadOnly:false, ModifiedAtSecond:0 from 10.0.0.1:8080 I0824 22:18:30.142735 volume_layout.go:815 Volume 129 becomes unwritable I0824 22:18:30.142755 volume_layout.go:335 volume 129 does not have enough copies I0824 22:18:30.142956 data_node.go:114 Deleting volume id: 130 I0824 22:18:30.143082 topology.go:534 removing volume info: Id:130, Size:1, ReplicaPlacement:000, Collection:c, Version:3, Ttl:, FileCount:0, DeleteCount:0, DeletedByteCount:0, ReadOnly:false, ModifiedAtSecond:0 from 10.0.0.1:8080 I0824 22:18:30.143146 volume_layout.go:815 Volume 130 becomes unwritable I0824 22:18:30.143167 volume_layout.go:335 volume 130 does not have enough copies I0824 22:18:30.143339 data_node.go:114 Deleting volume id: 131 I0824 22:18:30.143544 topology.go:534 removing volume info: Id:131, Size:1, ReplicaPlacement:000, Collection:c, Version:3, Ttl:, FileCount:0, DeleteCount:0, DeletedByteCount:0, ReadOnly:false, ModifiedAtSecond:0 from 10.0.0.1:8080 I0824 22:18:30.143605 volume_layout.go:815 Volume 131 becomes unwritable I0824 22:18:30.143635 volume_layout.go:335 volume 131 does not have enough copies I0824 22:18:30.143742 data_node.go:114 Deleting volume id: 132 I0824 22:18:30.143811 topology.go:534 removing volume info: Id:132, Size:1, ReplicaPlacement:000, Collection:c, Version:3, Ttl:, FileCount:0, DeleteCount:0, DeletedByteCount:0, ReadOnly:false, ModifiedAtSecond:0 from 10.0.0.1:8080 I0824 22:18:30.143850 volume_layout.go:815 Volume 132 becomes unwritable I0824 22:18:30.143877 volume_layout.go:335 volume 132 does not have enough copies I0824 22:18:30.144109 data_node.go:114 Deleting volume id: 133 I0824 22:18:30.144440 topology.go:534 removing volume info: Id:133, Size:1, ReplicaPlacement:000, Collection:c, Version:3, Ttl:, FileCount:0, DeleteCount:0, DeletedByteCount:0, ReadOnly:false, ModifiedAtSecond:0 from 10.0.0.1:8080 I0824 22:18:30.144868 volume_layout.go:815 Volume 133 becomes unwritable I0824 22:18:30.144988 volume_layout.go:335 volume 133 does not have enough copies I0824 22:18:30.145135 data_node.go:114 Deleting volume id: 134 I0824 22:18:30.145196 topology.go:534 removing volume info: Id:134, Size:1, ReplicaPlacement:000, Collection:c, Version:3, Ttl:, FileCount:0, DeleteCount:0, DeletedByteCount:0, ReadOnly:false, ModifiedAtSecond:0 from 10.0.0.1:8080 I0824 22:18:30.146450 volume_layout.go:815 Volume 134 becomes unwritable I0824 22:18:30.146568 volume_layout.go:335 volume 134 does not have enough copies I0824 22:18:30.146732 data_node.go:114 Deleting volume id: 135 I0824 22:18:30.146796 topology.go:534 removing volume info: Id:135, Size:1, ReplicaPlacement:000, Collection:c, Version:3, Ttl:, FileCount:0, DeleteCount:0, DeletedByteCount:0, ReadOnly:false, ModifiedAtSecond:0 from 10.0.0.1:8080 I0824 22:18:30.146866 volume_layout.go:815 Volume 135 becomes unwritable I0824 22:18:30.146893 volume_layout.go:335 volume 135 does not have enough copies I0824 22:18:30.146982 data_node.go:114 Deleting volume id: 136 I0824 22:18:30.147028 topology.go:534 removing volume info: Id:136, Size:1, ReplicaPlacement:000, Collection:c, Version:3, Ttl:, FileCount:0, DeleteCount:0, DeletedByteCount:0, ReadOnly:false, ModifiedAtSecond:0 from 10.0.0.1:8080 I0824 22:18:30.147062 volume_layout.go:815 Volume 136 becomes unwritable I0824 22:18:30.147082 volume_layout.go:335 volume 136 does not have enough copies I0824 22:18:30.147138 data_node.go:114 Deleting volume id: 137 I0824 22:18:30.147180 topology.go:534 removing volume info: Id:137, Size:1, ReplicaPlacement:000, Collection:c, Version:3, Ttl:, FileCount:0, DeleteCount:0, DeletedByteCount:0, ReadOnly:false, ModifiedAtSecond:0 from 10.0.0.1:8080 I0824 22:18:30.147231 volume_layout.go:815 Volume 137 becomes unwritable I0824 22:18:30.147250 volume_layout.go:335 volume 137 does not have enough copies I0824 22:18:30.147378 data_node.go:114 Deleting volume id: 138 I0824 22:18:30.147438 topology.go:534 removing volume info: Id:138, Size:1, ReplicaPlacement:000, Collection:c, Version:3, Ttl:, FileCount:0, DeleteCount:0, DeletedByteCount:0, ReadOnly:false, ModifiedAtSecond:0 from 10.0.0.1:8080 I0824 22:18:30.147622 volume_layout.go:815 Volume 138 becomes unwritable I0824 22:18:30.147651 volume_layout.go:335 volume 138 does not have enough copies I0824 22:18:30.147741 data_node.go:114 Deleting volume id: 139 I0824 22:18:30.147788 topology.go:534 removing volume info: Id:139, Size:1, ReplicaPlacement:000, Collection:c, Version:3, Ttl:, FileCount:0, DeleteCount:0, DeletedByteCount:0, ReadOnly:false, ModifiedAtSecond:0 from 10.0.0.1:8080 I0824 22:18:30.147821 volume_layout.go:815 Volume 139 becomes unwritable I0824 22:18:30.147840 volume_layout.go:335 volume 139 does not have enough copies I0824 22:18:30.148013 data_node.go:114 Deleting volume id: 140 I0824 22:18:30.148110 topology.go:534 removing volume info: Id:140, Size:1, ReplicaPlacement:000, Collection:c, Version:3, Ttl:, FileCount:0, DeleteCount:0, DeletedByteCount:0, ReadOnly:false, ModifiedAtSecond:0 from 10.0.0.1:8080 I0824 22:18:30.148164 volume_layout.go:815 Volume 140 becomes unwritable I0824 22:18:30.148184 volume_layout.go:335 volume 140 does not have enough copies I0824 22:18:30.148258 data_node.go:114 Deleting volume id: 141 I0824 22:18:30.148315 topology.go:534 removing volume info: Id:141, Size:1, ReplicaPlacement:000, Collection:c, Version:3, Ttl:, FileCount:0, DeleteCount:0, DeletedByteCount:0, ReadOnly:false, ModifiedAtSecond:0 from 10.0.0.1:8080 I0824 22:18:30.148346 volume_layout.go:815 Volume 141 becomes unwritable I0824 22:18:30.148365 volume_layout.go:335 volume 141 does not have enough copies I0824 22:18:30.148447 data_node.go:114 Deleting volume id: 142 I0824 22:18:30.148496 topology.go:534 removing volume info: Id:142, Size:1, ReplicaPlacement:000, Collection:c, Version:3, Ttl:, FileCount:0, DeleteCount:0, DeletedByteCount:0, ReadOnly:false, ModifiedAtSecond:0 from 10.0.0.1:8080 I0824 22:18:30.148531 volume_layout.go:815 Volume 142 becomes unwritable I0824 22:18:30.148553 volume_layout.go:335 volume 142 does not have enough copies I0824 22:18:30.148640 data_node.go:114 Deleting volume id: 143 I0824 22:18:30.148702 topology.go:534 removing volume info: Id:143, Size:1, ReplicaPlacement:000, Collection:c, Version:3, Ttl:, FileCount:0, DeleteCount:0, DeletedByteCount:0, ReadOnly:false, ModifiedAtSecond:0 from 10.0.0.1:8080 I0824 22:18:30.149695 volume_layout.go:815 Volume 143 becomes unwritable I0824 22:18:30.149752 volume_layout.go:335 volume 143 does not have enough copies I0824 22:18:30.149901 data_node.go:114 Deleting volume id: 144 I0824 22:18:30.150018 topology.go:534 removing volume info: Id:144, Size:1, ReplicaPlacement:000, Collection:c, Version:3, Ttl:, FileCount:0, DeleteCount:0, DeletedByteCount:0, ReadOnly:false, ModifiedAtSecond:0 from 10.0.0.1:8080 I0824 22:18:30.150104 volume_layout.go:815 Volume 144 becomes unwritable I0824 22:18:30.150125 volume_layout.go:335 volume 144 does not have enough copies I0824 22:18:30.150208 data_node.go:114 Deleting volume id: 145 I0824 22:18:30.150256 topology.go:534 removing volume info: Id:145, Size:1, ReplicaPlacement:000, Collection:c, Version:3, Ttl:, FileCount:0, DeleteCount:0, DeletedByteCount:0, ReadOnly:false, ModifiedAtSecond:0 from 10.0.0.1:8080 I0824 22:18:30.150305 volume_layout.go:815 Volume 145 becomes unwritable I0824 22:18:30.150333 volume_layout.go:335 volume 145 does not have enough copies I0824 22:18:30.150434 data_node.go:114 Deleting volume id: 146 I0824 22:18:30.150487 topology.go:534 removing volume info: Id:146, Size:1, ReplicaPlacement:000, Collection:c, Version:3, Ttl:, FileCount:0, DeleteCount:0, DeletedByteCount:0, ReadOnly:false, ModifiedAtSecond:0 from 10.0.0.1:8080 I0824 22:18:30.150640 volume_layout.go:815 Volume 146 becomes unwritable I0824 22:18:30.150670 volume_layout.go:335 volume 146 does not have enough copies I0824 22:18:30.150766 data_node.go:114 Deleting volume id: 147 I0824 22:18:30.150879 topology.go:534 removing volume info: Id:147, Size:1, ReplicaPlacement:000, Collection:c, Version:3, Ttl:, FileCount:0, DeleteCount:0, DeletedByteCount:0, ReadOnly:false, ModifiedAtSecond:0 from 10.0.0.1:8080 I0824 22:18:30.150982 volume_layout.go:815 Volume 147 becomes unwritable I0824 22:18:30.151006 volume_layout.go:335 volume 147 does not have enough copies I0824 22:18:30.151093 data_node.go:114 Deleting volume id: 148 I0824 22:18:30.151140 topology.go:534 removing volume info: Id:148, Size:1, ReplicaPlacement:000, Collection:c, Version:3, Ttl:, FileCount:0, DeleteCount:0, DeletedByteCount:0, ReadOnly:false, ModifiedAtSecond:0 from 10.0.0.1:8080 I0824 22:18:30.151172 volume_layout.go:815 Volume 148 becomes unwritable I0824 22:18:30.151228 volume_layout.go:335 volume 148 does not have enough copies I0824 22:18:30.151380 data_node.go:114 Deleting volume id: 149 I0824 22:18:30.151429 topology.go:534 removing volume info: Id:149, Size:1, ReplicaPlacement:000, Collection:c, Version:3, Ttl:, FileCount:0, DeleteCount:0, DeletedByteCount:0, ReadOnly:false, ModifiedAtSecond:0 from 10.0.0.1:8080 I0824 22:18:30.151461 volume_layout.go:815 Volume 149 becomes unwritable I0824 22:18:30.151482 volume_layout.go:335 volume 149 does not have enough copies I0824 22:18:30.151603 data_node.go:114 Deleting volume id: 100 I0824 22:18:30.151655 topology.go:534 removing volume info: Id:100, Size:1, ReplicaPlacement:000, Collection:c, Version:3, Ttl:, FileCount:0, DeleteCount:0, DeletedByteCount:0, ReadOnly:false, ModifiedAtSecond:0 from 10.0.0.1:8080 I0824 22:18:30.151704 volume_layout.go:815 Volume 100 becomes unwritable I0824 22:18:30.151756 volume_layout.go:335 volume 100 does not have enough copies I0824 22:18:30.151843 data_node.go:114 Deleting volume id: 101 I0824 22:18:30.151930 topology.go:534 removing volume info: Id:101, Size:1, ReplicaPlacement:000, Collection:c, Version:3, Ttl:, FileCount:0, DeleteCount:0, DeletedByteCount:0, ReadOnly:false, ModifiedAtSecond:0 from 10.0.0.1:8080 I0824 22:18:30.151963 volume_layout.go:815 Volume 101 becomes unwritable I0824 22:18:30.151981 volume_layout.go:335 volume 101 does not have enough copies I0824 22:18:30.152093 data_node.go:114 Deleting volume id: 102 I0824 22:18:30.152152 topology.go:534 removing volume info: Id:102, Size:1, ReplicaPlacement:000, Collection:c, Version:3, Ttl:, FileCount:0, DeleteCount:0, DeletedByteCount:0, ReadOnly:false, ModifiedAtSecond:0 from 10.0.0.1:8080 I0824 22:18:30.153490 volume_layout.go:815 Volume 102 becomes unwritable I0824 22:18:30.153547 volume_layout.go:335 volume 102 does not have enough copies I0824 22:18:30.153697 data_node.go:114 Deleting volume id: 103 I0824 22:18:30.153746 topology.go:534 removing volume info: Id:103, Size:1, ReplicaPlacement:000, Collection:c, Version:3, Ttl:, FileCount:0, DeleteCount:0, DeletedByteCount:0, ReadOnly:false, ModifiedAtSecond:0 from 10.0.0.1:8080 I0824 22:18:30.153783 volume_layout.go:815 Volume 103 becomes unwritable I0824 22:18:30.153803 volume_layout.go:335 volume 103 does not have enough copies I0824 22:18:30.154142 data_node.go:114 Deleting volume id: 104 I0824 22:18:30.154286 topology.go:534 removing volume info: Id:104, Size:1, ReplicaPlacement:000, Collection:c, Version:3, Ttl:, FileCount:0, DeleteCount:0, DeletedByteCount:0, ReadOnly:false, ModifiedAtSecond:0 from 10.0.0.1:8080 I0824 22:18:30.154457 volume_layout.go:815 Volume 104 becomes unwritable I0824 22:18:30.154517 volume_layout.go:335 volume 104 does not have enough copies I0824 22:18:30.154656 data_node.go:114 Deleting volume id: 105 I0824 22:18:30.154753 topology.go:534 removing volume info: Id:105, Size:1, ReplicaPlacement:000, Collection:c, Version:3, Ttl:, FileCount:0, DeleteCount:0, DeletedByteCount:0, ReadOnly:false, ModifiedAtSecond:0 from 10.0.0.1:8080 I0824 22:18:30.154792 volume_layout.go:815 Volume 105 becomes unwritable I0824 22:18:30.154814 volume_layout.go:335 volume 105 does not have enough copies I0824 22:18:30.154937 data_node.go:114 Deleting volume id: 106 I0824 22:18:30.155025 topology.go:534 removing volume info: Id:106, Size:1, ReplicaPlacement:000, Collection:c, Version:3, Ttl:, FileCount:0, DeleteCount:0, DeletedByteCount:0, ReadOnly:false, ModifiedAtSecond:0 from 10.0.0.1:8080 I0824 22:18:30.155062 volume_layout.go:815 Volume 106 becomes unwritable I0824 22:18:30.155103 volume_layout.go:335 volume 106 does not have enough copies I0824 22:18:30.155197 data_node.go:114 Deleting volume id: 107 I0824 22:18:30.155282 topology.go:534 removing volume info: Id:107, Size:1, ReplicaPlacement:000, Collection:c, Version:3, Ttl:, FileCount:0, DeleteCount:0, DeletedByteCount:0, ReadOnly:false, ModifiedAtSecond:0 from 10.0.0.1:8080 I0824 22:18:30.155360 volume_layout.go:815 Volume 107 becomes unwritable I0824 22:18:30.155404 volume_layout.go:335 volume 107 does not have enough copies I0824 22:18:30.155500 data_node.go:114 Deleting volume id: 108 I0824 22:18:30.155594 topology.go:534 removing volume info: Id:108, Size:1, ReplicaPlacement:000, Collection:c, Version:3, Ttl:, FileCount:0, DeleteCount:0, DeletedByteCount:0, ReadOnly:false, ModifiedAtSecond:0 from 10.0.0.1:8080 I0824 22:18:30.155642 volume_layout.go:815 Volume 108 becomes unwritable I0824 22:18:30.155661 volume_layout.go:335 volume 108 does not have enough copies I0824 22:18:30.155751 data_node.go:114 Deleting volume id: 109 I0824 22:18:30.155793 topology.go:534 removing volume info: Id:109, Size:1, ReplicaPlacement:000, Collection:c, Version:3, Ttl:, FileCount:0, DeleteCount:0, DeletedByteCount:0, ReadOnly:false, ModifiedAtSecond:0 from 10.0.0.1:8080 I0824 22:18:30.155824 volume_layout.go:815 Volume 109 becomes unwritable I0824 22:18:30.155843 volume_layout.go:335 volume 109 does not have enough copies I0824 22:18:30.155909 data_node.go:114 Deleting volume id: 110 I0824 22:18:30.155952 topology.go:534 removing volume info: Id:110, Size:1, ReplicaPlacement:000, Collection:c, Version:3, Ttl:, FileCount:0, DeleteCount:0, DeletedByteCount:0, ReadOnly:false, ModifiedAtSecond:0 from 10.0.0.1:8080 I0824 22:18:30.156032 volume_layout.go:815 Volume 110 becomes unwritable I0824 22:18:30.156053 volume_layout.go:335 volume 110 does not have enough copies I0824 22:18:30.156170 data_node.go:114 Deleting volume id: 111 I0824 22:18:30.156252 topology.go:534 removing volume info: Id:111, Size:1, ReplicaPlacement:000, Collection:c, Version:3, Ttl:, FileCount:0, DeleteCount:0, DeletedByteCount:0, ReadOnly:false, ModifiedAtSecond:0 from 10.0.0.1:8080 I0824 22:18:30.156318 volume_layout.go:815 Volume 111 becomes unwritable I0824 22:18:30.156339 volume_layout.go:335 volume 111 does not have enough copies I0824 22:18:30.156542 data_node.go:114 Deleting volume id: 112 I0824 22:18:30.156666 topology.go:534 removing volume info: Id:112, Size:1, ReplicaPlacement:000, Collection:c, Version:3, Ttl:, FileCount:0, DeleteCount:0, DeletedByteCount:0, ReadOnly:false, ModifiedAtSecond:0 from 10.0.0.1:8080 I0824 22:18:30.156699 volume_layout.go:815 Volume 112 becomes unwritable I0824 22:18:30.156719 volume_layout.go:335 volume 112 does not have enough copies I0824 22:18:30.156772 data_node.go:114 Deleting volume id: 113 I0824 22:18:30.156813 topology.go:534 removing volume info: Id:113, Size:1, ReplicaPlacement:000, Collection:c, Version:3, Ttl:, FileCount:0, DeleteCount:0, DeletedByteCount:0, ReadOnly:false, ModifiedAtSecond:0 from 10.0.0.1:8080 I0824 22:18:30.156843 volume_layout.go:815 Volume 113 becomes unwritable I0824 22:18:30.156861 volume_layout.go:335 volume 113 does not have enough copies I0824 22:18:30.156978 data_node.go:114 Deleting volume id: 114 I0824 22:18:30.157067 topology.go:534 removing volume info: Id:114, Size:1, ReplicaPlacement:000, Collection:c, Version:3, Ttl:, FileCount:0, DeleteCount:0, DeletedByteCount:0, ReadOnly:false, ModifiedAtSecond:0 from 10.0.0.1:8080 I0824 22:18:30.157104 volume_layout.go:815 Volume 114 becomes unwritable I0824 22:18:30.157128 volume_layout.go:335 volume 114 does not have enough copies I0824 22:18:30.157196 data_node.go:114 Deleting volume id: 115 I0824 22:18:30.157272 topology.go:534 removing volume info: Id:115, Size:1, ReplicaPlacement:000, Collection:c, Version:3, Ttl:, FileCount:0, DeleteCount:0, DeletedByteCount:0, ReadOnly:false, ModifiedAtSecond:0 from 10.0.0.1:8080 I0824 22:18:30.157334 volume_layout.go:815 Volume 115 becomes unwritable I0824 22:18:30.157388 volume_layout.go:335 volume 115 does not have enough copies I0824 22:18:30.157463 data_node.go:114 Deleting volume id: 116 I0824 22:18:30.157507 topology.go:534 removing volume info: Id:116, Size:1, ReplicaPlacement:000, Collection:c, Version:3, Ttl:, FileCount:0, DeleteCount:0, DeletedByteCount:0, ReadOnly:false, ModifiedAtSecond:0 from 10.0.0.1:8080 I0824 22:18:30.157537 volume_layout.go:815 Volume 116 becomes unwritable I0824 22:18:30.157555 volume_layout.go:335 volume 116 does not have enough copies I0824 22:18:30.157608 data_node.go:114 Deleting volume id: 117 I0824 22:18:30.157679 topology.go:534 removing volume info: Id:117, Size:1, ReplicaPlacement:000, Collection:c, Version:3, Ttl:, FileCount:0, DeleteCount:0, DeletedByteCount:0, ReadOnly:false, ModifiedAtSecond:0 from 10.0.0.1:8080 I0824 22:18:30.157710 volume_layout.go:815 Volume 117 becomes unwritable I0824 22:18:30.157731 volume_layout.go:335 volume 117 does not have enough copies I0824 22:18:30.157877 data_node.go:114 Deleting volume id: 118 I0824 22:18:30.157923 topology.go:534 removing volume info: Id:118, Size:1, ReplicaPlacement:000, Collection:c, Version:3, Ttl:, FileCount:0, DeleteCount:0, DeletedByteCount:0, ReadOnly:false, ModifiedAtSecond:0 from 10.0.0.1:8080 I0824 22:18:30.158008 volume_layout.go:815 Volume 118 becomes unwritable I0824 22:18:30.158027 volume_layout.go:335 volume 118 does not have enough copies I0824 22:18:30.158081 data_node.go:114 Deleting volume id: 119 I0824 22:18:30.158129 topology.go:534 removing volume info: Id:119, Size:1, ReplicaPlacement:000, Collection:c, Version:3, Ttl:, FileCount:0, DeleteCount:0, DeletedByteCount:0, ReadOnly:false, ModifiedAtSecond:0 from 10.0.0.1:8080 I0824 22:18:30.158176 volume_layout.go:815 Volume 119 becomes unwritable I0824 22:18:30.158256 volume_layout.go:335 volume 119 does not have enough copies I0824 22:18:30.158348 data_node.go:114 Deleting volume id: 120 I0824 22:18:30.158408 topology.go:534 removing volume info: Id:120, Size:1, ReplicaPlacement:000, Collection:c, Version:3, Ttl:, FileCount:0, DeleteCount:0, DeletedByteCount:0, ReadOnly:false, ModifiedAtSecond:0 from 10.0.0.1:8080 I0824 22:18:30.158494 volume_layout.go:815 Volume 120 becomes unwritable I0824 22:18:30.158523 volume_layout.go:335 volume 120 does not have enough copies I0824 22:18:30.158597 data_node.go:114 Deleting volume id: 121 I0824 22:18:30.158675 topology.go:534 removing volume info: Id:121, Size:1, ReplicaPlacement:000, Collection:c, Version:3, Ttl:, FileCount:0, DeleteCount:0, DeletedByteCount:0, ReadOnly:false, ModifiedAtSecond:0 from 10.0.0.1:8080 I0824 22:18:30.158708 volume_layout.go:815 Volume 121 becomes unwritable I0824 22:18:30.158727 volume_layout.go:335 volume 121 does not have enough copies I0824 22:18:30.158796 data_node.go:114 Deleting volume id: 122 I0824 22:18:30.158838 topology.go:534 removing volume info: Id:122, Size:1, ReplicaPlacement:000, Collection:c, Version:3, Ttl:, FileCount:0, DeleteCount:0, DeletedByteCount:0, ReadOnly:false, ModifiedAtSecond:0 from 10.0.0.1:8080 I0824 22:18:30.158876 volume_layout.go:815 Volume 122 becomes unwritable I0824 22:18:30.158895 volume_layout.go:335 volume 122 does not have enough copies I0824 22:18:30.158973 data_node.go:114 Deleting volume id: 123 I0824 22:18:30.159110 topology.go:534 removing volume info: Id:123, Size:1, ReplicaPlacement:000, Collection:c, Version:3, Ttl:, FileCount:0, DeleteCount:0, DeletedByteCount:0, ReadOnly:false, ModifiedAtSecond:0 from 10.0.0.1:8080 I0824 22:18:30.159236 volume_layout.go:815 Volume 123 becomes unwritable I0824 22:18:30.159259 volume_layout.go:335 volume 123 does not have enough copies I0824 22:18:30.159323 data_node.go:114 Deleting volume id: 124 I0824 22:18:30.159364 topology.go:534 removing volume info: Id:124, Size:1, ReplicaPlacement:000, Collection:c, Version:3, Ttl:, FileCount:0, DeleteCount:0, DeletedByteCount:0, ReadOnly:false, ModifiedAtSecond:0 from 10.0.0.1:8080 I0824 22:18:30.159395 volume_layout.go:815 Volume 124 becomes unwritable I0824 22:18:30.159414 volume_layout.go:335 volume 124 does not have enough copies I0824 22:18:30.159468 data_node.go:114 Deleting volume id: 125 I0824 22:18:30.159535 topology.go:534 removing volume info: Id:125, Size:1, ReplicaPlacement:000, Collection:c, Version:3, Ttl:, FileCount:0, DeleteCount:0, DeletedByteCount:0, ReadOnly:false, ModifiedAtSecond:0 from 10.0.0.1:8080 I0824 22:18:30.159694 volume_layout.go:815 Volume 125 becomes unwritable I0824 22:18:30.159715 volume_layout.go:335 volume 125 does not have enough copies I0824 22:18:30.159786 data_node.go:114 Deleting volume id: 126 I0824 22:18:30.159882 topology.go:534 removing volume info: Id:126, Size:1, ReplicaPlacement:000, Collection:c, Version:3, Ttl:, FileCount:0, DeleteCount:0, DeletedByteCount:0, ReadOnly:false, ModifiedAtSecond:0 from 10.0.0.1:8080 I0824 22:18:30.159913 volume_layout.go:815 Volume 126 becomes unwritable I0824 22:18:30.159933 volume_layout.go:335 volume 126 does not have enough copies I0824 22:18:30.159999 data_node.go:114 Deleting volume id: 127 I0824 22:18:30.160038 topology.go:534 removing volume info: Id:127, Size:1, ReplicaPlacement:000, Collection:c, Version:3, Ttl:, FileCount:0, DeleteCount:0, DeletedByteCount:0, ReadOnly:false, ModifiedAtSecond:0 from 10.0.0.1:8080 I0824 22:18:30.160069 volume_layout.go:815 Volume 127 becomes unwritable I0824 22:18:30.160088 volume_layout.go:335 volume 127 does not have enough copies I0824 22:18:30.160224 data_node.go:114 Deleting volume id: 128 I0824 22:18:30.160315 topology.go:534 removing volume info: Id:128, Size:1, ReplicaPlacement:000, Collection:c, Version:3, Ttl:, FileCount:0, DeleteCount:0, DeletedByteCount:0, ReadOnly:false, ModifiedAtSecond:0 from 10.0.0.1:8080 I0824 22:18:30.160349 volume_layout.go:815 Volume 128 becomes unwritable I0824 22:18:30.160367 volume_layout.go:335 volume 128 does not have enough copies I0824 22:18:30.160420 data_node.go:114 Deleting volume id: 129 I0824 22:18:30.160466 topology.go:534 removing volume info: Id:129, Size:1, ReplicaPlacement:000, Collection:c, Version:3, Ttl:, FileCount:0, DeleteCount:0, DeletedByteCount:0, ReadOnly:false, ModifiedAtSecond:0 from 10.0.0.1:8080 I0824 22:18:30.160509 volume_layout.go:815 Volume 129 becomes unwritable I0824 22:18:30.160555 volume_layout.go:335 volume 129 does not have enough copies I0824 22:18:30.160670 data_node.go:114 Deleting volume id: 130 I0824 22:18:30.160724 topology.go:534 removing volume info: Id:130, Size:1, ReplicaPlacement:000, Collection:c, Version:3, Ttl:, FileCount:0, DeleteCount:0, DeletedByteCount:0, ReadOnly:false, ModifiedAtSecond:0 from 10.0.0.1:8080 I0824 22:18:30.160767 volume_layout.go:815 Volume 130 becomes unwritable I0824 22:18:30.160791 volume_layout.go:335 volume 130 does not have enough copies I0824 22:18:30.160846 data_node.go:114 Deleting volume id: 131 I0824 22:18:30.160919 topology.go:534 removing volume info: Id:131, Size:1, ReplicaPlacement:000, Collection:c, Version:3, Ttl:, FileCount:0, DeleteCount:0, DeletedByteCount:0, ReadOnly:false, ModifiedAtSecond:0 from 10.0.0.1:8080 I0824 22:18:30.161070 volume_layout.go:815 Volume 131 becomes unwritable I0824 22:18:30.161172 volume_layout.go:335 volume 131 does not have enough copies I0824 22:18:30.161255 data_node.go:114 Deleting volume id: 132 I0824 22:18:30.161301 topology.go:534 removing volume info: Id:132, Size:1, ReplicaPlacement:000, Collection:c, Version:3, Ttl:, FileCount:0, DeleteCount:0, DeletedByteCount:0, ReadOnly:false, ModifiedAtSecond:0 from 10.0.0.1:8080 I0824 22:18:30.161332 volume_layout.go:815 Volume 132 becomes unwritable I0824 22:18:30.161351 volume_layout.go:335 volume 132 does not have enough copies I0824 22:18:30.161403 data_node.go:114 Deleting volume id: 133 I0824 22:18:30.161548 topology.go:534 removing volume info: Id:133, Size:1, ReplicaPlacement:000, Collection:c, Version:3, Ttl:, FileCount:0, DeleteCount:0, DeletedByteCount:0, ReadOnly:false, ModifiedAtSecond:0 from 10.0.0.1:8080 I0824 22:18:30.161580 volume_layout.go:815 Volume 133 becomes unwritable I0824 22:18:30.161642 volume_layout.go:335 volume 133 does not have enough copies I0824 22:18:30.161705 data_node.go:114 Deleting volume id: 134 I0824 22:18:30.161781 topology.go:534 removing volume info: Id:134, Size:1, ReplicaPlacement:000, Collection:c, Version:3, Ttl:, FileCount:0, DeleteCount:0, DeletedByteCount:0, ReadOnly:false, ModifiedAtSecond:0 from 10.0.0.1:8080 I0824 22:18:30.161838 volume_layout.go:815 Volume 134 becomes unwritable I0824 22:18:30.161858 volume_layout.go:335 volume 134 does not have enough copies I0824 22:18:30.161928 data_node.go:114 Deleting volume id: 135 I0824 22:18:30.161969 topology.go:534 removing volume info: Id:135, Size:1, ReplicaPlacement:000, Collection:c, Version:3, Ttl:, FileCount:0, DeleteCount:0, DeletedByteCount:0, ReadOnly:false, ModifiedAtSecond:0 from 10.0.0.1:8080 I0824 22:18:30.162009 volume_layout.go:815 Volume 135 becomes unwritable I0824 22:18:30.162070 volume_layout.go:335 volume 135 does not have enough copies I0824 22:18:30.162180 data_node.go:114 Deleting volume id: 136 I0824 22:18:30.162307 topology.go:534 removing volume info: Id:136, Size:1, ReplicaPlacement:000, Collection:c, Version:3, Ttl:, FileCount:0, DeleteCount:0, DeletedByteCount:0, ReadOnly:false, ModifiedAtSecond:0 from 10.0.0.1:8080 I0824 22:18:30.162402 volume_layout.go:815 Volume 136 becomes unwritable I0824 22:18:30.162422 volume_layout.go:335 volume 136 does not have enough copies I0824 22:18:30.162522 data_node.go:114 Deleting volume id: 137 I0824 22:18:30.162592 topology.go:534 removing volume info: Id:137, Size:1, ReplicaPlacement:000, Collection:c, Version:3, Ttl:, FileCount:0, DeleteCount:0, DeletedByteCount:0, ReadOnly:false, ModifiedAtSecond:0 from 10.0.0.1:8080 I0824 22:18:30.162623 volume_layout.go:815 Volume 137 becomes unwritable I0824 22:18:30.162641 volume_layout.go:335 volume 137 does not have enough copies I0824 22:18:30.162702 data_node.go:114 Deleting volume id: 138 I0824 22:18:30.162848 topology.go:534 removing volume info: Id:138, Size:1, ReplicaPlacement:000, Collection:c, Version:3, Ttl:, FileCount:0, DeleteCount:0, DeletedByteCount:0, ReadOnly:false, ModifiedAtSecond:0 from 10.0.0.1:8080 I0824 22:18:30.162968 volume_layout.go:815 Volume 138 becomes unwritable I0824 22:18:30.162988 volume_layout.go:335 volume 138 does not have enough copies I0824 22:18:30.163068 data_node.go:114 Deleting volume id: 139 I0824 22:18:30.163158 topology.go:534 removing volume info: Id:139, Size:1, ReplicaPlacement:000, Collection:c, Version:3, Ttl:, FileCount:0, DeleteCount:0, DeletedByteCount:0, ReadOnly:false, ModifiedAtSecond:0 from 10.0.0.1:8080 I0824 22:18:30.163189 volume_layout.go:815 Volume 139 becomes unwritable I0824 22:18:30.163210 volume_layout.go:335 volume 139 does not have enough copies I0824 22:18:30.163279 data_node.go:114 Deleting volume id: 140 I0824 22:18:30.163401 topology.go:534 removing volume info: Id:140, Size:1, ReplicaPlacement:000, Collection:c, Version:3, Ttl:, FileCount:0, DeleteCount:0, DeletedByteCount:0, ReadOnly:false, ModifiedAtSecond:0 from 10.0.0.1:8080 I0824 22:18:30.163650 volume_layout.go:815 Volume 140 becomes unwritable I0824 22:18:30.163673 volume_layout.go:335 volume 140 does not have enough copies I0824 22:18:30.163821 data_node.go:114 Deleting volume id: 141 I0824 22:18:30.163932 topology.go:534 removing volume info: Id:141, Size:1, ReplicaPlacement:000, Collection:c, Version:3, Ttl:, FileCount:0, DeleteCount:0, DeletedByteCount:0, ReadOnly:false, ModifiedAtSecond:0 from 10.0.0.1:8080 I0824 22:18:30.163976 volume_layout.go:815 Volume 141 becomes unwritable I0824 22:18:30.163995 volume_layout.go:335 volume 141 does not have enough copies I0824 22:18:30.164070 data_node.go:114 Deleting volume id: 142 I0824 22:18:30.164114 topology.go:534 removing volume info: Id:142, Size:1, ReplicaPlacement:000, Collection:c, Version:3, Ttl:, FileCount:0, DeleteCount:0, DeletedByteCount:0, ReadOnly:false, ModifiedAtSecond:0 from 10.0.0.1:8080 I0824 22:18:30.164145 volume_layout.go:815 Volume 142 becomes unwritable I0824 22:18:30.164164 volume_layout.go:335 volume 142 does not have enough copies I0824 22:18:30.164299 data_node.go:114 Deleting volume id: 143 I0824 22:18:30.164410 topology.go:534 removing volume info: Id:143, Size:1, ReplicaPlacement:000, Collection:c, Version:3, Ttl:, FileCount:0, DeleteCount:0, DeletedByteCount:0, ReadOnly:false, ModifiedAtSecond:0 from 10.0.0.1:8080 I0824 22:18:30.164443 volume_layout.go:815 Volume 143 becomes unwritable I0824 22:18:30.164462 volume_layout.go:335 volume 143 does not have enough copies I0824 22:18:30.164537 data_node.go:114 Deleting volume id: 144 I0824 22:18:30.164604 topology.go:534 removing volume info: Id:144, Size:1, ReplicaPlacement:000, Collection:c, Version:3, Ttl:, FileCount:0, DeleteCount:0, DeletedByteCount:0, ReadOnly:false, ModifiedAtSecond:0 from 10.0.0.1:8080 I0824 22:18:30.164635 volume_layout.go:815 Volume 144 becomes unwritable I0824 22:18:30.164654 volume_layout.go:335 volume 144 does not have enough copies I0824 22:18:30.164725 data_node.go:114 Deleting volume id: 145 I0824 22:18:30.164773 topology.go:534 removing volume info: Id:145, Size:1, ReplicaPlacement:000, Collection:c, Version:3, Ttl:, FileCount:0, DeleteCount:0, DeletedByteCount:0, ReadOnly:false, ModifiedAtSecond:0 from 10.0.0.1:8080 I0824 22:18:30.164804 volume_layout.go:815 Volume 145 becomes unwritable I0824 22:18:30.164825 volume_layout.go:335 volume 145 does not have enough copies I0824 22:18:30.165001 data_node.go:114 Deleting volume id: 146 I0824 22:18:30.165087 topology.go:534 removing volume info: Id:146, Size:1, ReplicaPlacement:000, Collection:c, Version:3, Ttl:, FileCount:0, DeleteCount:0, DeletedByteCount:0, ReadOnly:false, ModifiedAtSecond:0 from 10.0.0.1:8080 I0824 22:18:30.165117 volume_layout.go:815 Volume 146 becomes unwritable I0824 22:18:30.165135 volume_layout.go:335 volume 146 does not have enough copies I0824 22:18:30.165211 data_node.go:114 Deleting volume id: 147 I0824 22:18:30.165285 topology.go:534 removing volume info: Id:147, Size:1, ReplicaPlacement:000, Collection:c, Version:3, Ttl:, FileCount:0, DeleteCount:0, DeletedByteCount:0, ReadOnly:false, ModifiedAtSecond:0 from 10.0.0.1:8080 I0824 22:18:30.165329 volume_layout.go:815 Volume 147 becomes unwritable I0824 22:18:30.165348 volume_layout.go:335 volume 147 does not have enough copies I0824 22:18:30.165442 data_node.go:114 Deleting volume id: 148 I0824 22:18:30.165587 topology.go:534 removing volume info: Id:148, Size:1, ReplicaPlacement:000, Collection:c, Version:3, Ttl:, FileCount:0, DeleteCount:0, DeletedByteCount:0, ReadOnly:false, ModifiedAtSecond:0 from 10.0.0.1:8080 I0824 22:18:30.165666 volume_layout.go:815 Volume 148 becomes unwritable I0824 22:18:30.165702 volume_layout.go:335 volume 148 does not have enough copies I0824 22:18:30.165757 data_node.go:114 Deleting volume id: 149 I0824 22:18:30.165819 topology.go:534 removing volume info: Id:149, Size:1, ReplicaPlacement:000, Collection:c, Version:3, Ttl:, FileCount:0, DeleteCount:0, DeletedByteCount:0, ReadOnly:false, ModifiedAtSecond:0 from 10.0.0.1:8080 I0824 22:18:30.165864 volume_layout.go:815 Volume 149 becomes unwritable I0824 22:18:30.165882 volume_layout.go:335 volume 149 does not have enough copies I0824 22:18:30.165969 data_node.go:114 Deleting volume id: 100 I0824 22:18:30.166055 topology.go:534 removing volume info: Id:100, Size:1, ReplicaPlacement:000, Collection:c, Version:3, Ttl:, FileCount:0, DeleteCount:0, DeletedByteCount:0, ReadOnly:false, ModifiedAtSecond:0 from 10.0.0.1:8080 I0824 22:18:30.166088 volume_layout.go:815 Volume 100 becomes unwritable I0824 22:18:30.166106 volume_layout.go:335 volume 100 does not have enough copies I0824 22:18:30.166182 data_node.go:114 Deleting volume id: 101 I0824 22:18:30.166246 topology.go:534 removing volume info: Id:101, Size:1, ReplicaPlacement:000, Collection:c, Version:3, Ttl:, FileCount:0, DeleteCount:0, DeletedByteCount:0, ReadOnly:false, ModifiedAtSecond:0 from 10.0.0.1:8080 I0824 22:18:30.166308 volume_layout.go:815 Volume 101 becomes unwritable I0824 22:18:30.166376 volume_layout.go:335 volume 101 does not have enough copies I0824 22:18:30.166479 data_node.go:114 Deleting volume id: 102 I0824 22:18:30.166599 topology.go:534 removing volume info: Id:102, Size:1, ReplicaPlacement:000, Collection:c, Version:3, Ttl:, FileCount:0, DeleteCount:0, DeletedByteCount:0, ReadOnly:false, ModifiedAtSecond:0 from 10.0.0.1:8080 I0824 22:18:30.166635 volume_layout.go:815 Volume 102 becomes unwritable I0824 22:18:30.166672 volume_layout.go:335 volume 102 does not have enough copies I0824 22:18:30.166731 data_node.go:114 Deleting volume id: 103 I0824 22:18:30.166840 topology.go:534 removing volume info: Id:103, Size:1, ReplicaPlacement:000, Collection:c, Version:3, Ttl:, FileCount:0, DeleteCount:0, DeletedByteCount:0, ReadOnly:false, ModifiedAtSecond:0 from 10.0.0.1:8080 I0824 22:18:30.166873 volume_layout.go:815 Volume 103 becomes unwritable I0824 22:18:30.166891 volume_layout.go:335 volume 103 does not have enough copies I0824 22:18:30.166989 data_node.go:114 Deleting volume id: 104 I0824 22:18:30.167041 topology.go:534 removing volume info: Id:104, Size:1, ReplicaPlacement:000, Collection:c, Version:3, Ttl:, FileCount:0, DeleteCount:0, DeletedByteCount:0, ReadOnly:false, ModifiedAtSecond:0 from 10.0.0.1:8080 I0824 22:18:30.167073 volume_layout.go:815 Volume 104 becomes unwritable I0824 22:18:30.167091 volume_layout.go:335 volume 104 does not have enough copies I0824 22:18:30.167180 data_node.go:114 Deleting volume id: 105 I0824 22:18:30.167224 topology.go:534 removing volume info: Id:105, Size:1, ReplicaPlacement:000, Collection:c, Version:3, Ttl:, FileCount:0, DeleteCount:0, DeletedByteCount:0, ReadOnly:false, ModifiedAtSecond:0 from 10.0.0.1:8080 I0824 22:18:30.167261 volume_layout.go:815 Volume 105 becomes unwritable I0824 22:18:30.167281 volume_layout.go:335 volume 105 does not have enough copies I0824 22:18:30.167400 data_node.go:114 Deleting volume id: 106 I0824 22:18:30.167483 topology.go:534 removing volume info: Id:106, Size:1, ReplicaPlacement:000, Collection:c, Version:3, Ttl:, FileCount:0, DeleteCount:0, DeletedByteCount:0, ReadOnly:false, ModifiedAtSecond:0 from 10.0.0.1:8080 I0824 22:18:30.167516 volume_layout.go:815 Volume 106 becomes unwritable I0824 22:18:30.167535 volume_layout.go:335 volume 106 does not have enough copies I0824 22:18:30.167625 data_node.go:114 Deleting volume id: 107 I0824 22:18:30.167811 topology.go:534 removing volume info: Id:107, Size:1, ReplicaPlacement:000, Collection:c, Version:3, Ttl:, FileCount:0, DeleteCount:0, DeletedByteCount:0, ReadOnly:false, ModifiedAtSecond:0 from 10.0.0.1:8080 I0824 22:18:30.167876 volume_layout.go:815 Volume 107 becomes unwritable I0824 22:18:30.167897 volume_layout.go:335 volume 107 does not have enough copies I0824 22:18:30.167971 data_node.go:114 Deleting volume id: 108 I0824 22:18:30.168187 topology.go:534 removing volume info: Id:108, Size:1, ReplicaPlacement:000, Collection:c, Version:3, Ttl:, FileCount:0, DeleteCount:0, DeletedByteCount:0, ReadOnly:false, ModifiedAtSecond:0 from 10.0.0.1:8080 I0824 22:18:30.168229 volume_layout.go:815 Volume 108 becomes unwritable I0824 22:18:30.168249 volume_layout.go:335 volume 108 does not have enough copies I0824 22:18:30.168326 data_node.go:114 Deleting volume id: 109 I0824 22:18:30.168386 topology.go:534 removing volume info: Id:109, Size:1, ReplicaPlacement:000, Collection:c, Version:3, Ttl:, FileCount:0, DeleteCount:0, DeletedByteCount:0, ReadOnly:false, ModifiedAtSecond:0 from 10.0.0.1:8080 I0824 22:18:30.168418 volume_layout.go:815 Volume 109 becomes unwritable I0824 22:18:30.168437 volume_layout.go:335 volume 109 does not have enough copies I0824 22:18:30.168512 data_node.go:114 Deleting volume id: 110 I0824 22:18:30.168591 topology.go:534 removing volume info: Id:110, Size:1, ReplicaPlacement:000, Collection:c, Version:3, Ttl:, FileCount:0, DeleteCount:0, DeletedByteCount:0, ReadOnly:false, ModifiedAtSecond:0 from 10.0.0.1:8080 I0824 22:18:30.168655 volume_layout.go:815 Volume 110 becomes unwritable I0824 22:18:30.168700 volume_layout.go:335 volume 110 does not have enough copies I0824 22:18:30.168815 data_node.go:114 Deleting volume id: 111 I0824 22:18:30.168860 topology.go:534 removing volume info: Id:111, Size:1, ReplicaPlacement:000, Collection:c, Version:3, Ttl:, FileCount:0, DeleteCount:0, DeletedByteCount:0, ReadOnly:false, ModifiedAtSecond:0 from 10.0.0.1:8080 I0824 22:18:30.168891 volume_layout.go:815 Volume 111 becomes unwritable I0824 22:18:30.168911 volume_layout.go:335 volume 111 does not have enough copies I0824 22:18:30.168976 data_node.go:114 Deleting volume id: 112 I0824 22:18:30.169016 topology.go:534 removing volume info: Id:112, Size:1, ReplicaPlacement:000, Collection:c, Version:3, Ttl:, FileCount:0, DeleteCount:0, DeletedByteCount:0, ReadOnly:false, ModifiedAtSecond:0 from 10.0.0.1:8080 I0824 22:18:30.169049 volume_layout.go:815 Volume 112 becomes unwritable I0824 22:18:30.169093 volume_layout.go:335 volume 112 does not have enough copies I0824 22:18:30.169174 data_node.go:114 Deleting volume id: 113 I0824 22:18:30.169214 topology.go:534 removing volume info: Id:113, Size:1, ReplicaPlacement:000, Collection:c, Version:3, Ttl:, FileCount:0, DeleteCount:0, DeletedByteCount:0, ReadOnly:false, ModifiedAtSecond:0 from 10.0.0.1:8080 I0824 22:18:30.169243 volume_layout.go:815 Volume 113 becomes unwritable I0824 22:18:30.169269 volume_layout.go:335 volume 113 does not have enough copies I0824 22:18:30.169422 data_node.go:114 Deleting volume id: 114 I0824 22:18:30.169504 topology.go:534 removing volume info: Id:114, Size:1, ReplicaPlacement:000, Collection:c, Version:3, Ttl:, FileCount:0, DeleteCount:0, DeletedByteCount:0, ReadOnly:false, ModifiedAtSecond:0 from 10.0.0.1:8080 I0824 22:18:30.169551 volume_layout.go:815 Volume 114 becomes unwritable I0824 22:18:30.169570 volume_layout.go:335 volume 114 does not have enough copies I0824 22:18:30.169716 data_node.go:114 Deleting volume id: 115 I0824 22:18:30.169832 topology.go:534 removing volume info: Id:115, Size:1, ReplicaPlacement:000, Collection:c, Version:3, Ttl:, FileCount:0, DeleteCount:0, DeletedByteCount:0, ReadOnly:false, ModifiedAtSecond:0 from 10.0.0.1:8080 I0824 22:18:30.169866 volume_layout.go:815 Volume 115 becomes unwritable I0824 22:18:30.169886 volume_layout.go:335 volume 115 does not have enough copies I0824 22:18:30.169951 data_node.go:114 Deleting volume id: 116 I0824 22:18:30.170039 topology.go:534 removing volume info: Id:116, Size:1, ReplicaPlacement:000, Collection:c, Version:3, Ttl:, FileCount:0, DeleteCount:0, DeletedByteCount:0, ReadOnly:false, ModifiedAtSecond:0 from 10.0.0.1:8080 I0824 22:18:30.170117 volume_layout.go:815 Volume 116 becomes unwritable I0824 22:18:30.170155 volume_layout.go:335 volume 116 does not have enough copies I0824 22:18:30.170219 data_node.go:114 Deleting volume id: 117 I0824 22:18:30.170283 topology.go:534 removing volume info: Id:117, Size:1, ReplicaPlacement:000, Collection:c, Version:3, Ttl:, FileCount:0, DeleteCount:0, DeletedByteCount:0, ReadOnly:false, ModifiedAtSecond:0 from 10.0.0.1:8080 I0824 22:18:30.170316 volume_layout.go:815 Volume 117 becomes unwritable I0824 22:18:30.170334 volume_layout.go:335 volume 117 does not have enough copies I0824 22:18:30.170420 data_node.go:114 Deleting volume id: 118 I0824 22:18:30.170463 topology.go:534 removing volume info: Id:118, Size:1, ReplicaPlacement:000, Collection:c, Version:3, Ttl:, FileCount:0, DeleteCount:0, DeletedByteCount:0, ReadOnly:false, ModifiedAtSecond:0 from 10.0.0.1:8080 I0824 22:18:30.170493 volume_layout.go:815 Volume 118 becomes unwritable I0824 22:18:30.170512 volume_layout.go:335 volume 118 does not have enough copies I0824 22:18:30.170610 data_node.go:114 Deleting volume id: 119 I0824 22:18:30.170651 topology.go:534 removing volume info: Id:119, Size:1, ReplicaPlacement:000, Collection:c, Version:3, Ttl:, FileCount:0, DeleteCount:0, DeletedByteCount:0, ReadOnly:false, ModifiedAtSecond:0 from 10.0.0.1:8080 I0824 22:18:30.170682 volume_layout.go:815 Volume 119 becomes unwritable I0824 22:18:30.170700 volume_layout.go:335 volume 119 does not have enough copies I0824 22:18:30.170904 data_node.go:114 Deleting volume id: 120 I0824 22:18:30.170991 topology.go:534 removing volume info: Id:120, Size:1, ReplicaPlacement:000, Collection:c, Version:3, Ttl:, FileCount:0, DeleteCount:0, DeletedByteCount:0, ReadOnly:false, ModifiedAtSecond:0 from 10.0.0.1:8080 I0824 22:18:30.171054 volume_layout.go:815 Volume 120 becomes unwritable I0824 22:18:30.171091 volume_layout.go:335 volume 120 does not have enough copies I0824 22:18:30.171246 data_node.go:114 Deleting volume id: 121 I0824 22:18:30.171323 topology.go:534 removing volume info: Id:121, Size:1, ReplicaPlacement:000, Collection:c, Version:3, Ttl:, FileCount:0, DeleteCount:0, DeletedByteCount:0, ReadOnly:false, ModifiedAtSecond:0 from 10.0.0.1:8080 I0824 22:18:30.171362 volume_layout.go:815 Volume 121 becomes unwritable I0824 22:18:30.171381 volume_layout.go:335 volume 121 does not have enough copies I0824 22:18:30.171474 data_node.go:114 Deleting volume id: 122 I0824 22:18:30.171521 topology.go:534 removing volume info: Id:122, Size:1, ReplicaPlacement:000, Collection:c, Version:3, Ttl:, FileCount:0, DeleteCount:0, DeletedByteCount:0, ReadOnly:false, ModifiedAtSecond:0 from 10.0.0.1:8080 I0824 22:18:30.171552 volume_layout.go:815 Volume 122 becomes unwritable I0824 22:18:30.171570 volume_layout.go:335 volume 122 does not have enough copies I0824 22:18:30.171642 data_node.go:114 Deleting volume id: 123 I0824 22:18:30.171717 topology.go:534 removing volume info: Id:123, Size:1, ReplicaPlacement:000, Collection:c, Version:3, Ttl:, FileCount:0, DeleteCount:0, DeletedByteCount:0, ReadOnly:false, ModifiedAtSecond:0 from 10.0.0.1:8080 I0824 22:18:30.171815 volume_layout.go:815 Volume 123 becomes unwritable I0824 22:18:30.171837 volume_layout.go:335 volume 123 does not have enough copies I0824 22:18:30.171919 data_node.go:114 Deleting volume id: 124 I0824 22:18:30.172015 topology.go:534 removing volume info: Id:124, Size:1, ReplicaPlacement:000, Collection:c, Version:3, Ttl:, FileCount:0, DeleteCount:0, DeletedByteCount:0, ReadOnly:false, ModifiedAtSecond:0 from 10.0.0.1:8080 I0824 22:18:30.172050 volume_layout.go:815 Volume 124 becomes unwritable I0824 22:18:30.172103 volume_layout.go:335 volume 124 does not have enough copies I0824 22:18:30.172163 data_node.go:114 Deleting volume id: 125 I0824 22:18:30.172204 topology.go:534 removing volume info: Id:125, Size:1, ReplicaPlacement:000, Collection:c, Version:3, Ttl:, FileCount:0, DeleteCount:0, DeletedByteCount:0, ReadOnly:false, ModifiedAtSecond:0 from 10.0.0.1:8080 I0824 22:18:30.172243 volume_layout.go:815 Volume 125 becomes unwritable I0824 22:18:30.172262 volume_layout.go:335 volume 125 does not have enough copies I0824 22:18:30.172335 data_node.go:114 Deleting volume id: 126 I0824 22:18:30.172397 topology.go:534 removing volume info: Id:126, Size:1, ReplicaPlacement:000, Collection:c, Version:3, Ttl:, FileCount:0, DeleteCount:0, DeletedByteCount:0, ReadOnly:false, ModifiedAtSecond:0 from 10.0.0.1:8080 I0824 22:18:30.172430 volume_layout.go:815 Volume 126 becomes unwritable I0824 22:18:30.172449 volume_layout.go:335 volume 126 does not have enough copies I0824 22:18:30.172507 data_node.go:114 Deleting volume id: 127 I0824 22:18:30.172751 topology.go:534 removing volume info: Id:127, Size:1, ReplicaPlacement:000, Collection:c, Version:3, Ttl:, FileCount:0, DeleteCount:0, DeletedByteCount:0, ReadOnly:false, ModifiedAtSecond:0 from 10.0.0.1:8080 I0824 22:18:30.172784 volume_layout.go:815 Volume 127 becomes unwritable I0824 22:18:30.172803 volume_layout.go:335 volume 127 does not have enough copies I0824 22:18:30.172880 data_node.go:114 Deleting volume id: 128 I0824 22:18:30.172923 topology.go:534 removing volume info: Id:128, Size:1, ReplicaPlacement:000, Collection:c, Version:3, Ttl:, FileCount:0, DeleteCount:0, DeletedByteCount:0, ReadOnly:false, ModifiedAtSecond:0 from 10.0.0.1:8080 I0824 22:18:30.172953 volume_layout.go:815 Volume 128 becomes unwritable I0824 22:18:30.172971 volume_layout.go:335 volume 128 does not have enough copies I0824 22:18:30.173086 data_node.go:114 Deleting volume id: 129 I0824 22:18:30.173211 topology.go:534 removing volume info: Id:129, Size:1, ReplicaPlacement:000, Collection:c, Version:3, Ttl:, FileCount:0, DeleteCount:0, DeletedByteCount:0, ReadOnly:false, ModifiedAtSecond:0 from 10.0.0.1:8080 I0824 22:18:30.173244 volume_layout.go:815 Volume 129 becomes unwritable I0824 22:18:30.173265 volume_layout.go:335 volume 129 does not have enough copies I0824 22:18:30.173304 data_node.go:114 Deleting volume id: 130 I0824 22:18:30.173577 topology.go:534 removing volume info: Id:130, Size:1, ReplicaPlacement:000, Collection:c, Version:3, Ttl:, FileCount:0, DeleteCount:0, DeletedByteCount:0, ReadOnly:false, ModifiedAtSecond:0 from 10.0.0.1:8080 I0824 22:18:30.173748 volume_layout.go:815 Volume 130 becomes unwritable I0824 22:18:30.173784 volume_layout.go:335 volume 130 does not have enough copies I0824 22:18:30.173947 data_node.go:114 Deleting volume id: 131 I0824 22:18:30.174178 topology.go:534 removing volume info: Id:131, Size:1, ReplicaPlacement:000, Collection:c, Version:3, Ttl:, FileCount:0, DeleteCount:0, DeletedByteCount:0, ReadOnly:false, ModifiedAtSecond:0 from 10.0.0.1:8080 I0824 22:18:30.174227 volume_layout.go:815 Volume 131 becomes unwritable I0824 22:18:30.174254 volume_layout.go:335 volume 131 does not have enough copies I0824 22:18:30.174342 data_node.go:114 Deleting volume id: 132 I0824 22:18:30.174396 topology.go:534 removing volume info: Id:132, Size:1, ReplicaPlacement:000, Collection:c, Version:3, Ttl:, FileCount:0, DeleteCount:0, DeletedByteCount:0, ReadOnly:false, ModifiedAtSecond:0 from 10.0.0.1:8080 I0824 22:18:30.174466 volume_layout.go:815 Volume 132 becomes unwritable I0824 22:18:30.174491 volume_layout.go:335 volume 132 does not have enough copies I0824 22:18:30.174630 data_node.go:114 Deleting volume id: 133 I0824 22:18:30.174752 topology.go:534 removing volume info: Id:133, Size:1, ReplicaPlacement:000, Collection:c, Version:3, Ttl:, FileCount:0, DeleteCount:0, DeletedByteCount:0, ReadOnly:false, ModifiedAtSecond:0 from 10.0.0.1:8080 I0824 22:18:30.174790 volume_layout.go:815 Volume 133 becomes unwritable I0824 22:18:30.174813 volume_layout.go:335 volume 133 does not have enough copies I0824 22:18:30.174902 data_node.go:114 Deleting volume id: 134 I0824 22:18:30.175012 topology.go:534 removing volume info: Id:134, Size:1, ReplicaPlacement:000, Collection:c, Version:3, Ttl:, FileCount:0, DeleteCount:0, DeletedByteCount:0, ReadOnly:false, ModifiedAtSecond:0 from 10.0.0.1:8080 I0824 22:18:30.175177 volume_layout.go:815 Volume 134 becomes unwritable I0824 22:18:30.175251 volume_layout.go:335 volume 134 does not have enough copies I0824 22:18:30.175375 data_node.go:114 Deleting volume id: 135 I0824 22:18:30.175477 topology.go:534 removing volume info: Id:135, Size:1, ReplicaPlacement:000, Collection:c, Version:3, Ttl:, FileCount:0, DeleteCount:0, DeletedByteCount:0, ReadOnly:false, ModifiedAtSecond:0 from 10.0.0.1:8080 I0824 22:18:30.175513 volume_layout.go:815 Volume 135 becomes unwritable I0824 22:18:30.175533 volume_layout.go:335 volume 135 does not have enough copies I0824 22:18:30.175602 data_node.go:114 Deleting volume id: 136 I0824 22:18:30.175782 topology.go:534 removing volume info: Id:136, Size:1, ReplicaPlacement:000, Collection:c, Version:3, Ttl:, FileCount:0, DeleteCount:0, DeletedByteCount:0, ReadOnly:false, ModifiedAtSecond:0 from 10.0.0.1:8080 I0824 22:18:30.175873 volume_layout.go:815 Volume 136 becomes unwritable I0824 22:18:30.175894 volume_layout.go:335 volume 136 does not have enough copies I0824 22:18:30.176032 data_node.go:114 Deleting volume id: 137 I0824 22:18:30.176110 topology.go:534 removing volume info: Id:137, Size:1, ReplicaPlacement:000, Collection:c, Version:3, Ttl:, FileCount:0, DeleteCount:0, DeletedByteCount:0, ReadOnly:false, ModifiedAtSecond:0 from 10.0.0.1:8080 I0824 22:18:30.176242 volume_layout.go:815 Volume 137 becomes unwritable I0824 22:18:30.176317 volume_layout.go:335 volume 137 does not have enough copies I0824 22:18:30.176419 data_node.go:114 Deleting volume id: 138 I0824 22:18:30.176513 topology.go:534 removing volume info: Id:138, Size:1, ReplicaPlacement:000, Collection:c, Version:3, Ttl:, FileCount:0, DeleteCount:0, DeletedByteCount:0, ReadOnly:false, ModifiedAtSecond:0 from 10.0.0.1:8080 I0824 22:18:30.176546 volume_layout.go:815 Volume 138 becomes unwritable I0824 22:18:30.176566 volume_layout.go:335 volume 138 does not have enough copies I0824 22:18:30.176645 data_node.go:114 Deleting volume id: 139 I0824 22:18:30.176686 topology.go:534 removing volume info: Id:139, Size:1, ReplicaPlacement:000, Collection:c, Version:3, Ttl:, FileCount:0, DeleteCount:0, DeletedByteCount:0, ReadOnly:false, ModifiedAtSecond:0 from 10.0.0.1:8080 I0824 22:18:30.176717 volume_layout.go:815 Volume 139 becomes unwritable I0824 22:18:30.176735 volume_layout.go:335 volume 139 does not have enough copies I0824 22:18:30.176878 data_node.go:114 Deleting volume id: 140 I0824 22:18:30.177032 topology.go:534 removing volume info: Id:140, Size:1, ReplicaPlacement:000, Collection:c, Version:3, Ttl:, FileCount:0, DeleteCount:0, DeletedByteCount:0, ReadOnly:false, ModifiedAtSecond:0 from 10.0.0.1:8080 I0824 22:18:30.177066 volume_layout.go:815 Volume 140 becomes unwritable I0824 22:18:30.177085 volume_layout.go:335 volume 140 does not have enough copies I0824 22:18:30.177144 data_node.go:114 Deleting volume id: 141 I0824 22:18:30.177227 topology.go:534 removing volume info: Id:141, Size:1, ReplicaPlacement:000, Collection:c, Version:3, Ttl:, FileCount:0, DeleteCount:0, DeletedByteCount:0, ReadOnly:false, ModifiedAtSecond:0 from 10.0.0.1:8080 I0824 22:18:30.177292 volume_layout.go:815 Volume 141 becomes unwritable I0824 22:18:30.177321 volume_layout.go:335 volume 141 does not have enough copies I0824 22:18:30.177478 data_node.go:114 Deleting volume id: 142 I0824 22:18:30.177591 topology.go:534 removing volume info: Id:142, Size:1, ReplicaPlacement:000, Collection:c, Version:3, Ttl:, FileCount:0, DeleteCount:0, DeletedByteCount:0, ReadOnly:false, ModifiedAtSecond:0 from 10.0.0.1:8080 I0824 22:18:30.177671 volume_layout.go:815 Volume 142 becomes unwritable I0824 22:18:30.177742 volume_layout.go:335 volume 142 does not have enough copies I0824 22:18:30.177856 data_node.go:114 Deleting volume id: 143 I0824 22:18:30.177930 topology.go:534 removing volume info: Id:143, Size:1, ReplicaPlacement:000, Collection:c, Version:3, Ttl:, FileCount:0, DeleteCount:0, DeletedByteCount:0, ReadOnly:false, ModifiedAtSecond:0 from 10.0.0.1:8080 I0824 22:18:30.178144 volume_layout.go:815 Volume 143 becomes unwritable I0824 22:18:30.178329 volume_layout.go:335 volume 143 does not have enough copies I0824 22:18:30.178605 data_node.go:114 Deleting volume id: 144 I0824 22:18:30.178780 topology.go:534 removing volume info: Id:144, Size:1, ReplicaPlacement:000, Collection:c, Version:3, Ttl:, FileCount:0, DeleteCount:0, DeletedByteCount:0, ReadOnly:false, ModifiedAtSecond:0 from 10.0.0.1:8080 I0824 22:18:30.178916 volume_layout.go:815 Volume 144 becomes unwritable I0824 22:18:30.178951 volume_layout.go:335 volume 144 does not have enough copies I0824 22:18:30.179074 data_node.go:114 Deleting volume id: 145 I0824 22:18:30.179203 topology.go:534 removing volume info: Id:145, Size:1, ReplicaPlacement:000, Collection:c, Version:3, Ttl:, FileCount:0, DeleteCount:0, DeletedByteCount:0, ReadOnly:false, ModifiedAtSecond:0 from 10.0.0.1:8080 I0824 22:18:30.179302 volume_layout.go:815 Volume 145 becomes unwritable I0824 22:18:30.179338 volume_layout.go:335 volume 145 does not have enough copies I0824 22:18:30.179601 data_node.go:114 Deleting volume id: 146 I0824 22:18:30.179786 topology.go:534 removing volume info: Id:146, Size:1, ReplicaPlacement:000, Collection:c, Version:3, Ttl:, FileCount:0, DeleteCount:0, DeletedByteCount:0, ReadOnly:false, ModifiedAtSecond:0 from 10.0.0.1:8080 I0824 22:18:30.179919 volume_layout.go:815 Volume 146 becomes unwritable I0824 22:18:30.179974 volume_layout.go:335 volume 146 does not have enough copies I0824 22:18:30.180097 data_node.go:114 Deleting volume id: 147 I0824 22:18:30.180214 topology.go:534 removing volume info: Id:147, Size:1, ReplicaPlacement:000, Collection:c, Version:3, Ttl:, FileCount:0, DeleteCount:0, DeletedByteCount:0, ReadOnly:false, ModifiedAtSecond:0 from 10.0.0.1:8080 I0824 22:18:30.180263 volume_layout.go:815 Volume 147 becomes unwritable I0824 22:18:30.180318 volume_layout.go:335 volume 147 does not have enough copies I0824 22:18:30.180452 data_node.go:114 Deleting volume id: 148 I0824 22:18:30.180538 topology.go:534 removing volume info: Id:148, Size:1, ReplicaPlacement:000, Collection:c, Version:3, Ttl:, FileCount:0, DeleteCount:0, DeletedByteCount:0, ReadOnly:false, ModifiedAtSecond:0 from 10.0.0.1:8080 I0824 22:18:30.180577 volume_layout.go:815 Volume 148 becomes unwritable I0824 22:18:30.180635 volume_layout.go:335 volume 148 does not have enough copies I0824 22:18:30.180711 data_node.go:114 Deleting volume id: 149 I0824 22:18:30.180944 topology.go:534 removing volume info: Id:149, Size:1, ReplicaPlacement:000, Collection:c, Version:3, Ttl:, FileCount:0, DeleteCount:0, DeletedByteCount:0, ReadOnly:false, ModifiedAtSecond:0 from 10.0.0.1:8080 I0824 22:18:30.181017 volume_layout.go:815 Volume 149 becomes unwritable I0824 22:18:30.181047 volume_layout.go:335 volume 149 does not have enough copies I0824 22:18:30.181129 data_node.go:114 Deleting volume id: 100 I0824 22:18:30.181184 topology.go:534 removing volume info: Id:100, Size:1, ReplicaPlacement:000, Collection:c, Version:3, Ttl:, FileCount:0, DeleteCount:0, DeletedByteCount:0, ReadOnly:false, ModifiedAtSecond:0 from 10.0.0.1:8080 I0824 22:18:30.181218 volume_layout.go:815 Volume 100 becomes unwritable I0824 22:18:30.181238 volume_layout.go:335 volume 100 does not have enough copies I0824 22:18:30.181321 data_node.go:114 Deleting volume id: 101 I0824 22:18:30.181472 topology.go:534 removing volume info: Id:101, Size:1, ReplicaPlacement:000, Collection:c, Version:3, Ttl:, FileCount:0, DeleteCount:0, DeletedByteCount:0, ReadOnly:false, ModifiedAtSecond:0 from 10.0.0.1:8080 I0824 22:18:30.181507 volume_layout.go:815 Volume 101 becomes unwritable I0824 22:18:30.181528 volume_layout.go:335 volume 101 does not have enough copies I0824 22:18:30.181597 data_node.go:114 Deleting volume id: 102 I0824 22:18:30.181659 topology.go:534 removing volume info: Id:102, Size:1, ReplicaPlacement:000, Collection:c, Version:3, Ttl:, FileCount:0, DeleteCount:0, DeletedByteCount:0, ReadOnly:false, ModifiedAtSecond:0 from 10.0.0.1:8080 I0824 22:18:30.181693 volume_layout.go:815 Volume 102 becomes unwritable I0824 22:18:30.181757 volume_layout.go:335 volume 102 does not have enough copies I0824 22:18:30.181857 data_node.go:114 Deleting volume id: 103 I0824 22:18:30.181995 topology.go:534 removing volume info: Id:103, Size:1, ReplicaPlacement:000, Collection:c, Version:3, Ttl:, FileCount:0, DeleteCount:0, DeletedByteCount:0, ReadOnly:false, ModifiedAtSecond:0 from 10.0.0.1:8080 I0824 22:18:30.182075 volume_layout.go:815 Volume 103 becomes unwritable I0824 22:18:30.182096 volume_layout.go:335 volume 103 does not have enough copies I0824 22:18:30.182165 data_node.go:114 Deleting volume id: 104 I0824 22:18:30.182213 topology.go:534 removing volume info: Id:104, Size:1, ReplicaPlacement:000, Collection:c, Version:3, Ttl:, FileCount:0, DeleteCount:0, DeletedByteCount:0, ReadOnly:false, ModifiedAtSecond:0 from 10.0.0.1:8080 I0824 22:18:30.182349 volume_layout.go:815 Volume 104 becomes unwritable I0824 22:18:30.182372 volume_layout.go:335 volume 104 does not have enough copies I0824 22:18:30.182447 data_node.go:114 Deleting volume id: 105 I0824 22:18:30.182566 topology.go:534 removing volume info: Id:105, Size:1, ReplicaPlacement:000, Collection:c, Version:3, Ttl:, FileCount:0, DeleteCount:0, DeletedByteCount:0, ReadOnly:false, ModifiedAtSecond:0 from 10.0.0.1:8080 I0824 22:18:30.182623 volume_layout.go:815 Volume 105 becomes unwritable I0824 22:18:30.182642 volume_layout.go:335 volume 105 does not have enough copies I0824 22:18:30.182722 data_node.go:114 Deleting volume id: 106 I0824 22:18:30.182805 topology.go:534 removing volume info: Id:106, Size:1, ReplicaPlacement:000, Collection:c, Version:3, Ttl:, FileCount:0, DeleteCount:0, DeletedByteCount:0, ReadOnly:false, ModifiedAtSecond:0 from 10.0.0.1:8080 I0824 22:18:30.182862 volume_layout.go:815 Volume 106 becomes unwritable I0824 22:18:30.182886 volume_layout.go:335 volume 106 does not have enough copies I0824 22:18:30.182994 data_node.go:114 Deleting volume id: 107 I0824 22:18:30.183076 topology.go:534 removing volume info: Id:107, Size:1, ReplicaPlacement:000, Collection:c, Version:3, Ttl:, FileCount:0, DeleteCount:0, DeletedByteCount:0, ReadOnly:false, ModifiedAtSecond:0 from 10.0.0.1:8080 I0824 22:18:30.183153 volume_layout.go:815 Volume 107 becomes unwritable I0824 22:18:30.183175 volume_layout.go:335 volume 107 does not have enough copies I0824 22:18:30.183366 data_node.go:114 Deleting volume id: 108 I0824 22:18:30.183520 topology.go:534 removing volume info: Id:108, Size:1, ReplicaPlacement:000, Collection:c, Version:3, Ttl:, FileCount:0, DeleteCount:0, DeletedByteCount:0, ReadOnly:false, ModifiedAtSecond:0 from 10.0.0.1:8080 I0824 22:18:30.183622 volume_layout.go:815 Volume 108 becomes unwritable I0824 22:18:30.183647 volume_layout.go:335 volume 108 does not have enough copies I0824 22:18:30.183699 data_node.go:114 Deleting volume id: 109 I0824 22:18:30.183757 topology.go:534 removing volume info: Id:109, Size:1, ReplicaPlacement:000, Collection:c, Version:3, Ttl:, FileCount:0, DeleteCount:0, DeletedByteCount:0, ReadOnly:false, ModifiedAtSecond:0 from 10.0.0.1:8080 I0824 22:18:30.184058 volume_layout.go:815 Volume 109 becomes unwritable I0824 22:18:30.184235 volume_layout.go:335 volume 109 does not have enough copies I0824 22:18:30.184387 data_node.go:114 Deleting volume id: 110 I0824 22:18:30.184592 topology.go:534 removing volume info: Id:110, Size:1, ReplicaPlacement:000, Collection:c, Version:3, Ttl:, FileCount:0, DeleteCount:0, DeletedByteCount:0, ReadOnly:false, ModifiedAtSecond:0 from 10.0.0.1:8080 I0824 22:18:30.184678 volume_layout.go:815 Volume 110 becomes unwritable I0824 22:18:30.184704 volume_layout.go:335 volume 110 does not have enough copies I0824 22:18:30.184766 data_node.go:114 Deleting volume id: 111 I0824 22:18:30.184916 topology.go:534 removing volume info: Id:111, Size:1, ReplicaPlacement:000, Collection:c, Version:3, Ttl:, FileCount:0, DeleteCount:0, DeletedByteCount:0, ReadOnly:false, ModifiedAtSecond:0 from 10.0.0.1:8080 I0824 22:18:30.185033 volume_layout.go:815 Volume 111 becomes unwritable I0824 22:18:30.185065 volume_layout.go:335 volume 111 does not have enough copies I0824 22:18:30.185213 data_node.go:114 Deleting volume id: 112 I0824 22:18:30.185333 topology.go:534 removing volume info: Id:112, Size:1, ReplicaPlacement:000, Collection:c, Version:3, Ttl:, FileCount:0, DeleteCount:0, DeletedByteCount:0, ReadOnly:false, ModifiedAtSecond:0 from 10.0.0.1:8080 I0824 22:18:30.185451 volume_layout.go:815 Volume 112 becomes unwritable I0824 22:18:30.185491 volume_layout.go:335 volume 112 does not have enough copies I0824 22:18:30.185575 data_node.go:114 Deleting volume id: 113 I0824 22:18:30.185654 topology.go:534 removing volume info: Id:113, Size:1, ReplicaPlacement:000, Collection:c, Version:3, Ttl:, FileCount:0, DeleteCount:0, DeletedByteCount:0, ReadOnly:false, ModifiedAtSecond:0 from 10.0.0.1:8080 I0824 22:18:30.185730 volume_layout.go:815 Volume 113 becomes unwritable I0824 22:18:30.185755 volume_layout.go:335 volume 113 does not have enough copies I0824 22:18:30.185842 data_node.go:114 Deleting volume id: 114 I0824 22:18:30.185919 topology.go:534 removing volume info: Id:114, Size:1, ReplicaPlacement:000, Collection:c, Version:3, Ttl:, FileCount:0, DeleteCount:0, DeletedByteCount:0, ReadOnly:false, ModifiedAtSecond:0 from 10.0.0.1:8080 I0824 22:18:30.185983 volume_layout.go:815 Volume 114 becomes unwritable I0824 22:18:30.186069 volume_layout.go:335 volume 114 does not have enough copies I0824 22:18:30.186181 data_node.go:114 Deleting volume id: 115 I0824 22:18:30.186226 topology.go:534 removing volume info: Id:115, Size:1, ReplicaPlacement:000, Collection:c, Version:3, Ttl:, FileCount:0, DeleteCount:0, DeletedByteCount:0, ReadOnly:false, ModifiedAtSecond:0 from 10.0.0.1:8080 I0824 22:18:30.186258 volume_layout.go:815 Volume 115 becomes unwritable I0824 22:18:30.186296 volume_layout.go:335 volume 115 does not have enough copies I0824 22:18:30.186457 data_node.go:114 Deleting volume id: 116 I0824 22:18:30.186533 topology.go:534 removing volume info: Id:116, Size:1, ReplicaPlacement:000, Collection:c, Version:3, Ttl:, FileCount:0, DeleteCount:0, DeletedByteCount:0, ReadOnly:false, ModifiedAtSecond:0 from 10.0.0.1:8080 I0824 22:18:30.186567 volume_layout.go:815 Volume 116 becomes unwritable I0824 22:18:30.186586 volume_layout.go:335 volume 116 does not have enough copies I0824 22:18:30.186648 data_node.go:114 Deleting volume id: 117 I0824 22:18:30.186753 topology.go:534 removing volume info: Id:117, Size:1, ReplicaPlacement:000, Collection:c, Version:3, Ttl:, FileCount:0, DeleteCount:0, DeletedByteCount:0, ReadOnly:false, ModifiedAtSecond:0 from 10.0.0.1:8080 I0824 22:18:30.186823 volume_layout.go:815 Volume 117 becomes unwritable I0824 22:18:30.186843 volume_layout.go:335 volume 117 does not have enough copies I0824 22:18:30.186998 data_node.go:114 Deleting volume id: 118 I0824 22:18:30.187082 topology.go:534 removing volume info: Id:118, Size:1, ReplicaPlacement:000, Collection:c, Version:3, Ttl:, FileCount:0, DeleteCount:0, DeletedByteCount:0, ReadOnly:false, ModifiedAtSecond:0 from 10.0.0.1:8080 I0824 22:18:30.187116 volume_layout.go:815 Volume 118 becomes unwritable I0824 22:18:30.187156 volume_layout.go:335 volume 118 does not have enough copies I0824 22:18:30.187216 data_node.go:114 Deleting volume id: 119 I0824 22:18:30.187277 topology.go:534 removing volume info: Id:119, Size:1, ReplicaPlacement:000, Collection:c, Version:3, Ttl:, FileCount:0, DeleteCount:0, DeletedByteCount:0, ReadOnly:false, ModifiedAtSecond:0 from 10.0.0.1:8080 I0824 22:18:30.187440 volume_layout.go:815 Volume 119 becomes unwritable I0824 22:18:30.187462 volume_layout.go:335 volume 119 does not have enough copies I0824 22:18:30.187531 data_node.go:114 Deleting volume id: 120 I0824 22:18:30.187573 topology.go:534 removing volume info: Id:120, Size:1, ReplicaPlacement:000, Collection:c, Version:3, Ttl:, FileCount:0, DeleteCount:0, DeletedByteCount:0, ReadOnly:false, ModifiedAtSecond:0 from 10.0.0.1:8080 I0824 22:18:30.187625 volume_layout.go:815 Volume 120 becomes unwritable I0824 22:18:30.187646 volume_layout.go:335 volume 120 does not have enough copies I0824 22:18:30.187714 data_node.go:114 Deleting volume id: 121 I0824 22:18:30.187767 topology.go:534 removing volume info: Id:121, Size:1, ReplicaPlacement:000, Collection:c, Version:3, Ttl:, FileCount:0, DeleteCount:0, DeletedByteCount:0, ReadOnly:false, ModifiedAtSecond:0 from 10.0.0.1:8080 I0824 22:18:30.187827 volume_layout.go:815 Volume 121 becomes unwritable I0824 22:18:30.187847 volume_layout.go:335 volume 121 does not have enough copies I0824 22:18:30.188017 data_node.go:114 Deleting volume id: 122 I0824 22:18:30.188166 topology.go:534 removing volume info: Id:122, Size:1, ReplicaPlacement:000, Collection:c, Version:3, Ttl:, FileCount:0, DeleteCount:0, DeletedByteCount:0, ReadOnly:false, ModifiedAtSecond:0 from 10.0.0.1:8080 I0824 22:18:30.188226 volume_layout.go:815 Volume 122 becomes unwritable I0824 22:18:30.188247 volume_layout.go:335 volume 122 does not have enough copies I0824 22:18:30.188321 data_node.go:114 Deleting volume id: 123 I0824 22:18:30.188365 topology.go:534 removing volume info: Id:123, Size:1, ReplicaPlacement:000, Collection:c, Version:3, Ttl:, FileCount:0, DeleteCount:0, DeletedByteCount:0, ReadOnly:false, ModifiedAtSecond:0 from 10.0.0.1:8080 I0824 22:18:30.188395 volume_layout.go:815 Volume 123 becomes unwritable I0824 22:18:30.188416 volume_layout.go:335 volume 123 does not have enough copies I0824 22:18:30.188547 data_node.go:114 Deleting volume id: 124 I0824 22:18:30.188590 topology.go:534 removing volume info: Id:124, Size:1, ReplicaPlacement:000, Collection:c, Version:3, Ttl:, FileCount:0, DeleteCount:0, DeletedByteCount:0, ReadOnly:false, ModifiedAtSecond:0 from 10.0.0.1:8080 I0824 22:18:30.188640 volume_layout.go:815 Volume 124 becomes unwritable I0824 22:18:30.188664 volume_layout.go:335 volume 124 does not have enough copies I0824 22:18:30.188741 data_node.go:114 Deleting volume id: 125 I0824 22:18:30.188929 topology.go:534 removing volume info: Id:125, Size:1, ReplicaPlacement:000, Collection:c, Version:3, Ttl:, FileCount:0, DeleteCount:0, DeletedByteCount:0, ReadOnly:false, ModifiedAtSecond:0 from 10.0.0.1:8080 I0824 22:18:30.188967 volume_layout.go:815 Volume 125 becomes unwritable I0824 22:18:30.188986 volume_layout.go:335 volume 125 does not have enough copies I0824 22:18:30.189060 data_node.go:114 Deleting volume id: 126 I0824 22:18:30.189169 topology.go:534 removing volume info: Id:126, Size:1, ReplicaPlacement:000, Collection:c, Version:3, Ttl:, FileCount:0, DeleteCount:0, DeletedByteCount:0, ReadOnly:false, ModifiedAtSecond:0 from 10.0.0.1:8080 I0824 22:18:30.189226 volume_layout.go:815 Volume 126 becomes unwritable I0824 22:18:30.189281 volume_layout.go:335 volume 126 does not have enough copies I0824 22:18:30.189402 data_node.go:114 Deleting volume id: 127 I0824 22:18:30.189513 topology.go:534 removing volume info: Id:127, Size:1, ReplicaPlacement:000, Collection:c, Version:3, Ttl:, FileCount:0, DeleteCount:0, DeletedByteCount:0, ReadOnly:false, ModifiedAtSecond:0 from 10.0.0.1:8080 I0824 22:18:30.189547 volume_layout.go:815 Volume 127 becomes unwritable I0824 22:18:30.189568 volume_layout.go:335 volume 127 does not have enough copies I0824 22:18:30.189647 data_node.go:114 Deleting volume id: 128 I0824 22:18:30.189690 topology.go:534 removing volume info: Id:128, Size:1, ReplicaPlacement:000, Collection:c, Version:3, Ttl:, FileCount:0, DeleteCount:0, DeletedByteCount:0, ReadOnly:false, ModifiedAtSecond:0 from 10.0.0.1:8080 I0824 22:18:30.189787 volume_layout.go:815 Volume 128 becomes unwritable I0824 22:18:30.189845 volume_layout.go:335 volume 128 does not have enough copies I0824 22:18:30.189974 data_node.go:114 Deleting volume id: 129 I0824 22:18:30.190052 topology.go:534 removing volume info: Id:129, Size:1, ReplicaPlacement:000, Collection:c, Version:3, Ttl:, FileCount:0, DeleteCount:0, DeletedByteCount:0, ReadOnly:false, ModifiedAtSecond:0 from 10.0.0.1:8080 I0824 22:18:30.190083 volume_layout.go:815 Volume 129 becomes unwritable I0824 22:18:30.190103 volume_layout.go:335 volume 129 does not have enough copies I0824 22:18:30.190223 data_node.go:114 Deleting volume id: 130 I0824 22:18:30.190316 topology.go:534 removing volume info: Id:130, Size:1, ReplicaPlacement:000, Collection:c, Version:3, Ttl:, FileCount:0, DeleteCount:0, DeletedByteCount:0, ReadOnly:false, ModifiedAtSecond:0 from 10.0.0.1:8080 I0824 22:18:30.190447 volume_layout.go:815 Volume 130 becomes unwritable I0824 22:18:30.190516 volume_layout.go:335 volume 130 does not have enough copies I0824 22:18:30.190591 data_node.go:114 Deleting volume id: 131 I0824 22:18:30.190636 topology.go:534 removing volume info: Id:131, Size:1, ReplicaPlacement:000, Collection:c, Version:3, Ttl:, FileCount:0, DeleteCount:0, DeletedByteCount:0, ReadOnly:false, ModifiedAtSecond:0 from 10.0.0.1:8080 I0824 22:18:30.190667 volume_layout.go:815 Volume 131 becomes unwritable I0824 22:18:30.190687 volume_layout.go:335 volume 131 does not have enough copies I0824 22:18:30.190806 data_node.go:114 Deleting volume id: 132 I0824 22:18:30.190926 topology.go:534 removing volume info: Id:132, Size:1, ReplicaPlacement:000, Collection:c, Version:3, Ttl:, FileCount:0, DeleteCount:0, DeletedByteCount:0, ReadOnly:false, ModifiedAtSecond:0 from 10.0.0.1:8080 I0824 22:18:30.191011 volume_layout.go:815 Volume 132 becomes unwritable I0824 22:18:30.191031 volume_layout.go:335 volume 132 does not have enough copies I0824 22:18:30.191093 data_node.go:114 Deleting volume id: 133 I0824 22:18:30.191135 topology.go:534 removing volume info: Id:133, Size:1, ReplicaPlacement:000, Collection:c, Version:3, Ttl:, FileCount:0, DeleteCount:0, DeletedByteCount:0, ReadOnly:false, ModifiedAtSecond:0 from 10.0.0.1:8080 I0824 22:18:30.191181 volume_layout.go:815 Volume 133 becomes unwritable I0824 22:18:30.191239 volume_layout.go:335 volume 133 does not have enough copies I0824 22:18:30.191345 data_node.go:114 Deleting volume id: 134 I0824 22:18:30.191437 topology.go:534 removing volume info: Id:134, Size:1, ReplicaPlacement:000, Collection:c, Version:3, Ttl:, FileCount:0, DeleteCount:0, DeletedByteCount:0, ReadOnly:false, ModifiedAtSecond:0 from 10.0.0.1:8080 I0824 22:18:30.191487 volume_layout.go:815 Volume 134 becomes unwritable I0824 22:18:30.191526 volume_layout.go:335 volume 134 does not have enough copies I0824 22:18:30.191598 data_node.go:114 Deleting volume id: 135 I0824 22:18:30.191668 topology.go:534 removing volume info: Id:135, Size:1, ReplicaPlacement:000, Collection:c, Version:3, Ttl:, FileCount:0, DeleteCount:0, DeletedByteCount:0, ReadOnly:false, ModifiedAtSecond:0 from 10.0.0.1:8080 I0824 22:18:30.191701 volume_layout.go:815 Volume 135 becomes unwritable I0824 22:18:30.191721 volume_layout.go:335 volume 135 does not have enough copies I0824 22:18:30.191789 data_node.go:114 Deleting volume id: 136 I0824 22:18:30.191830 topology.go:534 removing volume info: Id:136, Size:1, ReplicaPlacement:000, Collection:c, Version:3, Ttl:, FileCount:0, DeleteCount:0, DeletedByteCount:0, ReadOnly:false, ModifiedAtSecond:0 from 10.0.0.1:8080 I0824 22:18:30.191876 volume_layout.go:815 Volume 136 becomes unwritable I0824 22:18:30.191894 volume_layout.go:335 volume 136 does not have enough copies I0824 22:18:30.191948 data_node.go:114 Deleting volume id: 137 I0824 22:18:30.192075 topology.go:534 removing volume info: Id:137, Size:1, ReplicaPlacement:000, Collection:c, Version:3, Ttl:, FileCount:0, DeleteCount:0, DeletedByteCount:0, ReadOnly:false, ModifiedAtSecond:0 from 10.0.0.1:8080 I0824 22:18:30.192110 volume_layout.go:815 Volume 137 becomes unwritable I0824 22:18:30.192128 volume_layout.go:335 volume 137 does not have enough copies I0824 22:18:30.192217 data_node.go:114 Deleting volume id: 138 I0824 22:18:30.192325 topology.go:534 removing volume info: Id:138, Size:1, ReplicaPlacement:000, Collection:c, Version:3, Ttl:, FileCount:0, DeleteCount:0, DeletedByteCount:0, ReadOnly:false, ModifiedAtSecond:0 from 10.0.0.1:8080 I0824 22:18:30.192432 volume_layout.go:815 Volume 138 becomes unwritable I0824 22:18:30.192469 volume_layout.go:335 volume 138 does not have enough copies I0824 22:18:30.192532 data_node.go:114 Deleting volume id: 139 I0824 22:18:30.192603 topology.go:534 removing volume info: Id:139, Size:1, ReplicaPlacement:000, Collection:c, Version:3, Ttl:, FileCount:0, DeleteCount:0, DeletedByteCount:0, ReadOnly:false, ModifiedAtSecond:0 from 10.0.0.1:8080 I0824 22:18:30.192730 volume_layout.go:815 Volume 139 becomes unwritable I0824 22:18:30.192787 volume_layout.go:335 volume 139 does not have enough copies I0824 22:18:30.192889 data_node.go:114 Deleting volume id: 140 I0824 22:18:30.192930 topology.go:534 removing volume info: Id:140, Size:1, ReplicaPlacement:000, Collection:c, Version:3, Ttl:, FileCount:0, DeleteCount:0, DeletedByteCount:0, ReadOnly:false, ModifiedAtSecond:0 from 10.0.0.1:8080 I0824 22:18:30.192994 volume_layout.go:815 Volume 140 becomes unwritable I0824 22:18:30.193017 volume_layout.go:335 volume 140 does not have enough copies I0824 22:18:30.193105 data_node.go:114 Deleting volume id: 141 I0824 22:18:30.193149 topology.go:534 removing volume info: Id:141, Size:1, ReplicaPlacement:000, Collection:c, Version:3, Ttl:, FileCount:0, DeleteCount:0, DeletedByteCount:0, ReadOnly:false, ModifiedAtSecond:0 from 10.0.0.1:8080 I0824 22:18:30.193179 volume_layout.go:815 Volume 141 becomes unwritable I0824 22:18:30.193227 volume_layout.go:335 volume 141 does not have enough copies I0824 22:18:30.193292 data_node.go:114 Deleting volume id: 142 I0824 22:18:30.193332 topology.go:534 removing volume info: Id:142, Size:1, ReplicaPlacement:000, Collection:c, Version:3, Ttl:, FileCount:0, DeleteCount:0, DeletedByteCount:0, ReadOnly:false, ModifiedAtSecond:0 from 10.0.0.1:8080 I0824 22:18:30.193392 volume_layout.go:815 Volume 142 becomes unwritable I0824 22:18:30.193412 volume_layout.go:335 volume 142 does not have enough copies I0824 22:18:30.193470 data_node.go:114 Deleting volume id: 143 I0824 22:18:30.193515 topology.go:534 removing volume info: Id:143, Size:1, ReplicaPlacement:000, Collection:c, Version:3, Ttl:, FileCount:0, DeleteCount:0, DeletedByteCount:0, ReadOnly:false, ModifiedAtSecond:0 from 10.0.0.1:8080 I0824 22:18:30.193609 volume_layout.go:815 Volume 143 becomes unwritable I0824 22:18:30.193700 volume_layout.go:335 volume 143 does not have enough copies I0824 22:18:30.193965 data_node.go:114 Deleting volume id: 144 I0824 22:18:30.194148 topology.go:534 removing volume info: Id:144, Size:1, ReplicaPlacement:000, Collection:c, Version:3, Ttl:, FileCount:0, DeleteCount:0, DeletedByteCount:0, ReadOnly:false, ModifiedAtSecond:0 from 10.0.0.1:8080 I0824 22:18:30.194231 volume_layout.go:815 Volume 144 becomes unwritable I0824 22:18:30.194299 volume_layout.go:335 volume 144 does not have enough copies I0824 22:18:30.194382 data_node.go:114 Deleting volume id: 145 I0824 22:18:30.194613 topology.go:534 removing volume info: Id:145, Size:1, ReplicaPlacement:000, Collection:c, Version:3, Ttl:, FileCount:0, DeleteCount:0, DeletedByteCount:0, ReadOnly:false, ModifiedAtSecond:0 from 10.0.0.1:8080 I0824 22:18:30.194782 volume_layout.go:815 Volume 145 becomes unwritable I0824 22:18:30.194823 volume_layout.go:335 volume 145 does not have enough copies I0824 22:18:30.195010 data_node.go:114 Deleting volume id: 146 I0824 22:18:30.195179 topology.go:534 removing volume info: Id:146, Size:1, ReplicaPlacement:000, Collection:c, Version:3, Ttl:, FileCount:0, DeleteCount:0, DeletedByteCount:0, ReadOnly:false, ModifiedAtSecond:0 from 10.0.0.1:8080 I0824 22:18:30.195255 volume_layout.go:815 Volume 146 becomes unwritable I0824 22:18:30.195281 volume_layout.go:335 volume 146 does not have enough copies I0824 22:18:30.195372 data_node.go:114 Deleting volume id: 147 I0824 22:18:30.195559 topology.go:534 removing volume info: Id:147, Size:1, ReplicaPlacement:000, Collection:c, Version:3, Ttl:, FileCount:0, DeleteCount:0, DeletedByteCount:0, ReadOnly:false, ModifiedAtSecond:0 from 10.0.0.1:8080 I0824 22:18:30.195627 volume_layout.go:815 Volume 147 becomes unwritable I0824 22:18:30.195654 volume_layout.go:335 volume 147 does not have enough copies I0824 22:18:30.195765 data_node.go:114 Deleting volume id: 148 I0824 22:18:30.195895 topology.go:534 removing volume info: Id:148, Size:1, ReplicaPlacement:000, Collection:c, Version:3, Ttl:, FileCount:0, DeleteCount:0, DeletedByteCount:0, ReadOnly:false, ModifiedAtSecond:0 from 10.0.0.1:8080 I0824 22:18:30.197196 volume_layout.go:815 Volume 148 becomes unwritable I0824 22:18:30.197301 volume_layout.go:335 volume 148 does not have enough copies I0824 22:18:30.197516 data_node.go:114 Deleting volume id: 149 I0824 22:18:30.197622 topology.go:534 removing volume info: Id:149, Size:1, ReplicaPlacement:000, Collection:c, Version:3, Ttl:, FileCount:0, DeleteCount:0, DeletedByteCount:0, ReadOnly:false, ModifiedAtSecond:0 from 10.0.0.1:8080 I0824 22:18:30.197738 volume_layout.go:815 Volume 149 becomes unwritable I0824 22:18:30.197771 volume_layout.go:335 volume 149 does not have enough copies I0824 22:18:30.197852 data_node.go:114 Deleting volume id: 100 I0824 22:18:30.198004 topology.go:534 removing volume info: Id:100, Size:1, ReplicaPlacement:000, Collection:c, Version:3, Ttl:, FileCount:0, DeleteCount:0, DeletedByteCount:0, ReadOnly:false, ModifiedAtSecond:0 from 10.0.0.1:8080 I0824 22:18:30.198044 volume_layout.go:815 Volume 100 becomes unwritable I0824 22:18:30.198098 volume_layout.go:335 volume 100 does not have enough copies I0824 22:18:30.198207 data_node.go:114 Deleting volume id: 101 I0824 22:18:30.198356 topology.go:534 removing volume info: Id:101, Size:1, ReplicaPlacement:000, Collection:c, Version:3, Ttl:, FileCount:0, DeleteCount:0, DeletedByteCount:0, ReadOnly:false, ModifiedAtSecond:0 from 10.0.0.1:8080 I0824 22:18:30.198398 volume_layout.go:815 Volume 101 becomes unwritable I0824 22:18:30.198418 volume_layout.go:335 volume 101 does not have enough copies I0824 22:18:30.198529 data_node.go:114 Deleting volume id: 102 I0824 22:18:30.198661 topology.go:534 removing volume info: Id:102, Size:1, ReplicaPlacement:000, Collection:c, Version:3, Ttl:, FileCount:0, DeleteCount:0, DeletedByteCount:0, ReadOnly:false, ModifiedAtSecond:0 from 10.0.0.1:8080 I0824 22:18:30.198740 volume_layout.go:815 Volume 102 becomes unwritable I0824 22:18:30.198821 volume_layout.go:335 volume 102 does not have enough copies I0824 22:18:30.198916 data_node.go:114 Deleting volume id: 103 I0824 22:18:30.199016 topology.go:534 removing volume info: Id:103, Size:1, ReplicaPlacement:000, Collection:c, Version:3, Ttl:, FileCount:0, DeleteCount:0, DeletedByteCount:0, ReadOnly:false, ModifiedAtSecond:0 from 10.0.0.1:8080 I0824 22:18:30.199102 volume_layout.go:815 Volume 103 becomes unwritable I0824 22:18:30.199131 volume_layout.go:335 volume 103 does not have enough copies I0824 22:18:30.199241 data_node.go:114 Deleting volume id: 104 I0824 22:18:30.199338 topology.go:534 removing volume info: Id:104, Size:1, ReplicaPlacement:000, Collection:c, Version:3, Ttl:, FileCount:0, DeleteCount:0, DeletedByteCount:0, ReadOnly:false, ModifiedAtSecond:0 from 10.0.0.1:8080 I0824 22:18:30.199376 volume_layout.go:815 Volume 104 becomes unwritable I0824 22:18:30.199399 volume_layout.go:335 volume 104 does not have enough copies I0824 22:18:30.199543 data_node.go:114 Deleting volume id: 105 I0824 22:18:30.199656 topology.go:534 removing volume info: Id:105, Size:1, ReplicaPlacement:000, Collection:c, Version:3, Ttl:, FileCount:0, DeleteCount:0, DeletedByteCount:0, ReadOnly:false, ModifiedAtSecond:0 from 10.0.0.1:8080 I0824 22:18:30.199692 volume_layout.go:815 Volume 105 becomes unwritable I0824 22:18:30.199743 volume_layout.go:335 volume 105 does not have enough copies I0824 22:18:30.199912 data_node.go:114 Deleting volume id: 106 I0824 22:18:30.199975 topology.go:534 removing volume info: Id:106, Size:1, ReplicaPlacement:000, Collection:c, Version:3, Ttl:, FileCount:0, DeleteCount:0, DeletedByteCount:0, ReadOnly:false, ModifiedAtSecond:0 from 10.0.0.1:8080 I0824 22:18:30.200044 volume_layout.go:815 Volume 106 becomes unwritable I0824 22:18:30.200168 volume_layout.go:335 volume 106 does not have enough copies I0824 22:18:30.200275 data_node.go:114 Deleting volume id: 107 I0824 22:18:30.200353 topology.go:534 removing volume info: Id:107, Size:1, ReplicaPlacement:000, Collection:c, Version:3, Ttl:, FileCount:0, DeleteCount:0, DeletedByteCount:0, ReadOnly:false, ModifiedAtSecond:0 from 10.0.0.1:8080 I0824 22:18:30.201435 volume_layout.go:815 Volume 107 becomes unwritable I0824 22:18:30.201552 volume_layout.go:335 volume 107 does not have enough copies I0824 22:18:30.201666 data_node.go:114 Deleting volume id: 108 I0824 22:18:30.201805 topology.go:534 removing volume info: Id:108, Size:1, ReplicaPlacement:000, Collection:c, Version:3, Ttl:, FileCount:0, DeleteCount:0, DeletedByteCount:0, ReadOnly:false, ModifiedAtSecond:0 from 10.0.0.1:8080 I0824 22:18:30.201850 volume_layout.go:815 Volume 108 becomes unwritable I0824 22:18:30.201873 volume_layout.go:335 volume 108 does not have enough copies I0824 22:18:30.201974 data_node.go:114 Deleting volume id: 109 I0824 22:18:30.202072 topology.go:534 removing volume info: Id:109, Size:1, ReplicaPlacement:000, Collection:c, Version:3, Ttl:, FileCount:0, DeleteCount:0, DeletedByteCount:0, ReadOnly:false, ModifiedAtSecond:0 from 10.0.0.1:8080 I0824 22:18:30.202111 volume_layout.go:815 Volume 109 becomes unwritable I0824 22:18:30.202179 volume_layout.go:335 volume 109 does not have enough copies I0824 22:18:30.202252 data_node.go:114 Deleting volume id: 110 I0824 22:18:30.202427 topology.go:534 removing volume info: Id:110, Size:1, ReplicaPlacement:000, Collection:c, Version:3, Ttl:, FileCount:0, DeleteCount:0, DeletedByteCount:0, ReadOnly:false, ModifiedAtSecond:0 from 10.0.0.1:8080 I0824 22:18:30.202533 volume_layout.go:815 Volume 110 becomes unwritable I0824 22:18:30.202559 volume_layout.go:335 volume 110 does not have enough copies I0824 22:18:30.202649 data_node.go:114 Deleting volume id: 111 I0824 22:18:30.202716 topology.go:534 removing volume info: Id:111, Size:1, ReplicaPlacement:000, Collection:c, Version:3, Ttl:, FileCount:0, DeleteCount:0, DeletedByteCount:0, ReadOnly:false, ModifiedAtSecond:0 from 10.0.0.1:8080 I0824 22:18:30.202809 volume_layout.go:815 Volume 111 becomes unwritable I0824 22:18:30.202843 volume_layout.go:335 volume 111 does not have enough copies I0824 22:18:30.202961 data_node.go:114 Deleting volume id: 112 I0824 22:18:30.203052 topology.go:534 removing volume info: Id:112, Size:1, ReplicaPlacement:000, Collection:c, Version:3, Ttl:, FileCount:0, DeleteCount:0, DeletedByteCount:0, ReadOnly:false, ModifiedAtSecond:0 from 10.0.0.1:8080 I0824 22:18:30.203090 volume_layout.go:815 Volume 112 becomes unwritable I0824 22:18:30.203110 volume_layout.go:335 volume 112 does not have enough copies I0824 22:18:30.203201 data_node.go:114 Deleting volume id: 113 I0824 22:18:30.203340 topology.go:534 removing volume info: Id:113, Size:1, ReplicaPlacement:000, Collection:c, Version:3, Ttl:, FileCount:0, DeleteCount:0, DeletedByteCount:0, ReadOnly:false, ModifiedAtSecond:0 from 10.0.0.1:8080 I0824 22:18:30.203377 volume_layout.go:815 Volume 113 becomes unwritable I0824 22:18:30.203413 volume_layout.go:335 volume 113 does not have enough copies I0824 22:18:30.203512 data_node.go:114 Deleting volume id: 114 I0824 22:18:30.203631 topology.go:534 removing volume info: Id:114, Size:1, ReplicaPlacement:000, Collection:c, Version:3, Ttl:, FileCount:0, DeleteCount:0, DeletedByteCount:0, ReadOnly:false, ModifiedAtSecond:0 from 10.0.0.1:8080 I0824 22:18:30.203666 volume_layout.go:815 Volume 114 becomes unwritable I0824 22:18:30.203727 volume_layout.go:335 volume 114 does not have enough copies I0824 22:18:30.203828 data_node.go:114 Deleting volume id: 115 I0824 22:18:30.203926 topology.go:534 removing volume info: Id:115, Size:1, ReplicaPlacement:000, Collection:c, Version:3, Ttl:, FileCount:0, DeleteCount:0, DeletedByteCount:0, ReadOnly:false, ModifiedAtSecond:0 from 10.0.0.1:8080 I0824 22:18:30.203966 volume_layout.go:815 Volume 115 becomes unwritable I0824 22:18:30.204011 volume_layout.go:335 volume 115 does not have enough copies I0824 22:18:30.204190 data_node.go:114 Deleting volume id: 116 I0824 22:18:30.204359 topology.go:534 removing volume info: Id:116, Size:1, ReplicaPlacement:000, Collection:c, Version:3, Ttl:, FileCount:0, DeleteCount:0, DeletedByteCount:0, ReadOnly:false, ModifiedAtSecond:0 from 10.0.0.1:8080 I0824 22:18:30.204462 volume_layout.go:815 Volume 116 becomes unwritable I0824 22:18:30.205395 volume_layout.go:335 volume 116 does not have enough copies I0824 22:18:30.205705 data_node.go:114 Deleting volume id: 117 I0824 22:18:30.205916 topology.go:534 removing volume info: Id:117, Size:1, ReplicaPlacement:000, Collection:c, Version:3, Ttl:, FileCount:0, DeleteCount:0, DeletedByteCount:0, ReadOnly:false, ModifiedAtSecond:0 from 10.0.0.1:8080 I0824 22:18:30.205966 volume_layout.go:815 Volume 117 becomes unwritable I0824 22:18:30.206007 volume_layout.go:335 volume 117 does not have enough copies I0824 22:18:30.206115 data_node.go:114 Deleting volume id: 118 I0824 22:18:30.206201 topology.go:534 removing volume info: Id:118, Size:1, ReplicaPlacement:000, Collection:c, Version:3, Ttl:, FileCount:0, DeleteCount:0, DeletedByteCount:0, ReadOnly:false, ModifiedAtSecond:0 from 10.0.0.1:8080 I0824 22:18:30.206244 volume_layout.go:815 Volume 118 becomes unwritable I0824 22:18:30.206267 volume_layout.go:335 volume 118 does not have enough copies I0824 22:18:30.206354 data_node.go:114 Deleting volume id: 119 I0824 22:18:30.206464 topology.go:534 removing volume info: Id:119, Size:1, ReplicaPlacement:000, Collection:c, Version:3, Ttl:, FileCount:0, DeleteCount:0, DeletedByteCount:0, ReadOnly:false, ModifiedAtSecond:0 from 10.0.0.1:8080 I0824 22:18:30.206511 volume_layout.go:815 Volume 119 becomes unwritable I0824 22:18:30.206533 volume_layout.go:335 volume 119 does not have enough copies I0824 22:18:30.206707 data_node.go:114 Deleting volume id: 120 I0824 22:18:30.206811 topology.go:534 removing volume info: Id:120, Size:1, ReplicaPlacement:000, Collection:c, Version:3, Ttl:, FileCount:0, DeleteCount:0, DeletedByteCount:0, ReadOnly:false, ModifiedAtSecond:0 from 10.0.0.1:8080 I0824 22:18:30.206867 volume_layout.go:815 Volume 120 becomes unwritable I0824 22:18:30.206907 volume_layout.go:335 volume 120 does not have enough copies I0824 22:18:30.207028 data_node.go:114 Deleting volume id: 121 I0824 22:18:30.207149 topology.go:534 removing volume info: Id:121, Size:1, ReplicaPlacement:000, Collection:c, Version:3, Ttl:, FileCount:0, DeleteCount:0, DeletedByteCount:0, ReadOnly:false, ModifiedAtSecond:0 from 10.0.0.1:8080 I0824 22:18:30.207277 volume_layout.go:815 Volume 121 becomes unwritable I0824 22:18:30.207303 volume_layout.go:335 volume 121 does not have enough copies I0824 22:18:30.207415 data_node.go:114 Deleting volume id: 122 I0824 22:18:30.208743 topology.go:534 removing volume info: Id:122, Size:1, ReplicaPlacement:000, Collection:c, Version:3, Ttl:, FileCount:0, DeleteCount:0, DeletedByteCount:0, ReadOnly:false, ModifiedAtSecond:0 from 10.0.0.1:8080 I0824 22:18:30.208948 volume_layout.go:815 Volume 122 becomes unwritable I0824 22:18:30.208996 volume_layout.go:335 volume 122 does not have enough copies I0824 22:18:30.209145 data_node.go:114 Deleting volume id: 123 I0824 22:18:30.209287 topology.go:534 removing volume info: Id:123, Size:1, ReplicaPlacement:000, Collection:c, Version:3, Ttl:, FileCount:0, DeleteCount:0, DeletedByteCount:0, ReadOnly:false, ModifiedAtSecond:0 from 10.0.0.1:8080 I0824 22:18:30.209330 volume_layout.go:815 Volume 123 becomes unwritable I0824 22:18:30.209353 volume_layout.go:335 volume 123 does not have enough copies I0824 22:18:30.209510 data_node.go:114 Deleting volume id: 124 I0824 22:18:30.209637 topology.go:534 removing volume info: Id:124, Size:1, ReplicaPlacement:000, Collection:c, Version:3, Ttl:, FileCount:0, DeleteCount:0, DeletedByteCount:0, ReadOnly:false, ModifiedAtSecond:0 from 10.0.0.1:8080 I0824 22:18:30.209688 volume_layout.go:815 Volume 124 becomes unwritable I0824 22:18:30.209710 volume_layout.go:335 volume 124 does not have enough copies I0824 22:18:30.209798 data_node.go:114 Deleting volume id: 125 I0824 22:18:30.209851 topology.go:534 removing volume info: Id:125, Size:1, ReplicaPlacement:000, Collection:c, Version:3, Ttl:, FileCount:0, DeleteCount:0, DeletedByteCount:0, ReadOnly:false, ModifiedAtSecond:0 from 10.0.0.1:8080 I0824 22:18:30.209954 volume_layout.go:815 Volume 125 becomes unwritable I0824 22:18:30.210856 volume_layout.go:335 volume 125 does not have enough copies I0824 22:18:30.211050 data_node.go:114 Deleting volume id: 126 I0824 22:18:30.211175 topology.go:534 removing volume info: Id:126, Size:1, ReplicaPlacement:000, Collection:c, Version:3, Ttl:, FileCount:0, DeleteCount:0, DeletedByteCount:0, ReadOnly:false, ModifiedAtSecond:0 from 10.0.0.1:8080 I0824 22:18:30.211276 volume_layout.go:815 Volume 126 becomes unwritable I0824 22:18:30.211300 volume_layout.go:335 volume 126 does not have enough copies I0824 22:18:30.211450 data_node.go:114 Deleting volume id: 127 I0824 22:18:30.211580 topology.go:534 removing volume info: Id:127, Size:1, ReplicaPlacement:000, Collection:c, Version:3, Ttl:, FileCount:0, DeleteCount:0, DeletedByteCount:0, ReadOnly:false, ModifiedAtSecond:0 from 10.0.0.1:8080 I0824 22:18:30.211712 volume_layout.go:815 Volume 127 becomes unwritable I0824 22:18:30.211737 volume_layout.go:335 volume 127 does not have enough copies I0824 22:18:30.211842 data_node.go:114 Deleting volume id: 128 I0824 22:18:30.211898 topology.go:534 removing volume info: Id:128, Size:1, ReplicaPlacement:000, Collection:c, Version:3, Ttl:, FileCount:0, DeleteCount:0, DeletedByteCount:0, ReadOnly:false, ModifiedAtSecond:0 from 10.0.0.1:8080 I0824 22:18:30.211951 volume_layout.go:815 Volume 128 becomes unwritable I0824 22:18:30.211972 volume_layout.go:335 volume 128 does not have enough copies I0824 22:18:30.212113 data_node.go:114 Deleting volume id: 129 I0824 22:18:30.212217 topology.go:534 removing volume info: Id:129, Size:1, ReplicaPlacement:000, Collection:c, Version:3, Ttl:, FileCount:0, DeleteCount:0, DeletedByteCount:0, ReadOnly:false, ModifiedAtSecond:0 from 10.0.0.1:8080 I0824 22:18:30.212273 volume_layout.go:815 Volume 129 becomes unwritable I0824 22:18:30.212329 volume_layout.go:335 volume 129 does not have enough copies I0824 22:18:30.212400 data_node.go:114 Deleting volume id: 130 I0824 22:18:30.212447 topology.go:534 removing volume info: Id:130, Size:1, ReplicaPlacement:000, Collection:c, Version:3, Ttl:, FileCount:0, DeleteCount:0, DeletedByteCount:0, ReadOnly:false, ModifiedAtSecond:0 from 10.0.0.1:8080 I0824 22:18:30.212479 volume_layout.go:815 Volume 130 becomes unwritable I0824 22:18:30.212500 volume_layout.go:335 volume 130 does not have enough copies I0824 22:18:30.212641 data_node.go:114 Deleting volume id: 131 I0824 22:18:30.212788 topology.go:534 removing volume info: Id:131, Size:1, ReplicaPlacement:000, Collection:c, Version:3, Ttl:, FileCount:0, DeleteCount:0, DeletedByteCount:0, ReadOnly:false, ModifiedAtSecond:0 from 10.0.0.1:8080 I0824 22:18:30.212828 volume_layout.go:815 Volume 131 becomes unwritable I0824 22:18:30.212869 volume_layout.go:335 volume 131 does not have enough copies I0824 22:18:30.212984 data_node.go:114 Deleting volume id: 132 I0824 22:18:30.213143 topology.go:534 removing volume info: Id:132, Size:1, ReplicaPlacement:000, Collection:c, Version:3, Ttl:, FileCount:0, DeleteCount:0, DeletedByteCount:0, ReadOnly:false, ModifiedAtSecond:0 from 10.0.0.1:8080 I0824 22:18:30.213184 volume_layout.go:815 Volume 132 becomes unwritable I0824 22:18:30.213238 volume_layout.go:335 volume 132 does not have enough copies I0824 22:18:30.213291 data_node.go:114 Deleting volume id: 133 I0824 22:18:30.213400 topology.go:534 removing volume info: Id:133, Size:1, ReplicaPlacement:000, Collection:c, Version:3, Ttl:, FileCount:0, DeleteCount:0, DeletedByteCount:0, ReadOnly:false, ModifiedAtSecond:0 from 10.0.0.1:8080 I0824 22:18:30.213439 volume_layout.go:815 Volume 133 becomes unwritable I0824 22:18:30.213459 volume_layout.go:335 volume 133 does not have enough copies I0824 22:18:30.213535 data_node.go:114 Deleting volume id: 134 I0824 22:18:30.213705 topology.go:534 removing volume info: Id:134, Size:1, ReplicaPlacement:000, Collection:c, Version:3, Ttl:, FileCount:0, DeleteCount:0, DeletedByteCount:0, ReadOnly:false, ModifiedAtSecond:0 from 10.0.0.1:8080 I0824 22:18:30.213751 volume_layout.go:815 Volume 134 becomes unwritable I0824 22:18:30.213773 volume_layout.go:335 volume 134 does not have enough copies I0824 22:18:30.215213 data_node.go:114 Deleting volume id: 135 I0824 22:18:30.215335 topology.go:534 removing volume info: Id:135, Size:1, ReplicaPlacement:000, Collection:c, Version:3, Ttl:, FileCount:0, DeleteCount:0, DeletedByteCount:0, ReadOnly:false, ModifiedAtSecond:0 from 10.0.0.1:8080 I0824 22:18:30.215483 volume_layout.go:815 Volume 135 becomes unwritable I0824 22:18:30.215513 volume_layout.go:335 volume 135 does not have enough copies I0824 22:18:30.215594 data_node.go:114 Deleting volume id: 136 I0824 22:18:30.215665 topology.go:534 removing volume info: Id:136, Size:1, ReplicaPlacement:000, Collection:c, Version:3, Ttl:, FileCount:0, DeleteCount:0, DeletedByteCount:0, ReadOnly:false, ModifiedAtSecond:0 from 10.0.0.1:8080 I0824 22:18:30.215774 volume_layout.go:815 Volume 136 becomes unwritable I0824 22:18:30.215798 volume_layout.go:335 volume 136 does not have enough copies I0824 22:18:30.215879 data_node.go:114 Deleting volume id: 137 I0824 22:18:30.215955 topology.go:534 removing volume info: Id:137, Size:1, ReplicaPlacement:000, Collection:c, Version:3, Ttl:, FileCount:0, DeleteCount:0, DeletedByteCount:0, ReadOnly:false, ModifiedAtSecond:0 from 10.0.0.1:8080 I0824 22:18:30.215991 volume_layout.go:815 Volume 137 becomes unwritable I0824 22:18:30.216014 volume_layout.go:335 volume 137 does not have enough copies I0824 22:18:30.216639 data_node.go:114 Deleting volume id: 138 I0824 22:18:30.217556 topology.go:534 removing volume info: Id:138, Size:1, ReplicaPlacement:000, Collection:c, Version:3, Ttl:, FileCount:0, DeleteCount:0, DeletedByteCount:0, ReadOnly:false, ModifiedAtSecond:0 from 10.0.0.1:8080 I0824 22:18:30.218072 volume_layout.go:815 Volume 138 becomes unwritable I0824 22:18:30.218286 volume_layout.go:335 volume 138 does not have enough copies I0824 22:18:30.218841 data_node.go:114 Deleting volume id: 139 I0824 22:18:30.219190 topology.go:534 removing volume info: Id:139, Size:1, ReplicaPlacement:000, Collection:c, Version:3, Ttl:, FileCount:0, DeleteCount:0, DeletedByteCount:0, ReadOnly:false, ModifiedAtSecond:0 from 10.0.0.1:8080 I0824 22:18:30.219491 volume_layout.go:815 Volume 139 becomes unwritable I0824 22:18:30.219663 volume_layout.go:335 volume 139 does not have enough copies I0824 22:18:30.220010 data_node.go:114 Deleting volume id: 140 I0824 22:18:30.220283 topology.go:534 removing volume info: Id:140, Size:1, ReplicaPlacement:000, Collection:c, Version:3, Ttl:, FileCount:0, DeleteCount:0, DeletedByteCount:0, ReadOnly:false, ModifiedAtSecond:0 from 10.0.0.1:8080 I0824 22:18:30.220515 volume_layout.go:815 Volume 140 becomes unwritable I0824 22:18:30.220560 volume_layout.go:335 volume 140 does not have enough copies I0824 22:18:30.220853 data_node.go:114 Deleting volume id: 141 I0824 22:18:30.221068 topology.go:534 removing volume info: Id:141, Size:1, ReplicaPlacement:000, Collection:c, Version:3, Ttl:, FileCount:0, DeleteCount:0, DeletedByteCount:0, ReadOnly:false, ModifiedAtSecond:0 from 10.0.0.1:8080 I0824 22:18:30.221342 volume_layout.go:815 Volume 141 becomes unwritable I0824 22:18:30.221474 volume_layout.go:335 volume 141 does not have enough copies I0824 22:18:30.221705 data_node.go:114 Deleting volume id: 142 I0824 22:18:30.221912 topology.go:534 removing volume info: Id:142, Size:1, ReplicaPlacement:000, Collection:c, Version:3, Ttl:, FileCount:0, DeleteCount:0, DeletedByteCount:0, ReadOnly:false, ModifiedAtSecond:0 from 10.0.0.1:8080 I0824 22:18:30.221986 volume_layout.go:815 Volume 142 becomes unwritable I0824 22:18:30.222026 volume_layout.go:335 volume 142 does not have enough copies I0824 22:18:30.222310 data_node.go:114 Deleting volume id: 143 I0824 22:18:30.222585 topology.go:534 removing volume info: Id:143, Size:1, ReplicaPlacement:000, Collection:c, Version:3, Ttl:, FileCount:0, DeleteCount:0, DeletedByteCount:0, ReadOnly:false, ModifiedAtSecond:0 from 10.0.0.1:8080 I0824 22:18:30.222798 volume_layout.go:815 Volume 143 becomes unwritable I0824 22:18:30.222852 volume_layout.go:335 volume 143 does not have enough copies I0824 22:18:30.223067 data_node.go:114 Deleting volume id: 144 I0824 22:18:30.225755 topology.go:534 removing volume info: Id:144, Size:1, ReplicaPlacement:000, Collection:c, Version:3, Ttl:, FileCount:0, DeleteCount:0, DeletedByteCount:0, ReadOnly:false, ModifiedAtSecond:0 from 10.0.0.1:8080 I0824 22:18:30.225988 volume_layout.go:815 Volume 144 becomes unwritable I0824 22:18:30.226027 volume_layout.go:335 volume 144 does not have enough copies I0824 22:18:30.226102 data_node.go:114 Deleting volume id: 145 I0824 22:18:30.226162 topology.go:534 removing volume info: Id:145, Size:1, ReplicaPlacement:000, Collection:c, Version:3, Ttl:, FileCount:0, DeleteCount:0, DeletedByteCount:0, ReadOnly:false, ModifiedAtSecond:0 from 10.0.0.1:8080 I0824 22:18:30.226296 volume_layout.go:815 Volume 145 becomes unwritable I0824 22:18:30.226322 volume_layout.go:335 volume 145 does not have enough copies I0824 22:18:30.226442 data_node.go:114 Deleting volume id: 146 I0824 22:18:30.226522 topology.go:534 removing volume info: Id:146, Size:1, ReplicaPlacement:000, Collection:c, Version:3, Ttl:, FileCount:0, DeleteCount:0, DeletedByteCount:0, ReadOnly:false, ModifiedAtSecond:0 from 10.0.0.1:8080 I0824 22:18:30.226563 volume_layout.go:815 Volume 146 becomes unwritable I0824 22:18:30.226586 volume_layout.go:335 volume 146 does not have enough copies I0824 22:18:30.226686 data_node.go:114 Deleting volume id: 147 I0824 22:18:30.226928 topology.go:534 removing volume info: Id:147, Size:1, ReplicaPlacement:000, Collection:c, Version:3, Ttl:, FileCount:0, DeleteCount:0, DeletedByteCount:0, ReadOnly:false, ModifiedAtSecond:0 from 10.0.0.1:8080 I0824 22:18:30.227014 volume_layout.go:815 Volume 147 becomes unwritable I0824 22:18:30.227043 volume_layout.go:335 volume 147 does not have enough copies I0824 22:18:30.227146 data_node.go:114 Deleting volume id: 148 I0824 22:18:30.227233 topology.go:534 removing volume info: Id:148, Size:1, ReplicaPlacement:000, Collection:c, Version:3, Ttl:, FileCount:0, DeleteCount:0, DeletedByteCount:0, ReadOnly:false, ModifiedAtSecond:0 from 10.0.0.1:8080 I0824 22:18:30.227279 volume_layout.go:815 Volume 148 becomes unwritable I0824 22:18:30.227307 volume_layout.go:335 volume 148 does not have enough copies I0824 22:18:30.227403 data_node.go:114 Deleting volume id: 149 I0824 22:18:30.227515 topology.go:534 removing volume info: Id:149, Size:1, ReplicaPlacement:000, Collection:c, Version:3, Ttl:, FileCount:0, DeleteCount:0, DeletedByteCount:0, ReadOnly:false, ModifiedAtSecond:0 from 10.0.0.1:8080 I0824 22:18:30.227656 volume_layout.go:815 Volume 149 becomes unwritable I0824 22:18:30.227690 volume_layout.go:335 volume 149 does not have enough copies I0824 22:18:30.227821 data_node.go:114 Deleting volume id: 100 I0824 22:18:30.228009 topology.go:534 removing volume info: Id:100, Size:1, ReplicaPlacement:000, Collection:c, Version:3, Ttl:, FileCount:0, DeleteCount:0, DeletedByteCount:0, ReadOnly:false, ModifiedAtSecond:0 from 10.0.0.1:8080 I0824 22:18:30.228307 volume_layout.go:815 Volume 100 becomes unwritable I0824 22:18:30.228350 volume_layout.go:335 volume 100 does not have enough copies I0824 22:18:30.228502 data_node.go:114 Deleting volume id: 101 I0824 22:18:30.228588 topology.go:534 removing volume info: Id:101, Size:1, ReplicaPlacement:000, Collection:c, Version:3, Ttl:, FileCount:0, DeleteCount:0, DeletedByteCount:0, ReadOnly:false, ModifiedAtSecond:0 from 10.0.0.1:8080 I0824 22:18:30.228732 volume_layout.go:815 Volume 101 becomes unwritable I0824 22:18:30.228839 volume_layout.go:335 volume 101 does not have enough copies I0824 22:18:30.228918 data_node.go:114 Deleting volume id: 102 I0824 22:18:30.229002 topology.go:534 removing volume info: Id:102, Size:1, ReplicaPlacement:000, Collection:c, Version:3, Ttl:, FileCount:0, DeleteCount:0, DeletedByteCount:0, ReadOnly:false, ModifiedAtSecond:0 from 10.0.0.1:8080 I0824 22:18:30.229091 volume_layout.go:815 Volume 102 becomes unwritable I0824 22:18:30.229141 volume_layout.go:335 volume 102 does not have enough copies I0824 22:18:30.229519 data_node.go:114 Deleting volume id: 103 I0824 22:18:30.229652 topology.go:534 removing volume info: Id:103, Size:1, ReplicaPlacement:000, Collection:c, Version:3, Ttl:, FileCount:0, DeleteCount:0, DeletedByteCount:0, ReadOnly:false, ModifiedAtSecond:0 from 10.0.0.1:8080 I0824 22:18:30.229737 volume_layout.go:815 Volume 103 becomes unwritable I0824 22:18:30.229774 volume_layout.go:335 volume 103 does not have enough copies I0824 22:18:30.229972 data_node.go:114 Deleting volume id: 104 I0824 22:18:30.230114 topology.go:534 removing volume info: Id:104, Size:1, ReplicaPlacement:000, Collection:c, Version:3, Ttl:, FileCount:0, DeleteCount:0, DeletedByteCount:0, ReadOnly:false, ModifiedAtSecond:0 from 10.0.0.1:8080 I0824 22:18:30.230209 volume_layout.go:815 Volume 104 becomes unwritable I0824 22:18:30.230286 volume_layout.go:335 volume 104 does not have enough copies I0824 22:18:30.230415 data_node.go:114 Deleting volume id: 105 I0824 22:18:30.230474 topology.go:534 removing volume info: Id:105, Size:1, ReplicaPlacement:000, Collection:c, Version:3, Ttl:, FileCount:0, DeleteCount:0, DeletedByteCount:0, ReadOnly:false, ModifiedAtSecond:0 from 10.0.0.1:8080 I0824 22:18:30.230510 volume_layout.go:815 Volume 105 becomes unwritable I0824 22:18:30.230531 volume_layout.go:335 volume 105 does not have enough copies I0824 22:18:30.230603 data_node.go:114 Deleting volume id: 106 I0824 22:18:30.230651 topology.go:534 removing volume info: Id:106, Size:1, ReplicaPlacement:000, Collection:c, Version:3, Ttl:, FileCount:0, DeleteCount:0, DeletedByteCount:0, ReadOnly:false, ModifiedAtSecond:0 from 10.0.0.1:8080 I0824 22:18:30.230852 volume_layout.go:815 Volume 106 becomes unwritable I0824 22:18:30.230890 volume_layout.go:335 volume 106 does not have enough copies I0824 22:18:30.230964 data_node.go:114 Deleting volume id: 107 I0824 22:18:30.231025 topology.go:534 removing volume info: Id:107, Size:1, ReplicaPlacement:000, Collection:c, Version:3, Ttl:, FileCount:0, DeleteCount:0, DeletedByteCount:0, ReadOnly:false, ModifiedAtSecond:0 from 10.0.0.1:8080 I0824 22:18:30.231060 volume_layout.go:815 Volume 107 becomes unwritable I0824 22:18:30.231079 volume_layout.go:335 volume 107 does not have enough copies I0824 22:18:30.231150 data_node.go:114 Deleting volume id: 108 I0824 22:18:30.231288 topology.go:534 removing volume info: Id:108, Size:1, ReplicaPlacement:000, Collection:c, Version:3, Ttl:, FileCount:0, DeleteCount:0, DeletedByteCount:0, ReadOnly:false, ModifiedAtSecond:0 from 10.0.0.1:8080 I0824 22:18:30.231364 volume_layout.go:815 Volume 108 becomes unwritable I0824 22:18:30.231383 volume_layout.go:335 volume 108 does not have enough copies I0824 22:18:30.231573 data_node.go:114 Deleting volume id: 109 I0824 22:18:30.231690 topology.go:534 removing volume info: Id:109, Size:1, ReplicaPlacement:000, Collection:c, Version:3, Ttl:, FileCount:0, DeleteCount:0, DeletedByteCount:0, ReadOnly:false, ModifiedAtSecond:0 from 10.0.0.1:8080 I0824 22:18:30.231762 volume_layout.go:815 Volume 109 becomes unwritable I0824 22:18:30.231796 volume_layout.go:335 volume 109 does not have enough copies I0824 22:18:30.232004 data_node.go:114 Deleting volume id: 110 I0824 22:18:30.232103 topology.go:534 removing volume info: Id:110, Size:1, ReplicaPlacement:000, Collection:c, Version:3, Ttl:, FileCount:0, DeleteCount:0, DeletedByteCount:0, ReadOnly:false, ModifiedAtSecond:0 from 10.0.0.1:8080 I0824 22:18:30.232206 volume_layout.go:815 Volume 110 becomes unwritable I0824 22:18:30.232229 volume_layout.go:335 volume 110 does not have enough copies I0824 22:18:30.232303 data_node.go:114 Deleting volume id: 111 I0824 22:18:30.232371 topology.go:534 removing volume info: Id:111, Size:1, ReplicaPlacement:000, Collection:c, Version:3, Ttl:, FileCount:0, DeleteCount:0, DeletedByteCount:0, ReadOnly:false, ModifiedAtSecond:0 from 10.0.0.1:8080 I0824 22:18:30.232405 volume_layout.go:815 Volume 111 becomes unwritable I0824 22:18:30.232423 volume_layout.go:335 volume 111 does not have enough copies I0824 22:18:30.232492 data_node.go:114 Deleting volume id: 112 I0824 22:18:30.232595 topology.go:534 removing volume info: Id:112, Size:1, ReplicaPlacement:000, Collection:c, Version:3, Ttl:, FileCount:0, DeleteCount:0, DeletedByteCount:0, ReadOnly:false, ModifiedAtSecond:0 from 10.0.0.1:8080 I0824 22:18:30.232644 volume_layout.go:815 Volume 112 becomes unwritable I0824 22:18:30.232668 volume_layout.go:335 volume 112 does not have enough copies I0824 22:18:30.232725 data_node.go:114 Deleting volume id: 113 I0824 22:18:30.232778 topology.go:534 removing volume info: Id:113, Size:1, ReplicaPlacement:000, Collection:c, Version:3, Ttl:, FileCount:0, DeleteCount:0, DeletedByteCount:0, ReadOnly:false, ModifiedAtSecond:0 from 10.0.0.1:8080 I0824 22:18:30.232812 volume_layout.go:815 Volume 113 becomes unwritable I0824 22:18:30.232833 volume_layout.go:335 volume 113 does not have enough copies I0824 22:18:30.232900 data_node.go:114 Deleting volume id: 114 I0824 22:18:30.232943 topology.go:534 removing volume info: Id:114, Size:1, ReplicaPlacement:000, Collection:c, Version:3, Ttl:, FileCount:0, DeleteCount:0, DeletedByteCount:0, ReadOnly:false, ModifiedAtSecond:0 from 10.0.0.1:8080 I0824 22:18:30.233180 volume_layout.go:815 Volume 114 becomes unwritable I0824 22:18:30.233205 volume_layout.go:335 volume 114 does not have enough copies I0824 22:18:30.233414 data_node.go:114 Deleting volume id: 115 I0824 22:18:30.233565 topology.go:534 removing volume info: Id:115, Size:1, ReplicaPlacement:000, Collection:c, Version:3, Ttl:, FileCount:0, DeleteCount:0, DeletedByteCount:0, ReadOnly:false, ModifiedAtSecond:0 from 10.0.0.1:8080 I0824 22:18:30.234513 volume_layout.go:815 Volume 115 becomes unwritable I0824 22:18:30.234618 volume_layout.go:335 volume 115 does not have enough copies I0824 22:18:30.234712 data_node.go:114 Deleting volume id: 116 I0824 22:18:30.234882 topology.go:534 removing volume info: Id:116, Size:1, ReplicaPlacement:000, Collection:c, Version:3, Ttl:, FileCount:0, DeleteCount:0, DeletedByteCount:0, ReadOnly:false, ModifiedAtSecond:0 from 10.0.0.1:8080 I0824 22:18:30.234998 volume_layout.go:815 Volume 116 becomes unwritable I0824 22:18:30.235034 volume_layout.go:335 volume 116 does not have enough copies I0824 22:18:30.235162 data_node.go:114 Deleting volume id: 117 I0824 22:18:30.235237 topology.go:534 removing volume info: Id:117, Size:1, ReplicaPlacement:000, Collection:c, Version:3, Ttl:, FileCount:0, DeleteCount:0, DeletedByteCount:0, ReadOnly:false, ModifiedAtSecond:0 from 10.0.0.1:8080 I0824 22:18:30.235280 volume_layout.go:815 Volume 117 becomes unwritable I0824 22:18:30.235300 volume_layout.go:335 volume 117 does not have enough copies I0824 22:18:30.235369 data_node.go:114 Deleting volume id: 118 I0824 22:18:30.235536 topology.go:534 removing volume info: Id:118, Size:1, ReplicaPlacement:000, Collection:c, Version:3, Ttl:, FileCount:0, DeleteCount:0, DeletedByteCount:0, ReadOnly:false, ModifiedAtSecond:0 from 10.0.0.1:8080 I0824 22:18:30.235642 volume_layout.go:815 Volume 118 becomes unwritable I0824 22:18:30.235697 volume_layout.go:335 volume 118 does not have enough copies I0824 22:18:30.235839 data_node.go:114 Deleting volume id: 119 I0824 22:18:30.235884 topology.go:534 removing volume info: Id:119, Size:1, ReplicaPlacement:000, Collection:c, Version:3, Ttl:, FileCount:0, DeleteCount:0, DeletedByteCount:0, ReadOnly:false, ModifiedAtSecond:0 from 10.0.0.1:8080 I0824 22:18:30.235918 volume_layout.go:815 Volume 119 becomes unwritable I0824 22:18:30.235939 volume_layout.go:335 volume 119 does not have enough copies I0824 22:18:30.236049 data_node.go:114 Deleting volume id: 120 I0824 22:18:30.236112 topology.go:534 removing volume info: Id:120, Size:1, ReplicaPlacement:000, Collection:c, Version:3, Ttl:, FileCount:0, DeleteCount:0, DeletedByteCount:0, ReadOnly:false, ModifiedAtSecond:0 from 10.0.0.1:8080 I0824 22:18:30.236159 volume_layout.go:815 Volume 120 becomes unwritable I0824 22:18:30.236195 volume_layout.go:335 volume 120 does not have enough copies I0824 22:18:30.236284 data_node.go:114 Deleting volume id: 121 I0824 22:18:30.236363 topology.go:534 removing volume info: Id:121, Size:1, ReplicaPlacement:000, Collection:c, Version:3, Ttl:, FileCount:0, DeleteCount:0, DeletedByteCount:0, ReadOnly:false, ModifiedAtSecond:0 from 10.0.0.1:8080 I0824 22:18:30.236412 volume_layout.go:815 Volume 121 becomes unwritable I0824 22:18:30.236432 volume_layout.go:335 volume 121 does not have enough copies I0824 22:18:30.236559 data_node.go:114 Deleting volume id: 122 I0824 22:18:30.236651 topology.go:534 removing volume info: Id:122, Size:1, ReplicaPlacement:000, Collection:c, Version:3, Ttl:, FileCount:0, DeleteCount:0, DeletedByteCount:0, ReadOnly:false, ModifiedAtSecond:0 from 10.0.0.1:8080 I0824 22:18:30.236684 volume_layout.go:815 Volume 122 becomes unwritable I0824 22:18:30.236704 volume_layout.go:335 volume 122 does not have enough copies I0824 22:18:30.236770 data_node.go:114 Deleting volume id: 123 I0824 22:18:30.236981 topology.go:534 removing volume info: Id:123, Size:1, ReplicaPlacement:000, Collection:c, Version:3, Ttl:, FileCount:0, DeleteCount:0, DeletedByteCount:0, ReadOnly:false, ModifiedAtSecond:0 from 10.0.0.1:8080 I0824 22:18:30.237053 volume_layout.go:815 Volume 123 becomes unwritable I0824 22:18:30.237073 volume_layout.go:335 volume 123 does not have enough copies I0824 22:18:30.237184 data_node.go:114 Deleting volume id: 124 I0824 22:18:30.237246 topology.go:534 removing volume info: Id:124, Size:1, ReplicaPlacement:000, Collection:c, Version:3, Ttl:, FileCount:0, DeleteCount:0, DeletedByteCount:0, ReadOnly:false, ModifiedAtSecond:0 from 10.0.0.1:8080 I0824 22:18:30.237279 volume_layout.go:815 Volume 124 becomes unwritable I0824 22:18:30.237297 volume_layout.go:335 volume 124 does not have enough copies I0824 22:18:30.237378 data_node.go:114 Deleting volume id: 125 I0824 22:18:30.237447 topology.go:534 removing volume info: Id:125, Size:1, ReplicaPlacement:000, Collection:c, Version:3, Ttl:, FileCount:0, DeleteCount:0, DeletedByteCount:0, ReadOnly:false, ModifiedAtSecond:0 from 10.0.0.1:8080 I0824 22:18:30.237497 volume_layout.go:815 Volume 125 becomes unwritable I0824 22:18:30.237523 volume_layout.go:335 volume 125 does not have enough copies I0824 22:18:30.237579 data_node.go:114 Deleting volume id: 126 I0824 22:18:30.237620 topology.go:534 removing volume info: Id:126, Size:1, ReplicaPlacement:000, Collection:c, Version:3, Ttl:, FileCount:0, DeleteCount:0, DeletedByteCount:0, ReadOnly:false, ModifiedAtSecond:0 from 10.0.0.1:8080 I0824 22:18:30.237649 volume_layout.go:815 Volume 126 becomes unwritable I0824 22:18:30.237667 volume_layout.go:335 volume 126 does not have enough copies I0824 22:18:30.237728 data_node.go:114 Deleting volume id: 127 I0824 22:18:30.237773 topology.go:534 removing volume info: Id:127, Size:1, ReplicaPlacement:000, Collection:c, Version:3, Ttl:, FileCount:0, DeleteCount:0, DeletedByteCount:0, ReadOnly:false, ModifiedAtSecond:0 from 10.0.0.1:8080 I0824 22:18:30.237803 volume_layout.go:815 Volume 127 becomes unwritable I0824 22:18:30.237823 volume_layout.go:335 volume 127 does not have enough copies I0824 22:18:30.237893 data_node.go:114 Deleting volume id: 128 I0824 22:18:30.237936 topology.go:534 removing volume info: Id:128, Size:1, ReplicaPlacement:000, Collection:c, Version:3, Ttl:, FileCount:0, DeleteCount:0, DeletedByteCount:0, ReadOnly:false, ModifiedAtSecond:0 from 10.0.0.1:8080 I0824 22:18:30.237983 volume_layout.go:815 Volume 128 becomes unwritable I0824 22:18:30.238025 volume_layout.go:335 volume 128 does not have enough copies I0824 22:18:30.238130 data_node.go:114 Deleting volume id: 129 I0824 22:18:30.238193 topology.go:534 removing volume info: Id:129, Size:1, ReplicaPlacement:000, Collection:c, Version:3, Ttl:, FileCount:0, DeleteCount:0, DeletedByteCount:0, ReadOnly:false, ModifiedAtSecond:0 from 10.0.0.1:8080 I0824 22:18:30.238291 volume_layout.go:815 Volume 129 becomes unwritable I0824 22:18:30.238313 volume_layout.go:335 volume 129 does not have enough copies I0824 22:18:30.238411 data_node.go:114 Deleting volume id: 130 I0824 22:18:30.238472 topology.go:534 removing volume info: Id:130, Size:1, ReplicaPlacement:000, Collection:c, Version:3, Ttl:, FileCount:0, DeleteCount:0, DeletedByteCount:0, ReadOnly:false, ModifiedAtSecond:0 from 10.0.0.1:8080 I0824 22:18:30.238513 volume_layout.go:815 Volume 130 becomes unwritable I0824 22:18:30.238532 volume_layout.go:335 volume 130 does not have enough copies I0824 22:18:30.238611 data_node.go:114 Deleting volume id: 131 I0824 22:18:30.238662 topology.go:534 removing volume info: Id:131, Size:1, ReplicaPlacement:000, Collection:c, Version:3, Ttl:, FileCount:0, DeleteCount:0, DeletedByteCount:0, ReadOnly:false, ModifiedAtSecond:0 from 10.0.0.1:8080 I0824 22:18:30.238692 volume_layout.go:815 Volume 131 becomes unwritable I0824 22:18:30.238712 volume_layout.go:335 volume 131 does not have enough copies I0824 22:18:30.238798 data_node.go:114 Deleting volume id: 132 I0824 22:18:30.238890 topology.go:534 removing volume info: Id:132, Size:1, ReplicaPlacement:000, Collection:c, Version:3, Ttl:, FileCount:0, DeleteCount:0, DeletedByteCount:0, ReadOnly:false, ModifiedAtSecond:0 from 10.0.0.1:8080 I0824 22:18:30.238935 volume_layout.go:815 Volume 132 becomes unwritable I0824 22:18:30.238953 volume_layout.go:335 volume 132 does not have enough copies I0824 22:18:30.239038 data_node.go:114 Deleting volume id: 133 I0824 22:18:30.239081 topology.go:534 removing volume info: Id:133, Size:1, ReplicaPlacement:000, Collection:c, Version:3, Ttl:, FileCount:0, DeleteCount:0, DeletedByteCount:0, ReadOnly:false, ModifiedAtSecond:0 from 10.0.0.1:8080 I0824 22:18:30.239112 volume_layout.go:815 Volume 133 becomes unwritable I0824 22:18:30.239131 volume_layout.go:335 volume 133 does not have enough copies I0824 22:18:30.239187 data_node.go:114 Deleting volume id: 134 I0824 22:18:30.239228 topology.go:534 removing volume info: Id:134, Size:1, ReplicaPlacement:000, Collection:c, Version:3, Ttl:, FileCount:0, DeleteCount:0, DeletedByteCount:0, ReadOnly:false, ModifiedAtSecond:0 from 10.0.0.1:8080 I0824 22:18:30.239258 volume_layout.go:815 Volume 134 becomes unwritable I0824 22:18:30.239278 volume_layout.go:335 volume 134 does not have enough copies I0824 22:18:30.239350 data_node.go:114 Deleting volume id: 135 I0824 22:18:30.239407 topology.go:534 removing volume info: Id:135, Size:1, ReplicaPlacement:000, Collection:c, Version:3, Ttl:, FileCount:0, DeleteCount:0, DeletedByteCount:0, ReadOnly:false, ModifiedAtSecond:0 from 10.0.0.1:8080 I0824 22:18:30.239437 volume_layout.go:815 Volume 135 becomes unwritable I0824 22:18:30.239523 volume_layout.go:335 volume 135 does not have enough copies I0824 22:18:30.239657 data_node.go:114 Deleting volume id: 136 I0824 22:18:30.239699 topology.go:534 removing volume info: Id:136, Size:1, ReplicaPlacement:000, Collection:c, Version:3, Ttl:, FileCount:0, DeleteCount:0, DeletedByteCount:0, ReadOnly:false, ModifiedAtSecond:0 from 10.0.0.1:8080 I0824 22:18:30.239729 volume_layout.go:815 Volume 136 becomes unwritable I0824 22:18:30.239764 volume_layout.go:335 volume 136 does not have enough copies I0824 22:18:30.239826 data_node.go:114 Deleting volume id: 137 I0824 22:18:30.239866 topology.go:534 removing volume info: Id:137, Size:1, ReplicaPlacement:000, Collection:c, Version:3, Ttl:, FileCount:0, DeleteCount:0, DeletedByteCount:0, ReadOnly:false, ModifiedAtSecond:0 from 10.0.0.1:8080 I0824 22:18:30.239942 volume_layout.go:815 Volume 137 becomes unwritable I0824 22:18:30.239966 volume_layout.go:335 volume 137 does not have enough copies I0824 22:18:30.240090 data_node.go:114 Deleting volume id: 138 I0824 22:18:30.240175 topology.go:534 removing volume info: Id:138, Size:1, ReplicaPlacement:000, Collection:c, Version:3, Ttl:, FileCount:0, DeleteCount:0, DeletedByteCount:0, ReadOnly:false, ModifiedAtSecond:0 from 10.0.0.1:8080 I0824 22:18:30.240207 volume_layout.go:815 Volume 138 becomes unwritable I0824 22:18:30.240225 volume_layout.go:335 volume 138 does not have enough copies I0824 22:18:30.240286 data_node.go:114 Deleting volume id: 139 I0824 22:18:30.240334 topology.go:534 removing volume info: Id:139, Size:1, ReplicaPlacement:000, Collection:c, Version:3, Ttl:, FileCount:0, DeleteCount:0, DeletedByteCount:0, ReadOnly:false, ModifiedAtSecond:0 from 10.0.0.1:8080 I0824 22:18:30.240364 volume_layout.go:815 Volume 139 becomes unwritable I0824 22:18:30.240393 volume_layout.go:335 volume 139 does not have enough copies I0824 22:18:30.240453 data_node.go:114 Deleting volume id: 140 I0824 22:18:30.240495 topology.go:534 removing volume info: Id:140, Size:1, ReplicaPlacement:000, Collection:c, Version:3, Ttl:, FileCount:0, DeleteCount:0, DeletedByteCount:0, ReadOnly:false, ModifiedAtSecond:0 from 10.0.0.1:8080 I0824 22:18:30.240642 volume_layout.go:815 Volume 140 becomes unwritable I0824 22:18:30.240663 volume_layout.go:335 volume 140 does not have enough copies I0824 22:18:30.240767 data_node.go:114 Deleting volume id: 141 I0824 22:18:30.240836 topology.go:534 removing volume info: Id:141, Size:1, ReplicaPlacement:000, Collection:c, Version:3, Ttl:, FileCount:0, DeleteCount:0, DeletedByteCount:0, ReadOnly:false, ModifiedAtSecond:0 from 10.0.0.1:8080 I0824 22:18:30.240869 volume_layout.go:815 Volume 141 becomes unwritable I0824 22:18:30.240888 volume_layout.go:335 volume 141 does not have enough copies I0824 22:18:30.240968 data_node.go:114 Deleting volume id: 142 I0824 22:18:30.241141 topology.go:534 removing volume info: Id:142, Size:1, ReplicaPlacement:000, Collection:c, Version:3, Ttl:, FileCount:0, DeleteCount:0, DeletedByteCount:0, ReadOnly:false, ModifiedAtSecond:0 from 10.0.0.1:8080 I0824 22:18:30.241257 volume_layout.go:815 Volume 142 becomes unwritable I0824 22:18:30.241277 volume_layout.go:335 volume 142 does not have enough copies I0824 22:18:30.241341 data_node.go:114 Deleting volume id: 143 I0824 22:18:30.241406 topology.go:534 removing volume info: Id:143, Size:1, ReplicaPlacement:000, Collection:c, Version:3, Ttl:, FileCount:0, DeleteCount:0, DeletedByteCount:0, ReadOnly:false, ModifiedAtSecond:0 from 10.0.0.1:8080 I0824 22:18:30.241436 volume_layout.go:815 Volume 143 becomes unwritable I0824 22:18:30.241454 volume_layout.go:335 volume 143 does not have enough copies I0824 22:18:30.241521 data_node.go:114 Deleting volume id: 144 I0824 22:18:30.241630 topology.go:534 removing volume info: Id:144, Size:1, ReplicaPlacement:000, Collection:c, Version:3, Ttl:, FileCount:0, DeleteCount:0, DeletedByteCount:0, ReadOnly:false, ModifiedAtSecond:0 from 10.0.0.1:8080 I0824 22:18:30.241716 volume_layout.go:815 Volume 144 becomes unwritable I0824 22:18:30.241737 volume_layout.go:335 volume 144 does not have enough copies I0824 22:18:30.241797 data_node.go:114 Deleting volume id: 145 I0824 22:18:30.241838 topology.go:534 removing volume info: Id:145, Size:1, ReplicaPlacement:000, Collection:c, Version:3, Ttl:, FileCount:0, DeleteCount:0, DeletedByteCount:0, ReadOnly:false, ModifiedAtSecond:0 from 10.0.0.1:8080 I0824 22:18:30.241868 volume_layout.go:815 Volume 145 becomes unwritable I0824 22:18:30.241886 volume_layout.go:335 volume 145 does not have enough copies I0824 22:18:30.241965 data_node.go:114 Deleting volume id: 146 I0824 22:18:30.242005 topology.go:534 removing volume info: Id:146, Size:1, ReplicaPlacement:000, Collection:c, Version:3, Ttl:, FileCount:0, DeleteCount:0, DeletedByteCount:0, ReadOnly:false, ModifiedAtSecond:0 from 10.0.0.1:8080 I0824 22:18:30.242110 volume_layout.go:815 Volume 146 becomes unwritable I0824 22:18:30.242170 volume_layout.go:335 volume 146 does not have enough copies I0824 22:18:30.242318 data_node.go:114 Deleting volume id: 147 I0824 22:18:30.242467 topology.go:534 removing volume info: Id:147, Size:1, ReplicaPlacement:000, Collection:c, Version:3, Ttl:, FileCount:0, DeleteCount:0, DeletedByteCount:0, ReadOnly:false, ModifiedAtSecond:0 from 10.0.0.1:8080 I0824 22:18:30.242499 volume_layout.go:815 Volume 147 becomes unwritable I0824 22:18:30.242532 volume_layout.go:335 volume 147 does not have enough copies I0824 22:18:30.242608 data_node.go:114 Deleting volume id: 148 I0824 22:18:30.242652 topology.go:534 removing volume info: Id:148, Size:1, ReplicaPlacement:000, Collection:c, Version:3, Ttl:, FileCount:0, DeleteCount:0, DeletedByteCount:0, ReadOnly:false, ModifiedAtSecond:0 from 10.0.0.1:8080 I0824 22:18:30.242682 volume_layout.go:815 Volume 148 becomes unwritable I0824 22:18:30.242710 volume_layout.go:335 volume 148 does not have enough copies I0824 22:18:30.242790 data_node.go:114 Deleting volume id: 149 I0824 22:18:30.242872 topology.go:534 removing volume info: Id:149, Size:1, ReplicaPlacement:000, Collection:c, Version:3, Ttl:, FileCount:0, DeleteCount:0, DeletedByteCount:0, ReadOnly:false, ModifiedAtSecond:0 from 10.0.0.1:8080 I0824 22:18:30.242992 volume_layout.go:815 Volume 149 becomes unwritable I0824 22:18:30.243014 volume_layout.go:335 volume 149 does not have enough copies I0824 22:18:30.243071 data_node.go:114 Deleting volume id: 100 I0824 22:18:30.243212 topology.go:534 removing volume info: Id:100, Size:1, ReplicaPlacement:000, Collection:c, Version:3, Ttl:, FileCount:0, DeleteCount:0, DeletedByteCount:0, ReadOnly:false, ModifiedAtSecond:0 from 10.0.0.1:8080 I0824 22:18:30.243322 volume_layout.go:815 Volume 100 becomes unwritable I0824 22:18:30.243349 volume_layout.go:335 volume 100 does not have enough copies I0824 22:18:30.243637 data_node.go:114 Deleting volume id: 101 I0824 22:18:30.243739 topology.go:534 removing volume info: Id:101, Size:1, ReplicaPlacement:000, Collection:c, Version:3, Ttl:, FileCount:0, DeleteCount:0, DeletedByteCount:0, ReadOnly:false, ModifiedAtSecond:0 from 10.0.0.1:8080 I0824 22:18:30.243790 volume_layout.go:815 Volume 101 becomes unwritable I0824 22:18:30.243810 volume_layout.go:335 volume 101 does not have enough copies I0824 22:18:30.243888 data_node.go:114 Deleting volume id: 102 I0824 22:18:30.243931 topology.go:534 removing volume info: Id:102, Size:1, ReplicaPlacement:000, Collection:c, Version:3, Ttl:, FileCount:0, DeleteCount:0, DeletedByteCount:0, ReadOnly:false, ModifiedAtSecond:0 from 10.0.0.1:8080 I0824 22:18:30.243963 volume_layout.go:815 Volume 102 becomes unwritable I0824 22:18:30.243982 volume_layout.go:335 volume 102 does not have enough copies I0824 22:18:30.244046 data_node.go:114 Deleting volume id: 103 I0824 22:18:30.244103 topology.go:534 removing volume info: Id:103, Size:1, ReplicaPlacement:000, Collection:c, Version:3, Ttl:, FileCount:0, DeleteCount:0, DeletedByteCount:0, ReadOnly:false, ModifiedAtSecond:0 from 10.0.0.1:8080 I0824 22:18:30.244178 volume_layout.go:815 Volume 103 becomes unwritable I0824 22:18:30.244220 volume_layout.go:335 volume 103 does not have enough copies I0824 22:18:30.244277 data_node.go:114 Deleting volume id: 104 I0824 22:18:30.244317 topology.go:534 removing volume info: Id:104, Size:1, ReplicaPlacement:000, Collection:c, Version:3, Ttl:, FileCount:0, DeleteCount:0, DeletedByteCount:0, ReadOnly:false, ModifiedAtSecond:0 from 10.0.0.1:8080 I0824 22:18:30.244348 volume_layout.go:815 Volume 104 becomes unwritable I0824 22:18:30.244365 volume_layout.go:335 volume 104 does not have enough copies I0824 22:18:30.244444 data_node.go:114 Deleting volume id: 105 I0824 22:18:30.244540 topology.go:534 removing volume info: Id:105, Size:1, ReplicaPlacement:000, Collection:c, Version:3, Ttl:, FileCount:0, DeleteCount:0, DeletedByteCount:0, ReadOnly:false, ModifiedAtSecond:0 from 10.0.0.1:8080 I0824 22:18:30.244598 volume_layout.go:815 Volume 105 becomes unwritable I0824 22:18:30.244662 volume_layout.go:335 volume 105 does not have enough copies I0824 22:18:30.244767 data_node.go:114 Deleting volume id: 106 I0824 22:18:30.244872 topology.go:534 removing volume info: Id:106, Size:1, ReplicaPlacement:000, Collection:c, Version:3, Ttl:, FileCount:0, DeleteCount:0, DeletedByteCount:0, ReadOnly:false, ModifiedAtSecond:0 from 10.0.0.1:8080 I0824 22:18:30.244957 volume_layout.go:815 Volume 106 becomes unwritable I0824 22:18:30.244980 volume_layout.go:335 volume 106 does not have enough copies I0824 22:18:30.245067 data_node.go:114 Deleting volume id: 107 I0824 22:18:30.245108 topology.go:534 removing volume info: Id:107, Size:1, ReplicaPlacement:000, Collection:c, Version:3, Ttl:, FileCount:0, DeleteCount:0, DeletedByteCount:0, ReadOnly:false, ModifiedAtSecond:0 from 10.0.0.1:8080 I0824 22:18:30.245139 volume_layout.go:815 Volume 107 becomes unwritable I0824 22:18:30.245157 volume_layout.go:335 volume 107 does not have enough copies I0824 22:18:30.245318 data_node.go:114 Deleting volume id: 108 I0824 22:18:30.245414 topology.go:534 removing volume info: Id:108, Size:1, ReplicaPlacement:000, Collection:c, Version:3, Ttl:, FileCount:0, DeleteCount:0, DeletedByteCount:0, ReadOnly:false, ModifiedAtSecond:0 from 10.0.0.1:8080 I0824 22:18:30.245447 volume_layout.go:815 Volume 108 becomes unwritable I0824 22:18:30.245489 volume_layout.go:335 volume 108 does not have enough copies I0824 22:18:30.245573 data_node.go:114 Deleting volume id: 109 I0824 22:18:30.245621 topology.go:534 removing volume info: Id:109, Size:1, ReplicaPlacement:000, Collection:c, Version:3, Ttl:, FileCount:0, DeleteCount:0, DeletedByteCount:0, ReadOnly:false, ModifiedAtSecond:0 from 10.0.0.1:8080 I0824 22:18:30.245651 volume_layout.go:815 Volume 109 becomes unwritable I0824 22:18:30.245699 volume_layout.go:335 volume 109 does not have enough copies I0824 22:18:30.245761 data_node.go:114 Deleting volume id: 110 I0824 22:18:30.245847 topology.go:534 removing volume info: Id:110, Size:1, ReplicaPlacement:000, Collection:c, Version:3, Ttl:, FileCount:0, DeleteCount:0, DeletedByteCount:0, ReadOnly:false, ModifiedAtSecond:0 from 10.0.0.1:8080 I0824 22:18:30.245921 volume_layout.go:815 Volume 110 becomes unwritable I0824 22:18:30.245943 volume_layout.go:335 volume 110 does not have enough copies I0824 22:18:30.246047 data_node.go:114 Deleting volume id: 111 I0824 22:18:30.246090 topology.go:534 removing volume info: Id:111, Size:1, ReplicaPlacement:000, Collection:c, Version:3, Ttl:, FileCount:0, DeleteCount:0, DeletedByteCount:0, ReadOnly:false, ModifiedAtSecond:0 from 10.0.0.1:8080 I0824 22:18:30.246122 volume_layout.go:815 Volume 111 becomes unwritable I0824 22:18:30.246139 volume_layout.go:335 volume 111 does not have enough copies I0824 22:18:30.246207 data_node.go:114 Deleting volume id: 112 I0824 22:18:30.246255 topology.go:534 removing volume info: Id:112, Size:1, ReplicaPlacement:000, Collection:c, Version:3, Ttl:, FileCount:0, DeleteCount:0, DeletedByteCount:0, ReadOnly:false, ModifiedAtSecond:0 from 10.0.0.1:8080 I0824 22:18:30.246287 volume_layout.go:815 Volume 112 becomes unwritable I0824 22:18:30.246304 volume_layout.go:335 volume 112 does not have enough copies I0824 22:18:30.246371 data_node.go:114 Deleting volume id: 113 I0824 22:18:30.246536 topology.go:534 removing volume info: Id:113, Size:1, ReplicaPlacement:000, Collection:c, Version:3, Ttl:, FileCount:0, DeleteCount:0, DeletedByteCount:0, ReadOnly:false, ModifiedAtSecond:0 from 10.0.0.1:8080 I0824 22:18:30.246618 volume_layout.go:815 Volume 113 becomes unwritable I0824 22:18:30.246638 volume_layout.go:335 volume 113 does not have enough copies I0824 22:18:30.246717 data_node.go:114 Deleting volume id: 114 I0824 22:18:30.246759 topology.go:534 removing volume info: Id:114, Size:1, ReplicaPlacement:000, Collection:c, Version:3, Ttl:, FileCount:0, DeleteCount:0, DeletedByteCount:0, ReadOnly:false, ModifiedAtSecond:0 from 10.0.0.1:8080 I0824 22:18:30.246789 volume_layout.go:815 Volume 114 becomes unwritable I0824 22:18:30.246807 volume_layout.go:335 volume 114 does not have enough copies I0824 22:18:30.246886 data_node.go:114 Deleting volume id: 115 I0824 22:18:30.246958 topology.go:534 removing volume info: Id:115, Size:1, ReplicaPlacement:000, Collection:c, Version:3, Ttl:, FileCount:0, DeleteCount:0, DeletedByteCount:0, ReadOnly:false, ModifiedAtSecond:0 from 10.0.0.1:8080 I0824 22:18:30.247048 volume_layout.go:815 Volume 115 becomes unwritable I0824 22:18:30.247069 volume_layout.go:335 volume 115 does not have enough copies I0824 22:18:30.247140 data_node.go:114 Deleting volume id: 116 I0824 22:18:30.247182 topology.go:534 removing volume info: Id:116, Size:1, ReplicaPlacement:000, Collection:c, Version:3, Ttl:, FileCount:0, DeleteCount:0, DeletedByteCount:0, ReadOnly:false, ModifiedAtSecond:0 from 10.0.0.1:8080 I0824 22:18:30.247212 volume_layout.go:815 Volume 116 becomes unwritable I0824 22:18:30.247230 volume_layout.go:335 volume 116 does not have enough copies I0824 22:18:30.247310 data_node.go:114 Deleting volume id: 117 I0824 22:18:30.247402 topology.go:534 removing volume info: Id:117, Size:1, ReplicaPlacement:000, Collection:c, Version:3, Ttl:, FileCount:0, DeleteCount:0, DeletedByteCount:0, ReadOnly:false, ModifiedAtSecond:0 from 10.0.0.1:8080 I0824 22:18:30.247474 volume_layout.go:815 Volume 117 becomes unwritable I0824 22:18:30.247496 volume_layout.go:335 volume 117 does not have enough copies I0824 22:18:30.247551 data_node.go:114 Deleting volume id: 118 I0824 22:18:30.247594 topology.go:534 removing volume info: Id:118, Size:1, ReplicaPlacement:000, Collection:c, Version:3, Ttl:, FileCount:0, DeleteCount:0, DeletedByteCount:0, ReadOnly:false, ModifiedAtSecond:0 from 10.0.0.1:8080 I0824 22:18:30.247639 volume_layout.go:815 Volume 118 becomes unwritable I0824 22:18:30.247661 volume_layout.go:335 volume 118 does not have enough copies I0824 22:18:30.247853 data_node.go:114 Deleting volume id: 119 I0824 22:18:30.247970 topology.go:534 removing volume info: Id:119, Size:1, ReplicaPlacement:000, Collection:c, Version:3, Ttl:, FileCount:0, DeleteCount:0, DeletedByteCount:0, ReadOnly:false, ModifiedAtSecond:0 from 10.0.0.1:8080 I0824 22:18:30.248003 volume_layout.go:815 Volume 119 becomes unwritable I0824 22:18:30.248022 volume_layout.go:335 volume 119 does not have enough copies I0824 22:18:30.248080 data_node.go:114 Deleting volume id: 120 I0824 22:18:30.248157 topology.go:534 removing volume info: Id:120, Size:1, ReplicaPlacement:000, Collection:c, Version:3, Ttl:, FileCount:0, DeleteCount:0, DeletedByteCount:0, ReadOnly:false, ModifiedAtSecond:0 from 10.0.0.1:8080 I0824 22:18:30.248191 volume_layout.go:815 Volume 120 becomes unwritable I0824 22:18:30.248224 volume_layout.go:335 volume 120 does not have enough copies I0824 22:18:30.248344 data_node.go:114 Deleting volume id: 121 I0824 22:18:30.248415 topology.go:534 removing volume info: Id:121, Size:1, ReplicaPlacement:000, Collection:c, Version:3, Ttl:, FileCount:0, DeleteCount:0, DeletedByteCount:0, ReadOnly:false, ModifiedAtSecond:0 from 10.0.0.1:8080 I0824 22:18:30.248462 volume_layout.go:815 Volume 121 becomes unwritable I0824 22:18:30.248481 volume_layout.go:335 volume 121 does not have enough copies I0824 22:18:30.248592 data_node.go:114 Deleting volume id: 122 I0824 22:18:30.248722 topology.go:534 removing volume info: Id:122, Size:1, ReplicaPlacement:000, Collection:c, Version:3, Ttl:, FileCount:0, DeleteCount:0, DeletedByteCount:0, ReadOnly:false, ModifiedAtSecond:0 from 10.0.0.1:8080 I0824 22:18:30.248755 volume_layout.go:815 Volume 122 becomes unwritable I0824 22:18:30.248774 volume_layout.go:335 volume 122 does not have enough copies I0824 22:18:30.248837 data_node.go:114 Deleting volume id: 123 I0824 22:18:30.248881 topology.go:534 removing volume info: Id:123, Size:1, ReplicaPlacement:000, Collection:c, Version:3, Ttl:, FileCount:0, DeleteCount:0, DeletedByteCount:0, ReadOnly:false, ModifiedAtSecond:0 from 10.0.0.1:8080 I0824 22:18:30.248912 volume_layout.go:815 Volume 123 becomes unwritable I0824 22:18:30.248933 volume_layout.go:335 volume 123 does not have enough copies I0824 22:18:30.248986 data_node.go:114 Deleting volume id: 124 I0824 22:18:30.249090 topology.go:534 removing volume info: Id:124, Size:1, ReplicaPlacement:000, Collection:c, Version:3, Ttl:, FileCount:0, DeleteCount:0, DeletedByteCount:0, ReadOnly:false, ModifiedAtSecond:0 from 10.0.0.1:8080 I0824 22:18:30.249193 volume_layout.go:815 Volume 124 becomes unwritable I0824 22:18:30.249229 volume_layout.go:335 volume 124 does not have enough copies I0824 22:18:30.249476 data_node.go:114 Deleting volume id: 125 I0824 22:18:30.249556 topology.go:534 removing volume info: Id:125, Size:1, ReplicaPlacement:000, Collection:c, Version:3, Ttl:, FileCount:0, DeleteCount:0, DeletedByteCount:0, ReadOnly:false, ModifiedAtSecond:0 from 10.0.0.1:8080 I0824 22:18:30.249590 volume_layout.go:815 Volume 125 becomes unwritable I0824 22:18:30.249609 volume_layout.go:335 volume 125 does not have enough copies I0824 22:18:30.249667 data_node.go:114 Deleting volume id: 126 I0824 22:18:30.249726 topology.go:534 removing volume info: Id:126, Size:1, ReplicaPlacement:000, Collection:c, Version:3, Ttl:, FileCount:0, DeleteCount:0, DeletedByteCount:0, ReadOnly:false, ModifiedAtSecond:0 from 10.0.0.1:8080 I0824 22:18:30.249766 volume_layout.go:815 Volume 126 becomes unwritable I0824 22:18:30.249791 volume_layout.go:335 volume 126 does not have enough copies I0824 22:18:30.249857 data_node.go:114 Deleting volume id: 127 I0824 22:18:30.249899 topology.go:534 removing volume info: Id:127, Size:1, ReplicaPlacement:000, Collection:c, Version:3, Ttl:, FileCount:0, DeleteCount:0, DeletedByteCount:0, ReadOnly:false, ModifiedAtSecond:0 from 10.0.0.1:8080 I0824 22:18:30.249945 volume_layout.go:815 Volume 127 becomes unwritable I0824 22:18:30.249965 volume_layout.go:335 volume 127 does not have enough copies I0824 22:18:30.250040 data_node.go:114 Deleting volume id: 128 I0824 22:18:30.250083 topology.go:534 removing volume info: Id:128, Size:1, ReplicaPlacement:000, Collection:c, Version:3, Ttl:, FileCount:0, DeleteCount:0, DeletedByteCount:0, ReadOnly:false, ModifiedAtSecond:0 from 10.0.0.1:8080 I0824 22:18:30.250181 volume_layout.go:815 Volume 128 becomes unwritable I0824 22:18:30.250202 volume_layout.go:335 volume 128 does not have enough copies I0824 22:18:30.250269 data_node.go:114 Deleting volume id: 129 I0824 22:18:30.250330 topology.go:534 removing volume info: Id:129, Size:1, ReplicaPlacement:000, Collection:c, Version:3, Ttl:, FileCount:0, DeleteCount:0, DeletedByteCount:0, ReadOnly:false, ModifiedAtSecond:0 from 10.0.0.1:8080 I0824 22:18:30.250361 volume_layout.go:815 Volume 129 becomes unwritable I0824 22:18:30.250380 volume_layout.go:335 volume 129 does not have enough copies I0824 22:18:30.250461 data_node.go:114 Deleting volume id: 130 I0824 22:18:30.250504 topology.go:534 removing volume info: Id:130, Size:1, ReplicaPlacement:000, Collection:c, Version:3, Ttl:, FileCount:0, DeleteCount:0, DeletedByteCount:0, ReadOnly:false, ModifiedAtSecond:0 from 10.0.0.1:8080 I0824 22:18:30.250729 volume_layout.go:815 Volume 130 becomes unwritable I0824 22:18:30.250751 volume_layout.go:335 volume 130 does not have enough copies I0824 22:18:30.250844 data_node.go:114 Deleting volume id: 131 I0824 22:18:30.250886 topology.go:534 removing volume info: Id:131, Size:1, ReplicaPlacement:000, Collection:c, Version:3, Ttl:, FileCount:0, DeleteCount:0, DeletedByteCount:0, ReadOnly:false, ModifiedAtSecond:0 from 10.0.0.1:8080 I0824 22:18:30.250920 volume_layout.go:815 Volume 131 becomes unwritable I0824 22:18:30.250939 volume_layout.go:335 volume 131 does not have enough copies I0824 22:18:30.250990 data_node.go:114 Deleting volume id: 132 I0824 22:18:30.251038 topology.go:534 removing volume info: Id:132, Size:1, ReplicaPlacement:000, Collection:c, Version:3, Ttl:, FileCount:0, DeleteCount:0, DeletedByteCount:0, ReadOnly:false, ModifiedAtSecond:0 from 10.0.0.1:8080 I0824 22:18:30.251070 volume_layout.go:815 Volume 132 becomes unwritable I0824 22:18:30.251089 volume_layout.go:335 volume 132 does not have enough copies I0824 22:18:30.251171 data_node.go:114 Deleting volume id: 133 I0824 22:18:30.251244 topology.go:534 removing volume info: Id:133, Size:1, ReplicaPlacement:000, Collection:c, Version:3, Ttl:, FileCount:0, DeleteCount:0, DeletedByteCount:0, ReadOnly:false, ModifiedAtSecond:0 from 10.0.0.1:8080 I0824 22:18:30.251316 volume_layout.go:815 Volume 133 becomes unwritable I0824 22:18:30.251338 volume_layout.go:335 volume 133 does not have enough copies I0824 22:18:30.251432 data_node.go:114 Deleting volume id: 134 I0824 22:18:30.251494 topology.go:534 removing volume info: Id:134, Size:1, ReplicaPlacement:000, Collection:c, Version:3, Ttl:, FileCount:0, DeleteCount:0, DeletedByteCount:0, ReadOnly:false, ModifiedAtSecond:0 from 10.0.0.1:8080 I0824 22:18:30.251525 volume_layout.go:815 Volume 134 becomes unwritable I0824 22:18:30.251575 volume_layout.go:335 volume 134 does not have enough copies I0824 22:18:30.251667 data_node.go:114 Deleting volume id: 135 I0824 22:18:30.251712 topology.go:534 removing volume info: Id:135, Size:1, ReplicaPlacement:000, Collection:c, Version:3, Ttl:, FileCount:0, DeleteCount:0, DeletedByteCount:0, ReadOnly:false, ModifiedAtSecond:0 from 10.0.0.1:8080 I0824 22:18:30.251750 volume_layout.go:815 Volume 135 becomes unwritable I0824 22:18:30.251772 volume_layout.go:335 volume 135 does not have enough copies I0824 22:18:30.251832 data_node.go:114 Deleting volume id: 136 I0824 22:18:30.251875 topology.go:534 removing volume info: Id:136, Size:1, ReplicaPlacement:000, Collection:c, Version:3, Ttl:, FileCount:0, DeleteCount:0, DeletedByteCount:0, ReadOnly:false, ModifiedAtSecond:0 from 10.0.0.1:8080 I0824 22:18:30.252047 volume_layout.go:815 Volume 136 becomes unwritable I0824 22:18:30.252082 volume_layout.go:335 volume 136 does not have enough copies I0824 22:18:30.252152 data_node.go:114 Deleting volume id: 137 I0824 22:18:30.252196 topology.go:534 removing volume info: Id:137, Size:1, ReplicaPlacement:000, Collection:c, Version:3, Ttl:, FileCount:0, DeleteCount:0, DeletedByteCount:0, ReadOnly:false, ModifiedAtSecond:0 from 10.0.0.1:8080 I0824 22:18:30.252226 volume_layout.go:815 Volume 137 becomes unwritable I0824 22:18:30.252245 volume_layout.go:335 volume 137 does not have enough copies I0824 22:18:30.252309 data_node.go:114 Deleting volume id: 138 I0824 22:18:30.252367 topology.go:534 removing volume info: Id:138, Size:1, ReplicaPlacement:000, Collection:c, Version:3, Ttl:, FileCount:0, DeleteCount:0, DeletedByteCount:0, ReadOnly:false, ModifiedAtSecond:0 from 10.0.0.1:8080 I0824 22:18:30.252511 volume_layout.go:815 Volume 138 becomes unwritable I0824 22:18:30.252534 volume_layout.go:335 volume 138 does not have enough copies I0824 22:18:30.252617 data_node.go:114 Deleting volume id: 139 I0824 22:18:30.252681 topology.go:534 removing volume info: Id:139, Size:1, ReplicaPlacement:000, Collection:c, Version:3, Ttl:, FileCount:0, DeleteCount:0, DeletedByteCount:0, ReadOnly:false, ModifiedAtSecond:0 from 10.0.0.1:8080 I0824 22:18:30.252714 volume_layout.go:815 Volume 139 becomes unwritable I0824 22:18:30.252755 volume_layout.go:335 volume 139 does not have enough copies I0824 22:18:30.252812 data_node.go:114 Deleting volume id: 140 I0824 22:18:30.252857 topology.go:534 removing volume info: Id:140, Size:1, ReplicaPlacement:000, Collection:c, Version:3, Ttl:, FileCount:0, DeleteCount:0, DeletedByteCount:0, ReadOnly:false, ModifiedAtSecond:0 from 10.0.0.1:8080 I0824 22:18:30.252888 volume_layout.go:815 Volume 140 becomes unwritable I0824 22:18:30.252930 volume_layout.go:335 volume 140 does not have enough copies I0824 22:18:30.253005 data_node.go:114 Deleting volume id: 141 I0824 22:18:30.253067 topology.go:534 removing volume info: Id:141, Size:1, ReplicaPlacement:000, Collection:c, Version:3, Ttl:, FileCount:0, DeleteCount:0, DeletedByteCount:0, ReadOnly:false, ModifiedAtSecond:0 from 10.0.0.1:8080 I0824 22:18:30.253303 volume_layout.go:815 Volume 141 becomes unwritable I0824 22:18:30.253421 volume_layout.go:335 volume 141 does not have enough copies I0824 22:18:30.253514 data_node.go:114 Deleting volume id: 142 I0824 22:18:30.253584 topology.go:534 removing volume info: Id:142, Size:1, ReplicaPlacement:000, Collection:c, Version:3, Ttl:, FileCount:0, DeleteCount:0, DeletedByteCount:0, ReadOnly:false, ModifiedAtSecond:0 from 10.0.0.1:8080 I0824 22:18:30.253628 volume_layout.go:815 Volume 142 becomes unwritable I0824 22:18:30.253648 volume_layout.go:335 volume 142 does not have enough copies I0824 22:18:30.253719 data_node.go:114 Deleting volume id: 143 I0824 22:18:30.253800 topology.go:534 removing volume info: Id:143, Size:1, ReplicaPlacement:000, Collection:c, Version:3, Ttl:, FileCount:0, DeleteCount:0, DeletedByteCount:0, ReadOnly:false, ModifiedAtSecond:0 from 10.0.0.1:8080 I0824 22:18:30.253872 volume_layout.go:815 Volume 143 becomes unwritable I0824 22:18:30.253908 volume_layout.go:335 volume 143 does not have enough copies I0824 22:18:30.253953 data_node.go:114 Deleting volume id: 144 I0824 22:18:30.254385 topology.go:534 removing volume info: Id:144, Size:1, ReplicaPlacement:000, Collection:c, Version:3, Ttl:, FileCount:0, DeleteCount:0, DeletedByteCount:0, ReadOnly:false, ModifiedAtSecond:0 from 10.0.0.1:8080 I0824 22:18:30.254744 volume_layout.go:815 Volume 144 becomes unwritable I0824 22:18:30.254793 volume_layout.go:335 volume 144 does not have enough copies I0824 22:18:30.254895 data_node.go:114 Deleting volume id: 145 I0824 22:18:30.255020 topology.go:534 removing volume info: Id:145, Size:1, ReplicaPlacement:000, Collection:c, Version:3, Ttl:, FileCount:0, DeleteCount:0, DeletedByteCount:0, ReadOnly:false, ModifiedAtSecond:0 from 10.0.0.1:8080 I0824 22:18:30.255092 volume_layout.go:815 Volume 145 becomes unwritable I0824 22:18:30.255117 volume_layout.go:335 volume 145 does not have enough copies I0824 22:18:30.255262 data_node.go:114 Deleting volume id: 146 I0824 22:18:30.255361 topology.go:534 removing volume info: Id:146, Size:1, ReplicaPlacement:000, Collection:c, Version:3, Ttl:, FileCount:0, DeleteCount:0, DeletedByteCount:0, ReadOnly:false, ModifiedAtSecond:0 from 10.0.0.1:8080 I0824 22:18:30.255398 volume_layout.go:815 Volume 146 becomes unwritable I0824 22:18:30.255418 volume_layout.go:335 volume 146 does not have enough copies I0824 22:18:30.255572 data_node.go:114 Deleting volume id: 147 I0824 22:18:30.255749 topology.go:534 removing volume info: Id:147, Size:1, ReplicaPlacement:000, Collection:c, Version:3, Ttl:, FileCount:0, DeleteCount:0, DeletedByteCount:0, ReadOnly:false, ModifiedAtSecond:0 from 10.0.0.1:8080 I0824 22:18:30.255810 volume_layout.go:815 Volume 147 becomes unwritable I0824 22:18:30.255848 volume_layout.go:335 volume 147 does not have enough copies I0824 22:18:30.255937 data_node.go:114 Deleting volume id: 148 I0824 22:18:30.256036 topology.go:534 removing volume info: Id:148, Size:1, ReplicaPlacement:000, Collection:c, Version:3, Ttl:, FileCount:0, DeleteCount:0, DeletedByteCount:0, ReadOnly:false, ModifiedAtSecond:0 from 10.0.0.1:8080 I0824 22:18:30.256069 volume_layout.go:815 Volume 148 becomes unwritable I0824 22:18:30.256088 volume_layout.go:335 volume 148 does not have enough copies I0824 22:18:30.256185 data_node.go:114 Deleting volume id: 149 I0824 22:18:30.256231 topology.go:534 removing volume info: Id:149, Size:1, ReplicaPlacement:000, Collection:c, Version:3, Ttl:, FileCount:0, DeleteCount:0, DeletedByteCount:0, ReadOnly:false, ModifiedAtSecond:0 from 10.0.0.1:8080 I0824 22:18:30.256262 volume_layout.go:815 Volume 149 becomes unwritable I0824 22:18:30.256280 volume_layout.go:335 volume 149 does not have enough copies I0824 22:18:30.256356 data_node.go:114 Deleting volume id: 100 I0824 22:18:30.256399 topology.go:534 removing volume info: Id:100, Size:1, ReplicaPlacement:000, Collection:c, Version:3, Ttl:, FileCount:0, DeleteCount:0, DeletedByteCount:0, ReadOnly:false, ModifiedAtSecond:0 from 10.0.0.1:8080 I0824 22:18:30.256522 volume_layout.go:815 Volume 100 becomes unwritable I0824 22:18:30.256556 volume_layout.go:335 volume 100 does not have enough copies I0824 22:18:30.256634 data_node.go:114 Deleting volume id: 101 I0824 22:18:30.256691 topology.go:534 removing volume info: Id:101, Size:1, ReplicaPlacement:000, Collection:c, Version:3, Ttl:, FileCount:0, DeleteCount:0, DeletedByteCount:0, ReadOnly:false, ModifiedAtSecond:0 from 10.0.0.1:8080 I0824 22:18:30.256723 volume_layout.go:815 Volume 101 becomes unwritable I0824 22:18:30.256743 volume_layout.go:335 volume 101 does not have enough copies I0824 22:18:30.256796 data_node.go:114 Deleting volume id: 102 I0824 22:18:30.256836 topology.go:534 removing volume info: Id:102, Size:1, ReplicaPlacement:000, Collection:c, Version:3, Ttl:, FileCount:0, DeleteCount:0, DeletedByteCount:0, ReadOnly:false, ModifiedAtSecond:0 from 10.0.0.1:8080 I0824 22:18:30.256907 volume_layout.go:815 Volume 102 becomes unwritable I0824 22:18:30.256928 volume_layout.go:335 volume 102 does not have enough copies I0824 22:18:30.257061 data_node.go:114 Deleting volume id: 103 I0824 22:18:30.257180 topology.go:534 removing volume info: Id:103, Size:1, ReplicaPlacement:000, Collection:c, Version:3, Ttl:, FileCount:0, DeleteCount:0, DeletedByteCount:0, ReadOnly:false, ModifiedAtSecond:0 from 10.0.0.1:8080 I0824 22:18:30.257227 volume_layout.go:815 Volume 103 becomes unwritable I0824 22:18:30.257248 volume_layout.go:335 volume 103 does not have enough copies I0824 22:18:30.257307 data_node.go:114 Deleting volume id: 104 I0824 22:18:30.257347 topology.go:534 removing volume info: Id:104, Size:1, ReplicaPlacement:000, Collection:c, Version:3, Ttl:, FileCount:0, DeleteCount:0, DeletedByteCount:0, ReadOnly:false, ModifiedAtSecond:0 from 10.0.0.1:8080 I0824 22:18:30.257377 volume_layout.go:815 Volume 104 becomes unwritable I0824 22:18:30.257396 volume_layout.go:335 volume 104 does not have enough copies I0824 22:18:30.257519 data_node.go:114 Deleting volume id: 105 I0824 22:18:30.257588 topology.go:534 removing volume info: Id:105, Size:1, ReplicaPlacement:000, Collection:c, Version:3, Ttl:, FileCount:0, DeleteCount:0, DeletedByteCount:0, ReadOnly:false, ModifiedAtSecond:0 from 10.0.0.1:8080 I0824 22:18:30.257720 volume_layout.go:815 Volume 105 becomes unwritable I0824 22:18:30.257745 volume_layout.go:335 volume 105 does not have enough copies I0824 22:18:30.257851 data_node.go:114 Deleting volume id: 106 I0824 22:18:30.257909 topology.go:534 removing volume info: Id:106, Size:1, ReplicaPlacement:000, Collection:c, Version:3, Ttl:, FileCount:0, DeleteCount:0, DeletedByteCount:0, ReadOnly:false, ModifiedAtSecond:0 from 10.0.0.1:8080 I0824 22:18:30.257942 volume_layout.go:815 Volume 106 becomes unwritable I0824 22:18:30.257961 volume_layout.go:335 volume 106 does not have enough copies I0824 22:18:30.258039 data_node.go:114 Deleting volume id: 107 I0824 22:18:30.258118 topology.go:534 removing volume info: Id:107, Size:1, ReplicaPlacement:000, Collection:c, Version:3, Ttl:, FileCount:0, DeleteCount:0, DeletedByteCount:0, ReadOnly:false, ModifiedAtSecond:0 from 10.0.0.1:8080 I0824 22:18:30.258149 volume_layout.go:815 Volume 107 becomes unwritable I0824 22:18:30.258167 volume_layout.go:335 volume 107 does not have enough copies I0824 22:18:30.258219 data_node.go:114 Deleting volume id: 108 I0824 22:18:30.258369 topology.go:534 removing volume info: Id:108, Size:1, ReplicaPlacement:000, Collection:c, Version:3, Ttl:, FileCount:0, DeleteCount:0, DeletedByteCount:0, ReadOnly:false, ModifiedAtSecond:0 from 10.0.0.1:8080 I0824 22:18:30.258468 volume_layout.go:815 Volume 108 becomes unwritable I0824 22:18:30.258489 volume_layout.go:335 volume 108 does not have enough copies I0824 22:18:30.258562 data_node.go:114 Deleting volume id: 109 I0824 22:18:30.258603 topology.go:534 removing volume info: Id:109, Size:1, ReplicaPlacement:000, Collection:c, Version:3, Ttl:, FileCount:0, DeleteCount:0, DeletedByteCount:0, ReadOnly:false, ModifiedAtSecond:0 from 10.0.0.1:8080 I0824 22:18:30.258634 volume_layout.go:815 Volume 109 becomes unwritable I0824 22:18:30.258653 volume_layout.go:335 volume 109 does not have enough copies I0824 22:18:30.258705 data_node.go:114 Deleting volume id: 110 I0824 22:18:30.258743 topology.go:534 removing volume info: Id:110, Size:1, ReplicaPlacement:000, Collection:c, Version:3, Ttl:, FileCount:0, DeleteCount:0, DeletedByteCount:0, ReadOnly:false, ModifiedAtSecond:0 from 10.0.0.1:8080 I0824 22:18:30.258773 volume_layout.go:815 Volume 110 becomes unwritable I0824 22:18:30.258797 volume_layout.go:335 volume 110 does not have enough copies I0824 22:18:30.258902 data_node.go:114 Deleting volume id: 111 I0824 22:18:30.258962 topology.go:534 removing volume info: Id:111, Size:1, ReplicaPlacement:000, Collection:c, Version:3, Ttl:, FileCount:0, DeleteCount:0, DeletedByteCount:0, ReadOnly:false, ModifiedAtSecond:0 from 10.0.0.1:8080 I0824 22:18:30.258997 volume_layout.go:815 Volume 111 becomes unwritable I0824 22:18:30.259015 volume_layout.go:335 volume 111 does not have enough copies I0824 22:18:30.259090 data_node.go:114 Deleting volume id: 112 I0824 22:18:30.259131 topology.go:534 removing volume info: Id:112, Size:1, ReplicaPlacement:000, Collection:c, Version:3, Ttl:, FileCount:0, DeleteCount:0, DeletedByteCount:0, ReadOnly:false, ModifiedAtSecond:0 from 10.0.0.1:8080 I0824 22:18:30.259196 volume_layout.go:815 Volume 112 becomes unwritable I0824 22:18:30.259219 volume_layout.go:335 volume 112 does not have enough copies I0824 22:18:30.259422 data_node.go:114 Deleting volume id: 113 I0824 22:18:30.259490 topology.go:534 removing volume info: Id:113, Size:1, ReplicaPlacement:000, Collection:c, Version:3, Ttl:, FileCount:0, DeleteCount:0, DeletedByteCount:0, ReadOnly:false, ModifiedAtSecond:0 from 10.0.0.1:8080 I0824 22:18:30.259527 volume_layout.go:815 Volume 113 becomes unwritable I0824 22:18:30.259546 volume_layout.go:335 volume 113 does not have enough copies I0824 22:18:30.259622 data_node.go:114 Deleting volume id: 114 I0824 22:18:30.259757 topology.go:534 removing volume info: Id:114, Size:1, ReplicaPlacement:000, Collection:c, Version:3, Ttl:, FileCount:0, DeleteCount:0, DeletedByteCount:0, ReadOnly:false, ModifiedAtSecond:0 from 10.0.0.1:8080 I0824 22:18:30.259836 volume_layout.go:815 Volume 114 becomes unwritable I0824 22:18:30.259855 volume_layout.go:335 volume 114 does not have enough copies I0824 22:18:30.259917 data_node.go:114 Deleting volume id: 115 I0824 22:18:30.259960 topology.go:534 removing volume info: Id:115, Size:1, ReplicaPlacement:000, Collection:c, Version:3, Ttl:, FileCount:0, DeleteCount:0, DeletedByteCount:0, ReadOnly:false, ModifiedAtSecond:0 from 10.0.0.1:8080 I0824 22:18:30.259991 volume_layout.go:815 Volume 115 becomes unwritable I0824 22:18:30.260008 volume_layout.go:335 volume 115 does not have enough copies I0824 22:18:30.260057 data_node.go:114 Deleting volume id: 116 I0824 22:18:30.260099 topology.go:534 removing volume info: Id:116, Size:1, ReplicaPlacement:000, Collection:c, Version:3, Ttl:, FileCount:0, DeleteCount:0, DeletedByteCount:0, ReadOnly:false, ModifiedAtSecond:0 from 10.0.0.1:8080 I0824 22:18:30.260129 volume_layout.go:815 Volume 116 becomes unwritable I0824 22:18:30.260149 volume_layout.go:335 volume 116 does not have enough copies I0824 22:18:30.260217 data_node.go:114 Deleting volume id: 117 I0824 22:18:30.260263 topology.go:534 removing volume info: Id:117, Size:1, ReplicaPlacement:000, Collection:c, Version:3, Ttl:, FileCount:0, DeleteCount:0, DeletedByteCount:0, ReadOnly:false, ModifiedAtSecond:0 from 10.0.0.1:8080 I0824 22:18:30.260294 volume_layout.go:815 Volume 117 becomes unwritable I0824 22:18:30.260314 volume_layout.go:335 volume 117 does not have enough copies I0824 22:18:30.260364 data_node.go:114 Deleting volume id: 118 I0824 22:18:30.260404 topology.go:534 removing volume info: Id:118, Size:1, ReplicaPlacement:000, Collection:c, Version:3, Ttl:, FileCount:0, DeleteCount:0, DeletedByteCount:0, ReadOnly:false, ModifiedAtSecond:0 from 10.0.0.1:8080 I0824 22:18:30.260433 volume_layout.go:815 Volume 118 becomes unwritable I0824 22:18:30.260501 volume_layout.go:335 volume 118 does not have enough copies I0824 22:18:30.260593 data_node.go:114 Deleting volume id: 119 I0824 22:18:30.260669 topology.go:534 removing volume info: Id:119, Size:1, ReplicaPlacement:000, Collection:c, Version:3, Ttl:, FileCount:0, DeleteCount:0, DeletedByteCount:0, ReadOnly:false, ModifiedAtSecond:0 from 10.0.0.1:8080 I0824 22:18:30.260806 volume_layout.go:815 Volume 119 becomes unwritable I0824 22:18:30.260840 volume_layout.go:335 volume 119 does not have enough copies I0824 22:18:30.261016 data_node.go:114 Deleting volume id: 120 I0824 22:18:30.261100 topology.go:534 removing volume info: Id:120, Size:1, ReplicaPlacement:000, Collection:c, Version:3, Ttl:, FileCount:0, DeleteCount:0, DeletedByteCount:0, ReadOnly:false, ModifiedAtSecond:0 from 10.0.0.1:8080 I0824 22:18:30.261133 volume_layout.go:815 Volume 120 becomes unwritable I0824 22:18:30.261153 volume_layout.go:335 volume 120 does not have enough copies I0824 22:18:30.261212 data_node.go:114 Deleting volume id: 121 I0824 22:18:30.261251 topology.go:534 removing volume info: Id:121, Size:1, ReplicaPlacement:000, Collection:c, Version:3, Ttl:, FileCount:0, DeleteCount:0, DeletedByteCount:0, ReadOnly:false, ModifiedAtSecond:0 from 10.0.0.1:8080 I0824 22:18:30.261280 volume_layout.go:815 Volume 121 becomes unwritable I0824 22:18:30.261310 volume_layout.go:335 volume 121 does not have enough copies I0824 22:18:30.261405 data_node.go:114 Deleting volume id: 122 I0824 22:18:30.261449 topology.go:534 removing volume info: Id:122, Size:1, ReplicaPlacement:000, Collection:c, Version:3, Ttl:, FileCount:0, DeleteCount:0, DeletedByteCount:0, ReadOnly:false, ModifiedAtSecond:0 from 10.0.0.1:8080 I0824 22:18:30.261480 volume_layout.go:815 Volume 122 becomes unwritable I0824 22:18:30.261498 volume_layout.go:335 volume 122 does not have enough copies I0824 22:18:30.261590 data_node.go:114 Deleting volume id: 123 I0824 22:18:30.261669 topology.go:534 removing volume info: Id:123, Size:1, ReplicaPlacement:000, Collection:c, Version:3, Ttl:, FileCount:0, DeleteCount:0, DeletedByteCount:0, ReadOnly:false, ModifiedAtSecond:0 from 10.0.0.1:8080 I0824 22:18:30.261718 volume_layout.go:815 Volume 123 becomes unwritable I0824 22:18:30.261738 volume_layout.go:335 volume 123 does not have enough copies I0824 22:18:30.261788 data_node.go:114 Deleting volume id: 124 I0824 22:18:30.261829 topology.go:534 removing volume info: Id:124, Size:1, ReplicaPlacement:000, Collection:c, Version:3, Ttl:, FileCount:0, DeleteCount:0, DeletedByteCount:0, ReadOnly:false, ModifiedAtSecond:0 from 10.0.0.1:8080 I0824 22:18:30.261859 volume_layout.go:815 Volume 124 becomes unwritable I0824 22:18:30.261877 volume_layout.go:335 volume 124 does not have enough copies I0824 22:18:30.261935 data_node.go:114 Deleting volume id: 125 I0824 22:18:30.261973 topology.go:534 removing volume info: Id:125, Size:1, ReplicaPlacement:000, Collection:c, Version:3, Ttl:, FileCount:0, DeleteCount:0, DeletedByteCount:0, ReadOnly:false, ModifiedAtSecond:0 from 10.0.0.1:8080 I0824 22:18:30.262003 volume_layout.go:815 Volume 125 becomes unwritable I0824 22:18:30.262021 volume_layout.go:335 volume 125 does not have enough copies I0824 22:18:30.262137 data_node.go:114 Deleting volume id: 126 I0824 22:18:30.262282 topology.go:534 removing volume info: Id:126, Size:1, ReplicaPlacement:000, Collection:c, Version:3, Ttl:, FileCount:0, DeleteCount:0, DeletedByteCount:0, ReadOnly:false, ModifiedAtSecond:0 from 10.0.0.1:8080 I0824 22:18:30.262331 volume_layout.go:815 Volume 126 becomes unwritable I0824 22:18:30.262349 volume_layout.go:335 volume 126 does not have enough copies I0824 22:18:30.262416 data_node.go:114 Deleting volume id: 127 I0824 22:18:30.262463 topology.go:534 removing volume info: Id:127, Size:1, ReplicaPlacement:000, Collection:c, Version:3, Ttl:, FileCount:0, DeleteCount:0, DeletedByteCount:0, ReadOnly:false, ModifiedAtSecond:0 from 10.0.0.1:8080 I0824 22:18:30.262547 volume_layout.go:815 Volume 127 becomes unwritable I0824 22:18:30.262565 volume_layout.go:335 volume 127 does not have enough copies I0824 22:18:30.262646 data_node.go:114 Deleting volume id: 128 I0824 22:18:30.262686 topology.go:534 removing volume info: Id:128, Size:1, ReplicaPlacement:000, Collection:c, Version:3, Ttl:, FileCount:0, DeleteCount:0, DeletedByteCount:0, ReadOnly:false, ModifiedAtSecond:0 from 10.0.0.1:8080 I0824 22:18:30.262716 volume_layout.go:815 Volume 128 becomes unwritable I0824 22:18:30.262778 volume_layout.go:335 volume 128 does not have enough copies I0824 22:18:30.262862 data_node.go:114 Deleting volume id: 129 I0824 22:18:30.262926 topology.go:534 removing volume info: Id:129, Size:1, ReplicaPlacement:000, Collection:c, Version:3, Ttl:, FileCount:0, DeleteCount:0, DeletedByteCount:0, ReadOnly:false, ModifiedAtSecond:0 from 10.0.0.1:8080 I0824 22:18:30.262958 volume_layout.go:815 Volume 129 becomes unwritable I0824 22:18:30.262976 volume_layout.go:335 volume 129 does not have enough copies I0824 22:18:30.263051 data_node.go:114 Deleting volume id: 130 I0824 22:18:30.263108 topology.go:534 removing volume info: Id:130, Size:1, ReplicaPlacement:000, Collection:c, Version:3, Ttl:, FileCount:0, DeleteCount:0, DeletedByteCount:0, ReadOnly:false, ModifiedAtSecond:0 from 10.0.0.1:8080 I0824 22:18:30.263169 volume_layout.go:815 Volume 130 becomes unwritable I0824 22:18:30.263209 volume_layout.go:335 volume 130 does not have enough copies I0824 22:18:30.263333 data_node.go:114 Deleting volume id: 131 I0824 22:18:30.263394 topology.go:534 removing volume info: Id:131, Size:1, ReplicaPlacement:000, Collection:c, Version:3, Ttl:, FileCount:0, DeleteCount:0, DeletedByteCount:0, ReadOnly:false, ModifiedAtSecond:0 from 10.0.0.1:8080 I0824 22:18:30.263426 volume_layout.go:815 Volume 131 becomes unwritable I0824 22:18:30.263444 volume_layout.go:335 volume 131 does not have enough copies I0824 22:18:30.263574 data_node.go:114 Deleting volume id: 132 I0824 22:18:30.263648 topology.go:534 removing volume info: Id:132, Size:1, ReplicaPlacement:000, Collection:c, Version:3, Ttl:, FileCount:0, DeleteCount:0, DeletedByteCount:0, ReadOnly:false, ModifiedAtSecond:0 from 10.0.0.1:8080 I0824 22:18:30.263958 volume_layout.go:815 Volume 132 becomes unwritable I0824 22:18:30.263988 volume_layout.go:335 volume 132 does not have enough copies I0824 22:18:30.264152 data_node.go:114 Deleting volume id: 133 I0824 22:18:30.264246 topology.go:534 removing volume info: Id:133, Size:1, ReplicaPlacement:000, Collection:c, Version:3, Ttl:, FileCount:0, DeleteCount:0, DeletedByteCount:0, ReadOnly:false, ModifiedAtSecond:0 from 10.0.0.1:8080 I0824 22:18:30.264306 volume_layout.go:815 Volume 133 becomes unwritable I0824 22:18:30.264329 volume_layout.go:335 volume 133 does not have enough copies I0824 22:18:30.264395 data_node.go:114 Deleting volume id: 134 I0824 22:18:30.264464 topology.go:534 removing volume info: Id:134, Size:1, ReplicaPlacement:000, Collection:c, Version:3, Ttl:, FileCount:0, DeleteCount:0, DeletedByteCount:0, ReadOnly:false, ModifiedAtSecond:0 from 10.0.0.1:8080 I0824 22:18:30.264499 volume_layout.go:815 Volume 134 becomes unwritable I0824 22:18:30.264518 volume_layout.go:335 volume 134 does not have enough copies I0824 22:18:30.264629 data_node.go:114 Deleting volume id: 135 I0824 22:18:30.264816 topology.go:534 removing volume info: Id:135, Size:1, ReplicaPlacement:000, Collection:c, Version:3, Ttl:, FileCount:0, DeleteCount:0, DeletedByteCount:0, ReadOnly:false, ModifiedAtSecond:0 from 10.0.0.1:8080 I0824 22:18:30.264887 volume_layout.go:815 Volume 135 becomes unwritable I0824 22:18:30.264909 volume_layout.go:335 volume 135 does not have enough copies I0824 22:18:30.264949 data_node.go:114 Deleting volume id: 136 I0824 22:18:30.265018 topology.go:534 removing volume info: Id:136, Size:1, ReplicaPlacement:000, Collection:c, Version:3, Ttl:, FileCount:0, DeleteCount:0, DeletedByteCount:0, ReadOnly:false, ModifiedAtSecond:0 from 10.0.0.1:8080 I0824 22:18:30.265081 volume_layout.go:815 Volume 136 becomes unwritable I0824 22:18:30.265100 volume_layout.go:335 volume 136 does not have enough copies I0824 22:18:30.265237 data_node.go:114 Deleting volume id: 137 I0824 22:18:30.265306 topology.go:534 removing volume info: Id:137, Size:1, ReplicaPlacement:000, Collection:c, Version:3, Ttl:, FileCount:0, DeleteCount:0, DeletedByteCount:0, ReadOnly:false, ModifiedAtSecond:0 from 10.0.0.1:8080 I0824 22:18:30.265348 volume_layout.go:815 Volume 137 becomes unwritable I0824 22:18:30.265367 volume_layout.go:335 volume 137 does not have enough copies I0824 22:18:30.265427 data_node.go:114 Deleting volume id: 138 I0824 22:18:30.265467 topology.go:534 removing volume info: Id:138, Size:1, ReplicaPlacement:000, Collection:c, Version:3, Ttl:, FileCount:0, DeleteCount:0, DeletedByteCount:0, ReadOnly:false, ModifiedAtSecond:0 from 10.0.0.1:8080 I0824 22:18:30.265585 volume_layout.go:815 Volume 138 becomes unwritable I0824 22:18:30.265607 volume_layout.go:335 volume 138 does not have enough copies I0824 22:18:30.265748 data_node.go:114 Deleting volume id: 139 I0824 22:18:30.265793 topology.go:534 removing volume info: Id:139, Size:1, ReplicaPlacement:000, Collection:c, Version:3, Ttl:, FileCount:0, DeleteCount:0, DeletedByteCount:0, ReadOnly:false, ModifiedAtSecond:0 from 10.0.0.1:8080 I0824 22:18:30.265826 volume_layout.go:815 Volume 139 becomes unwritable I0824 22:18:30.265845 volume_layout.go:335 volume 139 does not have enough copies I0824 22:18:30.265897 data_node.go:114 Deleting volume id: 140 I0824 22:18:30.265999 topology.go:534 removing volume info: Id:140, Size:1, ReplicaPlacement:000, Collection:c, Version:3, Ttl:, FileCount:0, DeleteCount:0, DeletedByteCount:0, ReadOnly:false, ModifiedAtSecond:0 from 10.0.0.1:8080 I0824 22:18:30.266079 volume_layout.go:815 Volume 140 becomes unwritable I0824 22:18:30.266130 volume_layout.go:335 volume 140 does not have enough copies I0824 22:18:30.266223 data_node.go:114 Deleting volume id: 141 I0824 22:18:30.266345 topology.go:534 removing volume info: Id:141, Size:1, ReplicaPlacement:000, Collection:c, Version:3, Ttl:, FileCount:0, DeleteCount:0, DeletedByteCount:0, ReadOnly:false, ModifiedAtSecond:0 from 10.0.0.1:8080 I0824 22:18:30.266378 volume_layout.go:815 Volume 141 becomes unwritable I0824 22:18:30.266397 volume_layout.go:335 volume 141 does not have enough copies I0824 22:18:30.266451 data_node.go:114 Deleting volume id: 142 I0824 22:18:30.266523 topology.go:534 removing volume info: Id:142, Size:1, ReplicaPlacement:000, Collection:c, Version:3, Ttl:, FileCount:0, DeleteCount:0, DeletedByteCount:0, ReadOnly:false, ModifiedAtSecond:0 from 10.0.0.1:8080 I0824 22:18:30.266555 volume_layout.go:815 Volume 142 becomes unwritable I0824 22:18:30.266573 volume_layout.go:335 volume 142 does not have enough copies I0824 22:18:30.266639 data_node.go:114 Deleting volume id: 143 I0824 22:18:30.266797 topology.go:534 removing volume info: Id:143, Size:1, ReplicaPlacement:000, Collection:c, Version:3, Ttl:, FileCount:0, DeleteCount:0, DeletedByteCount:0, ReadOnly:false, ModifiedAtSecond:0 from 10.0.0.1:8080 I0824 22:18:30.266850 volume_layout.go:815 Volume 143 becomes unwritable I0824 22:18:30.266877 volume_layout.go:335 volume 143 does not have enough copies I0824 22:18:30.266933 data_node.go:114 Deleting volume id: 144 I0824 22:18:30.266988 topology.go:534 removing volume info: Id:144, Size:1, ReplicaPlacement:000, Collection:c, Version:3, Ttl:, FileCount:0, DeleteCount:0, DeletedByteCount:0, ReadOnly:false, ModifiedAtSecond:0 from 10.0.0.1:8080 I0824 22:18:30.267027 volume_layout.go:815 Volume 144 becomes unwritable I0824 22:18:30.267045 volume_layout.go:335 volume 144 does not have enough copies I0824 22:18:30.267102 data_node.go:114 Deleting volume id: 145 I0824 22:18:30.267141 topology.go:534 removing volume info: Id:145, Size:1, ReplicaPlacement:000, Collection:c, Version:3, Ttl:, FileCount:0, DeleteCount:0, DeletedByteCount:0, ReadOnly:false, ModifiedAtSecond:0 from 10.0.0.1:8080 I0824 22:18:30.267190 volume_layout.go:815 Volume 145 becomes unwritable I0824 22:18:30.267225 volume_layout.go:335 volume 145 does not have enough copies I0824 22:18:30.267387 data_node.go:114 Deleting volume id: 146 I0824 22:18:30.267484 topology.go:534 removing volume info: Id:146, Size:1, ReplicaPlacement:000, Collection:c, Version:3, Ttl:, FileCount:0, DeleteCount:0, DeletedByteCount:0, ReadOnly:false, ModifiedAtSecond:0 from 10.0.0.1:8080 I0824 22:18:30.267517 volume_layout.go:815 Volume 146 becomes unwritable I0824 22:18:30.267571 volume_layout.go:335 volume 146 does not have enough copies I0824 22:18:30.267636 data_node.go:114 Deleting volume id: 147 I0824 22:18:30.267678 topology.go:534 removing volume info: Id:147, Size:1, ReplicaPlacement:000, Collection:c, Version:3, Ttl:, FileCount:0, DeleteCount:0, DeletedByteCount:0, ReadOnly:false, ModifiedAtSecond:0 from 10.0.0.1:8080 I0824 22:18:30.267708 volume_layout.go:815 Volume 147 becomes unwritable I0824 22:18:30.267765 volume_layout.go:335 volume 147 does not have enough copies I0824 22:18:30.267871 data_node.go:114 Deleting volume id: 148 I0824 22:18:30.267913 topology.go:534 removing volume info: Id:148, Size:1, ReplicaPlacement:000, Collection:c, Version:3, Ttl:, FileCount:0, DeleteCount:0, DeletedByteCount:0, ReadOnly:false, ModifiedAtSecond:0 from 10.0.0.1:8080 I0824 22:18:30.267962 volume_layout.go:815 Volume 148 becomes unwritable I0824 22:18:30.267979 volume_layout.go:335 volume 148 does not have enough copies I0824 22:18:30.268037 data_node.go:114 Deleting volume id: 149 I0824 22:18:30.268103 topology.go:534 removing volume info: Id:149, Size:1, ReplicaPlacement:000, Collection:c, Version:3, Ttl:, FileCount:0, DeleteCount:0, DeletedByteCount:0, ReadOnly:false, ModifiedAtSecond:0 from 10.0.0.1:8080 I0824 22:18:30.268135 volume_layout.go:815 Volume 149 becomes unwritable I0824 22:18:30.268153 volume_layout.go:335 volume 149 does not have enough copies I0824 22:18:30.268204 data_node.go:114 Deleting volume id: 100 I0824 22:18:30.268244 topology.go:534 removing volume info: Id:100, Size:1, ReplicaPlacement:000, Collection:c, Version:3, Ttl:, FileCount:0, DeleteCount:0, DeletedByteCount:0, ReadOnly:false, ModifiedAtSecond:0 from 10.0.0.1:8080 I0824 22:18:30.268274 volume_layout.go:815 Volume 100 becomes unwritable I0824 22:18:30.268293 volume_layout.go:335 volume 100 does not have enough copies I0824 22:18:30.268357 data_node.go:114 Deleting volume id: 101 I0824 22:18:30.268399 topology.go:534 removing volume info: Id:101, Size:1, ReplicaPlacement:000, Collection:c, Version:3, Ttl:, FileCount:0, DeleteCount:0, DeletedByteCount:0, ReadOnly:false, ModifiedAtSecond:0 from 10.0.0.1:8080 I0824 22:18:30.268461 volume_layout.go:815 Volume 101 becomes unwritable I0824 22:18:30.268485 volume_layout.go:335 volume 101 does not have enough copies I0824 22:18:30.268582 data_node.go:114 Deleting volume id: 102 I0824 22:18:30.268728 topology.go:534 removing volume info: Id:102, Size:1, ReplicaPlacement:000, Collection:c, Version:3, Ttl:, FileCount:0, DeleteCount:0, DeletedByteCount:0, ReadOnly:false, ModifiedAtSecond:0 from 10.0.0.1:8080 I0824 22:18:30.268776 volume_layout.go:815 Volume 102 becomes unwritable I0824 22:18:30.268796 volume_layout.go:335 volume 102 does not have enough copies I0824 22:18:30.268924 data_node.go:114 Deleting volume id: 103 I0824 22:18:30.269012 topology.go:534 removing volume info: Id:103, Size:1, ReplicaPlacement:000, Collection:c, Version:3, Ttl:, FileCount:0, DeleteCount:0, DeletedByteCount:0, ReadOnly:false, ModifiedAtSecond:0 from 10.0.0.1:8080 I0824 22:18:30.269046 volume_layout.go:815 Volume 103 becomes unwritable I0824 22:18:30.269064 volume_layout.go:335 volume 103 does not have enough copies I0824 22:18:30.269149 data_node.go:114 Deleting volume id: 104 I0824 22:18:30.269233 topology.go:534 removing volume info: Id:104, Size:1, ReplicaPlacement:000, Collection:c, Version:3, Ttl:, FileCount:0, DeleteCount:0, DeletedByteCount:0, ReadOnly:false, ModifiedAtSecond:0 from 10.0.0.1:8080 I0824 22:18:30.269264 volume_layout.go:815 Volume 104 becomes unwritable I0824 22:18:30.269283 volume_layout.go:335 volume 104 does not have enough copies I0824 22:18:30.269382 data_node.go:114 Deleting volume id: 105 I0824 22:18:30.269423 topology.go:534 removing volume info: Id:105, Size:1, ReplicaPlacement:000, Collection:c, Version:3, Ttl:, FileCount:0, DeleteCount:0, DeletedByteCount:0, ReadOnly:false, ModifiedAtSecond:0 from 10.0.0.1:8080 I0824 22:18:30.269460 volume_layout.go:815 Volume 105 becomes unwritable I0824 22:18:30.269478 volume_layout.go:335 volume 105 does not have enough copies I0824 22:18:30.269564 data_node.go:114 Deleting volume id: 106 I0824 22:18:30.269657 topology.go:534 removing volume info: Id:106, Size:1, ReplicaPlacement:000, Collection:c, Version:3, Ttl:, FileCount:0, DeleteCount:0, DeletedByteCount:0, ReadOnly:false, ModifiedAtSecond:0 from 10.0.0.1:8080 I0824 22:18:30.269690 volume_layout.go:815 Volume 106 becomes unwritable I0824 22:18:30.269833 volume_layout.go:335 volume 106 does not have enough copies I0824 22:18:30.269897 data_node.go:114 Deleting volume id: 107 I0824 22:18:30.269943 topology.go:534 removing volume info: Id:107, Size:1, ReplicaPlacement:000, Collection:c, Version:3, Ttl:, FileCount:0, DeleteCount:0, DeletedByteCount:0, ReadOnly:false, ModifiedAtSecond:0 from 10.0.0.1:8080 I0824 22:18:30.269974 volume_layout.go:815 Volume 107 becomes unwritable I0824 22:18:30.270011 volume_layout.go:335 volume 107 does not have enough copies I0824 22:18:30.270095 data_node.go:114 Deleting volume id: 108 I0824 22:18:30.270196 topology.go:534 removing volume info: Id:108, Size:1, ReplicaPlacement:000, Collection:c, Version:3, Ttl:, FileCount:0, DeleteCount:0, DeletedByteCount:0, ReadOnly:false, ModifiedAtSecond:0 from 10.0.0.1:8080 I0824 22:18:30.270821 volume_layout.go:815 Volume 108 becomes unwritable I0824 22:18:30.270928 volume_layout.go:335 volume 108 does not have enough copies I0824 22:18:30.271206 data_node.go:114 Deleting volume id: 109 I0824 22:18:30.271290 topology.go:534 removing volume info: Id:109, Size:1, ReplicaPlacement:000, Collection:c, Version:3, Ttl:, FileCount:0, DeleteCount:0, DeletedByteCount:0, ReadOnly:false, ModifiedAtSecond:0 from 10.0.0.1:8080 I0824 22:18:30.271334 volume_layout.go:815 Volume 109 becomes unwritable I0824 22:18:30.271354 volume_layout.go:335 volume 109 does not have enough copies I0824 22:18:30.271429 data_node.go:114 Deleting volume id: 110 I0824 22:18:30.271559 topology.go:534 removing volume info: Id:110, Size:1, ReplicaPlacement:000, Collection:c, Version:3, Ttl:, FileCount:0, DeleteCount:0, DeletedByteCount:0, ReadOnly:false, ModifiedAtSecond:0 from 10.0.0.1:8080 I0824 22:18:30.271593 volume_layout.go:815 Volume 110 becomes unwritable I0824 22:18:30.271613 volume_layout.go:335 volume 110 does not have enough copies I0824 22:18:30.271725 data_node.go:114 Deleting volume id: 111 I0824 22:18:30.271796 topology.go:534 removing volume info: Id:111, Size:1, ReplicaPlacement:000, Collection:c, Version:3, Ttl:, FileCount:0, DeleteCount:0, DeletedByteCount:0, ReadOnly:false, ModifiedAtSecond:0 from 10.0.0.1:8080 I0824 22:18:30.271829 volume_layout.go:815 Volume 111 becomes unwritable I0824 22:18:30.271877 volume_layout.go:335 volume 111 does not have enough copies I0824 22:18:30.271939 data_node.go:114 Deleting volume id: 112 I0824 22:18:30.271978 topology.go:534 removing volume info: Id:112, Size:1, ReplicaPlacement:000, Collection:c, Version:3, Ttl:, FileCount:0, DeleteCount:0, DeletedByteCount:0, ReadOnly:false, ModifiedAtSecond:0 from 10.0.0.1:8080 I0824 22:18:30.272023 volume_layout.go:815 Volume 112 becomes unwritable I0824 22:18:30.272043 volume_layout.go:335 volume 112 does not have enough copies I0824 22:18:30.272174 data_node.go:114 Deleting volume id: 113 I0824 22:18:30.272257 topology.go:534 removing volume info: Id:113, Size:1, ReplicaPlacement:000, Collection:c, Version:3, Ttl:, FileCount:0, DeleteCount:0, DeletedByteCount:0, ReadOnly:false, ModifiedAtSecond:0 from 10.0.0.1:8080 I0824 22:18:30.272316 volume_layout.go:815 Volume 113 becomes unwritable I0824 22:18:30.272340 volume_layout.go:335 volume 113 does not have enough copies I0824 22:18:30.272442 data_node.go:114 Deleting volume id: 114 I0824 22:18:30.272575 topology.go:534 removing volume info: Id:114, Size:1, ReplicaPlacement:000, Collection:c, Version:3, Ttl:, FileCount:0, DeleteCount:0, DeletedByteCount:0, ReadOnly:false, ModifiedAtSecond:0 from 10.0.0.1:8080 I0824 22:18:30.272647 volume_layout.go:815 Volume 114 becomes unwritable I0824 22:18:30.272669 volume_layout.go:335 volume 114 does not have enough copies I0824 22:18:30.272773 data_node.go:114 Deleting volume id: 115 I0824 22:18:30.272816 topology.go:534 removing volume info: Id:115, Size:1, ReplicaPlacement:000, Collection:c, Version:3, Ttl:, FileCount:0, DeleteCount:0, DeletedByteCount:0, ReadOnly:false, ModifiedAtSecond:0 from 10.0.0.1:8080 I0824 22:18:30.272848 volume_layout.go:815 Volume 115 becomes unwritable I0824 22:18:30.272868 volume_layout.go:335 volume 115 does not have enough copies I0824 22:18:30.272924 data_node.go:114 Deleting volume id: 116 I0824 22:18:30.272996 topology.go:534 removing volume info: Id:116, Size:1, ReplicaPlacement:000, Collection:c, Version:3, Ttl:, FileCount:0, DeleteCount:0, DeletedByteCount:0, ReadOnly:false, ModifiedAtSecond:0 from 10.0.0.1:8080 I0824 22:18:30.273029 volume_layout.go:815 Volume 116 becomes unwritable I0824 22:18:30.273047 volume_layout.go:335 volume 116 does not have enough copies I0824 22:18:30.273106 data_node.go:114 Deleting volume id: 117 I0824 22:18:30.273149 topology.go:534 removing volume info: Id:117, Size:1, ReplicaPlacement:000, Collection:c, Version:3, Ttl:, FileCount:0, DeleteCount:0, DeletedByteCount:0, ReadOnly:false, ModifiedAtSecond:0 from 10.0.0.1:8080 I0824 22:18:30.273179 volume_layout.go:815 Volume 117 becomes unwritable I0824 22:18:30.273197 volume_layout.go:335 volume 117 does not have enough copies I0824 22:18:30.273366 data_node.go:114 Deleting volume id: 118 I0824 22:18:30.273463 topology.go:534 removing volume info: Id:118, Size:1, ReplicaPlacement:000, Collection:c, Version:3, Ttl:, FileCount:0, DeleteCount:0, DeletedByteCount:0, ReadOnly:false, ModifiedAtSecond:0 from 10.0.0.1:8080 I0824 22:18:30.273504 volume_layout.go:815 Volume 118 becomes unwritable I0824 22:18:30.273523 volume_layout.go:335 volume 118 does not have enough copies I0824 22:18:30.273588 data_node.go:114 Deleting volume id: 119 I0824 22:18:30.273751 topology.go:534 removing volume info: Id:119, Size:1, ReplicaPlacement:000, Collection:c, Version:3, Ttl:, FileCount:0, DeleteCount:0, DeletedByteCount:0, ReadOnly:false, ModifiedAtSecond:0 from 10.0.0.1:8080 I0824 22:18:30.273904 volume_layout.go:815 Volume 119 becomes unwritable I0824 22:18:30.274051 volume_layout.go:335 volume 119 does not have enough copies I0824 22:18:30.274130 data_node.go:114 Deleting volume id: 120 I0824 22:18:30.274232 topology.go:534 removing volume info: Id:120, Size:1, ReplicaPlacement:000, Collection:c, Version:3, Ttl:, FileCount:0, DeleteCount:0, DeletedByteCount:0, ReadOnly:false, ModifiedAtSecond:0 from 10.0.0.1:8080 I0824 22:18:30.274542 volume_layout.go:815 Volume 120 becomes unwritable I0824 22:18:30.274601 volume_layout.go:335 volume 120 does not have enough copies I0824 22:18:30.274728 data_node.go:114 Deleting volume id: 121 I0824 22:18:30.274911 topology.go:534 removing volume info: Id:121, Size:1, ReplicaPlacement:000, Collection:c, Version:3, Ttl:, FileCount:0, DeleteCount:0, DeletedByteCount:0, ReadOnly:false, ModifiedAtSecond:0 from 10.0.0.1:8080 I0824 22:18:30.275059 volume_layout.go:815 Volume 121 becomes unwritable I0824 22:18:30.275112 volume_layout.go:335 volume 121 does not have enough copies I0824 22:18:30.275333 data_node.go:114 Deleting volume id: 122 I0824 22:18:30.275465 topology.go:534 removing volume info: Id:122, Size:1, ReplicaPlacement:000, Collection:c, Version:3, Ttl:, FileCount:0, DeleteCount:0, DeletedByteCount:0, ReadOnly:false, ModifiedAtSecond:0 from 10.0.0.1:8080 I0824 22:18:30.275653 volume_layout.go:815 Volume 122 becomes unwritable I0824 22:18:30.275807 volume_layout.go:335 volume 122 does not have enough copies I0824 22:18:30.276125 data_node.go:114 Deleting volume id: 123 I0824 22:18:30.276421 topology.go:534 removing volume info: Id:123, Size:1, ReplicaPlacement:000, Collection:c, Version:3, Ttl:, FileCount:0, DeleteCount:0, DeletedByteCount:0, ReadOnly:false, ModifiedAtSecond:0 from 10.0.0.1:8080 I0824 22:18:30.276619 volume_layout.go:815 Volume 123 becomes unwritable I0824 22:18:30.276731 volume_layout.go:335 volume 123 does not have enough copies I0824 22:18:30.277091 data_node.go:114 Deleting volume id: 124 I0824 22:18:30.277557 topology.go:534 removing volume info: Id:124, Size:1, ReplicaPlacement:000, Collection:c, Version:3, Ttl:, FileCount:0, DeleteCount:0, DeletedByteCount:0, ReadOnly:false, ModifiedAtSecond:0 from 10.0.0.1:8080 I0824 22:18:30.277800 volume_layout.go:815 Volume 124 becomes unwritable I0824 22:18:30.278009 volume_layout.go:335 volume 124 does not have enough copies I0824 22:18:30.278647 data_node.go:114 Deleting volume id: 125 I0824 22:18:30.278881 topology.go:534 removing volume info: Id:125, Size:1, ReplicaPlacement:000, Collection:c, Version:3, Ttl:, FileCount:0, DeleteCount:0, DeletedByteCount:0, ReadOnly:false, ModifiedAtSecond:0 from 10.0.0.1:8080 I0824 22:18:30.279038 volume_layout.go:815 Volume 125 becomes unwritable I0824 22:18:30.279168 volume_layout.go:335 volume 125 does not have enough copies I0824 22:18:30.279391 data_node.go:114 Deleting volume id: 126 I0824 22:18:30.279527 topology.go:534 removing volume info: Id:126, Size:1, ReplicaPlacement:000, Collection:c, Version:3, Ttl:, FileCount:0, DeleteCount:0, DeletedByteCount:0, ReadOnly:false, ModifiedAtSecond:0 from 10.0.0.1:8080 I0824 22:18:30.279572 volume_layout.go:815 Volume 126 becomes unwritable I0824 22:18:30.279606 volume_layout.go:335 volume 126 does not have enough copies I0824 22:18:30.279820 data_node.go:114 Deleting volume id: 127 I0824 22:18:30.280050 topology.go:534 removing volume info: Id:127, Size:1, ReplicaPlacement:000, Collection:c, Version:3, Ttl:, FileCount:0, DeleteCount:0, DeletedByteCount:0, ReadOnly:false, ModifiedAtSecond:0 from 10.0.0.1:8080 I0824 22:18:30.280261 volume_layout.go:815 Volume 127 becomes unwritable I0824 22:18:30.280452 volume_layout.go:335 volume 127 does not have enough copies I0824 22:18:30.280778 data_node.go:114 Deleting volume id: 128 I0824 22:18:30.281244 topology.go:534 removing volume info: Id:128, Size:1, ReplicaPlacement:000, Collection:c, Version:3, Ttl:, FileCount:0, DeleteCount:0, DeletedByteCount:0, ReadOnly:false, ModifiedAtSecond:0 from 10.0.0.1:8080 I0824 22:18:30.281472 volume_layout.go:815 Volume 128 becomes unwritable I0824 22:18:30.281648 volume_layout.go:335 volume 128 does not have enough copies I0824 22:18:30.282020 data_node.go:114 Deleting volume id: 129 I0824 22:18:30.282480 topology.go:534 removing volume info: Id:129, Size:1, ReplicaPlacement:000, Collection:c, Version:3, Ttl:, FileCount:0, DeleteCount:0, DeletedByteCount:0, ReadOnly:false, ModifiedAtSecond:0 from 10.0.0.1:8080 I0824 22:18:30.282768 volume_layout.go:815 Volume 129 becomes unwritable I0824 22:18:30.283028 volume_layout.go:335 volume 129 does not have enough copies I0824 22:18:30.283362 data_node.go:114 Deleting volume id: 130 I0824 22:18:30.283670 topology.go:534 removing volume info: Id:130, Size:1, ReplicaPlacement:000, Collection:c, Version:3, Ttl:, FileCount:0, DeleteCount:0, DeletedByteCount:0, ReadOnly:false, ModifiedAtSecond:0 from 10.0.0.1:8080 I0824 22:18:30.283790 volume_layout.go:815 Volume 130 becomes unwritable I0824 22:18:30.283824 volume_layout.go:335 volume 130 does not have enough copies I0824 22:18:30.284037 data_node.go:114 Deleting volume id: 131 I0824 22:18:30.284160 topology.go:534 removing volume info: Id:131, Size:1, ReplicaPlacement:000, Collection:c, Version:3, Ttl:, FileCount:0, DeleteCount:0, DeletedByteCount:0, ReadOnly:false, ModifiedAtSecond:0 from 10.0.0.1:8080 I0824 22:18:30.284220 volume_layout.go:815 Volume 131 becomes unwritable I0824 22:18:30.284245 volume_layout.go:335 volume 131 does not have enough copies I0824 22:18:30.284404 data_node.go:114 Deleting volume id: 132 I0824 22:18:30.284543 topology.go:534 removing volume info: Id:132, Size:1, ReplicaPlacement:000, Collection:c, Version:3, Ttl:, FileCount:0, DeleteCount:0, DeletedByteCount:0, ReadOnly:false, ModifiedAtSecond:0 from 10.0.0.1:8080 I0824 22:18:30.284613 volume_layout.go:815 Volume 132 becomes unwritable I0824 22:18:30.284641 volume_layout.go:335 volume 132 does not have enough copies I0824 22:18:30.284755 data_node.go:114 Deleting volume id: 133 I0824 22:18:30.284832 topology.go:534 removing volume info: Id:133, Size:1, ReplicaPlacement:000, Collection:c, Version:3, Ttl:, FileCount:0, DeleteCount:0, DeletedByteCount:0, ReadOnly:false, ModifiedAtSecond:0 from 10.0.0.1:8080 I0824 22:18:30.284872 volume_layout.go:815 Volume 133 becomes unwritable I0824 22:18:30.284899 volume_layout.go:335 volume 133 does not have enough copies I0824 22:18:30.285056 data_node.go:114 Deleting volume id: 134 I0824 22:18:30.285192 topology.go:534 removing volume info: Id:134, Size:1, ReplicaPlacement:000, Collection:c, Version:3, Ttl:, FileCount:0, DeleteCount:0, DeletedByteCount:0, ReadOnly:false, ModifiedAtSecond:0 from 10.0.0.1:8080 I0824 22:18:30.285231 volume_layout.go:815 Volume 134 becomes unwritable I0824 22:18:30.285256 volume_layout.go:335 volume 134 does not have enough copies I0824 22:18:30.285406 data_node.go:114 Deleting volume id: 135 I0824 22:18:30.285519 topology.go:534 removing volume info: Id:135, Size:1, ReplicaPlacement:000, Collection:c, Version:3, Ttl:, FileCount:0, DeleteCount:0, DeletedByteCount:0, ReadOnly:false, ModifiedAtSecond:0 from 10.0.0.1:8080 I0824 22:18:30.285583 volume_layout.go:815 Volume 135 becomes unwritable I0824 22:18:30.285604 volume_layout.go:335 volume 135 does not have enough copies I0824 22:18:30.285700 data_node.go:114 Deleting volume id: 136 I0824 22:18:30.285854 topology.go:534 removing volume info: Id:136, Size:1, ReplicaPlacement:000, Collection:c, Version:3, Ttl:, FileCount:0, DeleteCount:0, DeletedByteCount:0, ReadOnly:false, ModifiedAtSecond:0 from 10.0.0.1:8080 I0824 22:18:30.285914 volume_layout.go:815 Volume 136 becomes unwritable I0824 22:18:30.285942 volume_layout.go:335 volume 136 does not have enough copies I0824 22:18:30.286082 data_node.go:114 Deleting volume id: 137 I0824 22:18:30.286161 topology.go:534 removing volume info: Id:137, Size:1, ReplicaPlacement:000, Collection:c, Version:3, Ttl:, FileCount:0, DeleteCount:0, DeletedByteCount:0, ReadOnly:false, ModifiedAtSecond:0 from 10.0.0.1:8080 I0824 22:18:30.286194 volume_layout.go:815 Volume 137 becomes unwritable I0824 22:18:30.286214 volume_layout.go:335 volume 137 does not have enough copies I0824 22:18:30.286280 data_node.go:114 Deleting volume id: 138 I0824 22:18:30.286324 topology.go:534 removing volume info: Id:138, Size:1, ReplicaPlacement:000, Collection:c, Version:3, Ttl:, FileCount:0, DeleteCount:0, DeletedByteCount:0, ReadOnly:false, ModifiedAtSecond:0 from 10.0.0.1:8080 I0824 22:18:30.286357 volume_layout.go:815 Volume 138 becomes unwritable I0824 22:18:30.286375 volume_layout.go:335 volume 138 does not have enough copies I0824 22:18:30.286522 data_node.go:114 Deleting volume id: 139 I0824 22:18:30.286617 topology.go:534 removing volume info: Id:139, Size:1, ReplicaPlacement:000, Collection:c, Version:3, Ttl:, FileCount:0, DeleteCount:0, DeletedByteCount:0, ReadOnly:false, ModifiedAtSecond:0 from 10.0.0.1:8080 I0824 22:18:30.286652 volume_layout.go:815 Volume 139 becomes unwritable I0824 22:18:30.286689 volume_layout.go:335 volume 139 does not have enough copies I0824 22:18:30.286822 data_node.go:114 Deleting volume id: 140 I0824 22:18:30.286927 topology.go:534 removing volume info: Id:140, Size:1, ReplicaPlacement:000, Collection:c, Version:3, Ttl:, FileCount:0, DeleteCount:0, DeletedByteCount:0, ReadOnly:false, ModifiedAtSecond:0 from 10.0.0.1:8080 I0824 22:18:30.286960 volume_layout.go:815 Volume 140 becomes unwritable I0824 22:18:30.286978 volume_layout.go:335 volume 140 does not have enough copies I0824 22:18:30.287058 data_node.go:114 Deleting volume id: 141 I0824 22:18:30.287210 topology.go:534 removing volume info: Id:141, Size:1, ReplicaPlacement:000, Collection:c, Version:3, Ttl:, FileCount:0, DeleteCount:0, DeletedByteCount:0, ReadOnly:false, ModifiedAtSecond:0 from 10.0.0.1:8080 I0824 22:18:30.287295 volume_layout.go:815 Volume 141 becomes unwritable I0824 22:18:30.287365 volume_layout.go:335 volume 141 does not have enough copies I0824 22:18:30.287433 data_node.go:114 Deleting volume id: 142 I0824 22:18:30.287475 topology.go:534 removing volume info: Id:142, Size:1, ReplicaPlacement:000, Collection:c, Version:3, Ttl:, FileCount:0, DeleteCount:0, DeletedByteCount:0, ReadOnly:false, ModifiedAtSecond:0 from 10.0.0.1:8080 I0824 22:18:30.287507 volume_layout.go:815 Volume 142 becomes unwritable I0824 22:18:30.287526 volume_layout.go:335 volume 142 does not have enough copies I0824 22:18:30.287651 data_node.go:114 Deleting volume id: 143 I0824 22:18:30.287749 topology.go:534 removing volume info: Id:143, Size:1, ReplicaPlacement:000, Collection:c, Version:3, Ttl:, FileCount:0, DeleteCount:0, DeletedByteCount:0, ReadOnly:false, ModifiedAtSecond:0 from 10.0.0.1:8080 I0824 22:18:30.287783 volume_layout.go:815 Volume 143 becomes unwritable I0824 22:18:30.287803 volume_layout.go:335 volume 143 does not have enough copies I0824 22:18:30.287863 data_node.go:114 Deleting volume id: 144 I0824 22:18:30.287919 topology.go:534 removing volume info: Id:144, Size:1, ReplicaPlacement:000, Collection:c, Version:3, Ttl:, FileCount:0, DeleteCount:0, DeletedByteCount:0, ReadOnly:false, ModifiedAtSecond:0 from 10.0.0.1:8080 I0824 22:18:30.287951 volume_layout.go:815 Volume 144 becomes unwritable I0824 22:18:30.287989 volume_layout.go:335 volume 144 does not have enough copies I0824 22:18:30.288049 data_node.go:114 Deleting volume id: 145 I0824 22:18:30.288093 topology.go:534 removing volume info: Id:145, Size:1, ReplicaPlacement:000, Collection:c, Version:3, Ttl:, FileCount:0, DeleteCount:0, DeletedByteCount:0, ReadOnly:false, ModifiedAtSecond:0 from 10.0.0.1:8080 I0824 22:18:30.288127 volume_layout.go:815 Volume 145 becomes unwritable I0824 22:18:30.288147 volume_layout.go:335 volume 145 does not have enough copies --- PASS: TestVolumeFilterListsWhileVolumesChange (0.22s) === RUN TestVolumeGrowth_ReservationBasedAllocation I0824 22:18:30.288616 node.go:471 weedfs adds child dc1 I0824 22:18:30.288692 node.go:471 weedfs:dc1 adds child rack1 I0824 22:18:30.288745 node.go:471 weedfs:dc1:rack1 adds child server1 I0824 22:18:30.288790 node.go:471 weedfs:dc1:rack1:server1 adds child I0824 22:18:30.289090 node.go:220 weedfs failed to pick 1 from 0 node candidates --- PASS: TestVolumeGrowth_ReservationBasedAllocation (0.00s) === RUN TestVolumeGrowth_ConcurrentAllocationPreventsRaceCondition I0824 22:18:30.289271 node.go:471 weedfs adds child dc1 I0824 22:18:30.289303 node.go:471 weedfs:dc1 adds child rack1 I0824 22:18:30.289336 node.go:471 weedfs:dc1:rack1 adds child server1 I0824 22:18:30.289373 node.go:471 weedfs:dc1:rack1:server1 adds child volume_growth_reservation_test.go:148: Request 9 succeeded, got reservation volume_growth_reservation_test.go:148: Request 0 succeeded, got reservation volume_growth_reservation_test.go:148: Request 1 succeeded, got reservation volume_growth_reservation_test.go:148: Request 2 succeeded, got reservation volume_growth_reservation_test.go:148: Request 3 succeeded, got reservation I0824 22:18:30.290372 node.go:220 weedfs failed to pick 1 from 0 node candidates volume_growth_reservation_test.go:145: Request 4 failed as expected: Not enough data nodes found! I0824 22:18:30.290445 node.go:220 weedfs failed to pick 1 from 0 node candidates volume_growth_reservation_test.go:145: Request 5 failed as expected: Not enough data nodes found! I0824 22:18:30.290507 node.go:220 weedfs failed to pick 1 from 0 node candidates volume_growth_reservation_test.go:145: Request 6 failed as expected: Not enough data nodes found! I0824 22:18:30.290568 node.go:220 weedfs failed to pick 1 from 0 node candidates volume_growth_reservation_test.go:145: Request 7 failed as expected: Not enough data nodes found! I0824 22:18:30.290625 node.go:220 weedfs failed to pick 1 from 0 node candidates volume_growth_reservation_test.go:145: Request 8 failed as expected: Not enough data nodes found! volume_growth_reservation_test.go:204: Concurrent test completed: 5 successes, 5 failures --- PASS: TestVolumeGrowth_ConcurrentAllocationPreventsRaceCondition (0.00s) === RUN TestVolumeGrowth_ReservationFailureRollback I0824 22:18:30.290872 node.go:471 weedfs adds child dc1 I0824 22:18:30.291091 node.go:471 weedfs:dc1 adds child rack1 I0824 22:18:30.291132 node.go:471 weedfs:dc1:rack1 adds child server1 I0824 22:18:30.291158 node.go:471 weedfs:dc1:rack1 adds child server2 I0824 22:18:30.291194 node.go:471 weedfs:dc1:rack1:server1 adds child I0824 22:18:30.291225 node.go:471 weedfs:dc1:rack1:server2 adds child --- PASS: TestVolumeGrowth_ReservationFailureRollback (0.00s) === RUN TestVolumeGrowth_ReservationTimeout --- PASS: TestVolumeGrowth_ReservationTimeout (0.00s) === RUN TestFindEmptySlotsForOneVolume data: map[dc1:map[rack1:map[server111:map[limit:3 volumes:[map[id:1 size:12312] map[id:2 size:12312] map[id:3 size:12312]]] server112:map[limit:10 volumes:[map[id:4 size:12312] map[id:5 size:12312] map[id:6 size:12312]]]] rack2:map[server121:map[limit:4 volumes:[map[id:4 size:12312] map[id:5 size:12312] map[id:6 size:12312]]] server122:map[limit:4 volumes:[]] server123:map[limit:5 volumes:[map[id:2 size:12312] map[id:3 size:12312] map[id:4 size:12312]]]]] dc2:map[] dc3:map[rack2:map[server321:map[limit:4 volumes:[map[id:1 size:12312] map[id:3 size:12312] map[id:5 size:12312]]]]]] I0824 22:18:30.292411 node.go:471 weedfs adds child dc1 I0824 22:18:30.292449 node.go:471 weedfs:dc1 adds child rack1 I0824 22:18:30.292488 node.go:471 weedfs:dc1:rack1 adds child server111 I0824 22:18:30.292519 node.go:471 weedfs:dc1:rack1:server111 adds child I0824 22:18:30.292600 node.go:471 weedfs:dc1:rack1 adds child server112 I0824 22:18:30.292628 node.go:471 weedfs:dc1:rack1:server112 adds child I0824 22:18:30.292716 node.go:471 weedfs:dc1 adds child rack2 I0824 22:18:30.292748 node.go:471 weedfs:dc1:rack2 adds child server121 I0824 22:18:30.292776 node.go:471 weedfs:dc1:rack2:server121 adds child I0824 22:18:30.292828 node.go:471 weedfs:dc1:rack2 adds child server122 I0824 22:18:30.292871 node.go:471 weedfs:dc1:rack2:server122 adds child I0824 22:18:30.292904 node.go:471 weedfs:dc1:rack2 adds child server123 I0824 22:18:30.292957 node.go:471 weedfs:dc1:rack2:server123 adds child I0824 22:18:30.293009 node.go:471 weedfs adds child dc2 I0824 22:18:30.293033 node.go:471 weedfs adds child dc3 I0824 22:18:30.293055 node.go:471 weedfs:dc3 adds child rack2 I0824 22:18:30.293117 node.go:471 weedfs:dc3:rack2 adds child server321 I0824 22:18:30.293154 node.go:471 weedfs:dc3:rack2:server321 adds child assigned node : server123 assigned node : server121 assigned node : server122 --- PASS: TestFindEmptySlotsForOneVolume (0.00s) === RUN TestReplication011 data: map[dc1:map[rack1:map[server111:map[limit:300 volumes:[map[id:1 size:12312] map[id:2 size:12312] map[id:3 size:12312]]] server112:map[limit:300 volumes:[map[id:4 size:12312] map[id:5 size:12312] map[id:6 size:12312]]] server113:map[limit:300 volumes:[]] server114:map[limit:300 volumes:[]] server115:map[limit:300 volumes:[]] server116:map[limit:300 volumes:[]]] rack2:map[server121:map[limit:300 volumes:[map[id:4 size:12312] map[id:5 size:12312] map[id:6 size:12312]]] server122:map[limit:300 volumes:[]] server123:map[limit:300 volumes:[map[id:2 size:12312] map[id:3 size:12312] map[id:4 size:12312]]] server124:map[limit:300 volumes:[]] server125:map[limit:300 volumes:[]] server126:map[limit:300 volumes:[]]] rack3:map[server131:map[limit:300 volumes:[]] server132:map[limit:300 volumes:[]] server133:map[limit:300 volumes:[]] server134:map[limit:300 volumes:[]] server135:map[limit:300 volumes:[]] server136:map[limit:300 volumes:[]]]]] I0824 22:18:30.294763 node.go:471 weedfs adds child dc1 I0824 22:18:30.294810 node.go:471 weedfs:dc1 adds child rack1 I0824 22:18:30.294880 node.go:471 weedfs:dc1:rack1 adds child server116 I0824 22:18:30.295021 node.go:471 weedfs:dc1:rack1:server116 adds child I0824 22:18:30.295114 node.go:471 weedfs:dc1:rack1 adds child server111 I0824 22:18:30.295150 node.go:471 weedfs:dc1:rack1:server111 adds child I0824 22:18:30.295212 node.go:471 weedfs:dc1:rack1 adds child server112 I0824 22:18:30.295255 node.go:471 weedfs:dc1:rack1:server112 adds child I0824 22:18:30.295333 node.go:471 weedfs:dc1:rack1 adds child server113 I0824 22:18:30.295360 node.go:471 weedfs:dc1:rack1:server113 adds child I0824 22:18:30.295407 node.go:471 weedfs:dc1:rack1 adds child server114 I0824 22:18:30.295430 node.go:471 weedfs:dc1:rack1:server114 adds child I0824 22:18:30.295458 node.go:471 weedfs:dc1:rack1 adds child server115 I0824 22:18:30.295480 node.go:471 weedfs:dc1:rack1:server115 adds child I0824 22:18:30.295506 node.go:471 weedfs:dc1 adds child rack2 I0824 22:18:30.295530 node.go:471 weedfs:dc1:rack2 adds child server124 I0824 22:18:30.295551 node.go:471 weedfs:dc1:rack2:server124 adds child I0824 22:18:30.295579 node.go:471 weedfs:dc1:rack2 adds child server125 I0824 22:18:30.295601 node.go:471 weedfs:dc1:rack2:server125 adds child I0824 22:18:30.295638 node.go:471 weedfs:dc1:rack2 adds child server126 I0824 22:18:30.295664 node.go:471 weedfs:dc1:rack2:server126 adds child I0824 22:18:30.295742 node.go:471 weedfs:dc1:rack2 adds child server121 I0824 22:18:30.295846 node.go:471 weedfs:dc1:rack2:server121 adds child I0824 22:18:30.295960 node.go:471 weedfs:dc1:rack2 adds child server122 I0824 22:18:30.295985 node.go:471 weedfs:dc1:rack2:server122 adds child I0824 22:18:30.296016 node.go:471 weedfs:dc1:rack2 adds child server123 I0824 22:18:30.296055 node.go:471 weedfs:dc1:rack2:server123 adds child I0824 22:18:30.296119 node.go:471 weedfs:dc1 adds child rack3 I0824 22:18:30.296174 node.go:471 weedfs:dc1:rack3 adds child server135 I0824 22:18:30.296239 node.go:471 weedfs:dc1:rack3:server135 adds child I0824 22:18:30.296277 node.go:471 weedfs:dc1:rack3 adds child server136 I0824 22:18:30.296309 node.go:471 weedfs:dc1:rack3:server136 adds child I0824 22:18:30.296340 node.go:471 weedfs:dc1:rack3 adds child server131 I0824 22:18:30.296365 node.go:471 weedfs:dc1:rack3:server131 adds child I0824 22:18:30.296396 node.go:471 weedfs:dc1:rack3 adds child server132 I0824 22:18:30.296418 node.go:471 weedfs:dc1:rack3:server132 adds child I0824 22:18:30.296447 node.go:471 weedfs:dc1:rack3 adds child server133 I0824 22:18:30.296468 node.go:471 weedfs:dc1:rack3:server133 adds child I0824 22:18:30.296496 node.go:471 weedfs:dc1:rack3 adds child server134 I0824 22:18:30.296543 node.go:471 weedfs:dc1:rack3:server134 adds child assigned node : server136 assigned node : server131 assigned node : server126 --- PASS: TestReplication011 (0.00s) === RUN TestFindEmptySlotsForOneVolumeScheduleByWeight data: map[dc1:map[rack1:map[server111:map[limit:2000 volumes:[]]]] dc2:map[rack2:map[server222:map[limit:2000 volumes:[]]]] dc3:map[rack3:map[server333:map[limit:1000 volumes:[]]]] dc4:map[rack4:map[server444:map[limit:1000 volumes:[]]]] dc5:map[rack5:map[server555:map[limit:500 volumes:[]]]] dc6:map[rack6:map[server666:map[limit:500 volumes:[]]]]] I0824 22:18:30.297404 node.go:471 weedfs adds child dc5 I0824 22:18:30.297440 node.go:471 weedfs:dc5 adds child rack5 I0824 22:18:30.297479 node.go:471 weedfs:dc5:rack5 adds child server555 I0824 22:18:30.297548 node.go:471 weedfs:dc5:rack5:server555 adds child I0824 22:18:30.297595 node.go:471 weedfs adds child dc6 I0824 22:18:30.297619 node.go:471 weedfs:dc6 adds child rack6 I0824 22:18:30.297662 node.go:471 weedfs:dc6:rack6 adds child server666 I0824 22:18:30.297687 node.go:471 weedfs:dc6:rack6:server666 adds child I0824 22:18:30.297719 node.go:471 weedfs adds child dc1 I0824 22:18:30.297741 node.go:471 weedfs:dc1 adds child rack1 I0824 22:18:30.297785 node.go:471 weedfs:dc1:rack1 adds child server111 I0824 22:18:30.297812 node.go:471 weedfs:dc1:rack1:server111 adds child I0824 22:18:30.297845 node.go:471 weedfs adds child dc2 I0824 22:18:30.297885 node.go:471 weedfs:dc2 adds child rack2 I0824 22:18:30.299248 node.go:471 weedfs:dc2:rack2 adds child server222 I0824 22:18:30.299320 node.go:471 weedfs:dc2:rack2:server222 adds child I0824 22:18:30.299557 node.go:471 weedfs adds child dc3 I0824 22:18:30.299589 node.go:471 weedfs:dc3 adds child rack3 I0824 22:18:30.299613 node.go:471 weedfs:dc3:rack3 adds child server333 I0824 22:18:30.299636 node.go:471 weedfs:dc3:rack3:server333 adds child I0824 22:18:30.299671 node.go:471 weedfs adds child dc4 I0824 22:18:30.299694 node.go:471 weedfs:dc4 adds child rack4 I0824 22:18:30.299716 node.go:471 weedfs:dc4:rack4 adds child server444 I0824 22:18:30.299736 node.go:471 weedfs:dc4:rack4:server444 adds child server444 : 319 server555 : 169 server666 : 147 server333 : 279 server222 : 530 server111 : 556 --- PASS: TestFindEmptySlotsForOneVolumeScheduleByWeight (0.05s) === RUN TestPickForWrite data: map[dc1:map[rack1:map[serverdc111:map[ip:127.0.0.1 limit:100 volumes:[map[collection:test id:1 replication:001 size:12312] map[collection:test id:2 replication:100 size:12312] map[collection:test id:4 replication:100 size:12312] map[collection:test id:6 replication:010 size:12312]]]]] dc2:map[rack1:map[serverdc211:map[ip:127.0.0.2 limit:100 volumes:[map[collection:test id:2 replication:100 size:12312] map[collection:test id:3 replication:010 size:12312] map[collection:test id:5 replication:001 size:12312] map[collection:test id:6 replication:010 size:12312]]]]] dc3:map[rack1:map[serverdc311:map[ip:127.0.0.3 limit:100 volumes:[map[collection:test id:1 replication:001 size:12312] map[collection:test id:3 replication:010 size:12312] map[collection:test id:4 replication:100 size:12312] map[collection:test id:5 replication:001 size:12312]]]]]] I0824 22:18:30.351072 node.go:471 weedfs adds child dc3 I0824 22:18:30.351144 node.go:471 weedfs:dc3 adds child rack1 I0824 22:18:30.351220 node.go:471 weedfs:dc3:rack1 adds child serverdc311 I0824 22:18:30.351296 node.go:471 weedfs:dc3:rack1:serverdc311 adds child I0824 22:18:30.351528 node.go:471 weedfs adds child dc1 I0824 22:18:30.351594 node.go:471 weedfs:dc1 adds child rack1 I0824 22:18:30.351626 node.go:471 weedfs:dc1:rack1 adds child serverdc111 I0824 22:18:30.351671 volume_layout.go:815 Volume 1 becomes unwritable I0824 22:18:30.351706 node.go:471 weedfs:dc1:rack1:serverdc111 adds child I0824 22:18:30.351772 volume_layout.go:815 Volume 4 becomes unwritable I0824 22:18:30.351897 node.go:471 weedfs adds child dc2 I0824 22:18:30.351926 node.go:471 weedfs:dc2 adds child rack1 I0824 22:18:30.351951 node.go:471 weedfs:dc2:rack1 adds child serverdc211 I0824 22:18:30.351978 volume_layout.go:815 Volume 2 becomes unwritable I0824 22:18:30.352002 node.go:471 weedfs:dc2:rack1:serverdc211 adds child I0824 22:18:30.352051 volume_layout.go:815 Volume 3 becomes unwritable I0824 22:18:30.352118 volume_layout.go:815 Volume 5 becomes unwritable I0824 22:18:30.352156 volume_layout.go:815 Volume 6 becomes unwritable --- PASS: TestPickForWrite (0.00s) === RUN TestPickNodesByWeightPrefersDistinctHosts I0824 22:18:30.352708 node.go:471 rack1 adds child s1a I0824 22:18:30.352751 node.go:471 rack1:s1a adds child I0824 22:18:30.352791 node.go:471 rack1 adds child s1b I0824 22:18:30.352872 node.go:471 rack1:s1b adds child I0824 22:18:30.352902 node.go:471 rack1 adds child s2a I0824 22:18:30.352925 node.go:471 rack1:s2a adds child I0824 22:18:30.352952 node.go:471 rack1 adds child s2b I0824 22:18:30.352977 node.go:471 rack1:s2b adds child --- PASS: TestPickNodesByWeightPrefersDistinctHosts (0.00s) === RUN TestSetVolumeCapacityFullStampsFullSinceAndRecovers I0824 22:18:30.354185 node.go:471 weedfs adds child dc1 I0824 22:18:30.354250 node.go:471 weedfs:dc1 adds child rack1 I0824 22:18:30.354296 node.go:471 weedfs:dc1:rack1 adds child server1 I0824 22:18:30.354538 node.go:471 weedfs:dc1:rack1:server1 adds child I0824 22:18:30.354621 volume_layout.go:815 Volume 1 becomes unwritable I0824 22:18:30.354670 volume_layout.go:944 Volume 1 reaches full capacity. I0824 22:18:30.354756 volume_layout.go:268 Volume 1 recovered to writable (effective=4000, reported=4000, limit=10000). --- PASS: TestSetVolumeCapacityFullStampsFullSinceAndRecovers (0.00s) === RUN TestSetVolumeAvailableRestoresActiveCountForCapacityFullVolume I0824 22:18:30.355388 node.go:471 weedfs adds child dc1 I0824 22:18:30.355519 node.go:471 weedfs:dc1 adds child rack1 I0824 22:18:30.355675 node.go:471 weedfs:dc1:rack1 adds child server1 I0824 22:18:30.355848 node.go:471 weedfs:dc1:rack1:server1 adds child I0824 22:18:30.356046 volume_layout.go:815 Volume 1 becomes unwritable I0824 22:18:30.356092 volume_layout.go:944 Volume 1 reaches full capacity. --- PASS: TestSetVolumeAvailableRestoresActiveCountForCapacityFullVolume (0.00s) === RUN TestGetPendingSize I0824 22:18:30.356861 node.go:471 weedfs adds child dc1 I0824 22:18:30.356922 node.go:471 weedfs:dc1 adds child rack1 I0824 22:18:30.357027 node.go:471 weedfs:dc1:rack1 adds child server1 I0824 22:18:30.357128 node.go:471 weedfs:dc1:rack1:server1 adds child --- PASS: TestGetPendingSize (0.00s) === RUN TestGetPendingSize_CompactionResets I0824 22:18:30.357499 node.go:471 weedfs adds child dc1 I0824 22:18:30.357636 node.go:471 weedfs:dc1 adds child rack1 I0824 22:18:30.357675 node.go:471 weedfs:dc1:rack1 adds child server1 I0824 22:18:30.357736 node.go:471 weedfs:dc1:rack1:server1 adds child --- PASS: TestGetPendingSize_CompactionResets (0.00s) === RUN TestDrainAndRemoveFromWritable_NoPending I0824 22:18:30.358131 node.go:471 weedfs adds child dc1 I0824 22:18:30.358225 node.go:471 weedfs:dc1 adds child rack1 I0824 22:18:30.358276 node.go:471 weedfs:dc1:rack1 adds child server1 I0824 22:18:30.358329 node.go:471 weedfs:dc1:rack1:server1 adds child I0824 22:18:30.358417 volume_layout.go:815 Volume 1 becomes unwritable --- PASS: TestDrainAndRemoveFromWritable_NoPending (0.00s) === RUN TestDrainAndRemoveFromWritable_WithPending I0824 22:18:30.358729 node.go:471 weedfs adds child dc1 I0824 22:18:30.358814 node.go:471 weedfs:dc1 adds child rack1 I0824 22:18:30.358876 node.go:471 weedfs:dc1:rack1 adds child server1 I0824 22:18:30.358969 node.go:471 weedfs:dc1:rack1:server1 adds child I0824 22:18:30.359032 volume_layout.go:815 Volume 1 becomes unwritable I0824 22:18:30.359106 volume_layout.go:391 Volume 1 reaches full capacity (effective=2098151, limit=10000). --- PASS: TestDrainAndRemoveFromWritable_WithPending (0.00s) === RUN TestDrainAndRemoveFromWritable_DecaysViaConcurrentHeartbeat I0824 22:18:30.359367 node.go:471 weedfs adds child dc1 I0824 22:18:30.359474 node.go:471 weedfs:dc1 adds child rack1 I0824 22:18:30.359506 node.go:471 weedfs:dc1:rack1 adds child server1 I0824 22:18:30.359552 node.go:471 weedfs:dc1:rack1:server1 adds child I0824 22:18:30.359618 volume_layout.go:815 Volume 1 becomes unwritable I0824 22:18:30.359644 volume_layout.go:391 Volume 1 reaches full capacity (effective=104858600, limit=10000). I0824 22:18:30.863920 volume_layout.go:959 Volume 1 becomes crowded --- PASS: TestDrainAndRemoveFromWritable_DecaysViaConcurrentHeartbeat (5.08s) === RUN TestSetVolumeReadOnly_PreservesPending I0824 22:18:35.443293 node.go:471 weedfs adds child dc1 I0824 22:18:35.443558 node.go:471 weedfs:dc1 adds child rack1 I0824 22:18:35.443649 node.go:471 weedfs:dc1:rack1 adds child server1 I0824 22:18:35.443744 node.go:471 weedfs:dc1:rack1:server1 adds child I0824 22:18:35.443862 volume_layout.go:815 Volume 1 becomes unwritable --- PASS: TestSetVolumeReadOnly_PreservesPending (0.00s) === RUN TestShouldGrowVolumesByDcAndRack_Issue8986 data: map[dc_edge:map[rack1:map[server1:map[ip:10.0.0.1 limit:30 volumes:[map[id:1 replication:010 size:12312] map[id:2 replication:010 size:12312]]]] rack2:map[server2:map[ip:10.0.0.2 limit:30 volumes:[map[id:1 replication:010 size:12312] map[id:2 replication:010 size:12312]]]] rack3:map[server3:map[ip:10.0.0.3 limit:30 volumes:[]]]]] I0824 22:18:35.444648 node.go:471 weedfs adds child dc_edge I0824 22:18:35.444744 node.go:471 weedfs:dc_edge adds child rack2 I0824 22:18:35.444811 node.go:471 weedfs:dc_edge:rack2 adds child server2 I0824 22:18:35.444895 node.go:471 weedfs:dc_edge:rack2:server2 adds child I0824 22:18:35.445031 node.go:471 weedfs:dc_edge adds child rack3 I0824 22:18:35.445072 node.go:471 weedfs:dc_edge:rack3 adds child server3 I0824 22:18:35.445104 node.go:471 weedfs:dc_edge:rack3:server3 adds child I0824 22:18:35.445162 node.go:471 weedfs:dc_edge adds child rack1 I0824 22:18:35.445198 node.go:471 weedfs:dc_edge:rack1 adds child server1 I0824 22:18:35.445236 volume_layout.go:815 Volume 1 becomes unwritable I0824 22:18:35.445269 node.go:471 weedfs:dc_edge:rack1:server1 adds child I0824 22:18:35.445333 volume_layout.go:815 Volume 2 becomes unwritable --- PASS: TestShouldGrowVolumesByDcAndRack_Issue8986 (0.00s) === RUN TestPlanRackAwareGrowth_Issue9832 I0824 22:18:35.445924 node.go:471 weedfs adds child datacenter1 I0824 22:18:35.445987 node.go:471 weedfs:datacenter1 adds child rack1 I0824 22:18:35.446037 node.go:471 weedfs:datacenter1:rack1 adds child node-a I0824 22:18:35.446119 node.go:471 weedfs:datacenter1:rack1:node-a adds child I0824 22:18:35.446204 node.go:471 weedfs:datacenter1 adds child rack2 I0824 22:18:35.446410 node.go:471 weedfs:datacenter1:rack2 adds child node-b I0824 22:18:35.446493 volume_layout.go:815 Volume 12 becomes unwritable I0824 22:18:35.446538 node.go:471 weedfs:datacenter1:rack2:node-b adds child --- PASS: TestPlanRackAwareGrowth_Issue9832 (0.00s) === RUN TestPlanRackAwareGrowth_DefaultStepNotMultiplied I0824 22:18:35.447133 node.go:471 weedfs adds child datacenter1 I0824 22:18:35.447193 node.go:471 weedfs:datacenter1 adds child rack1 I0824 22:18:35.447238 node.go:471 weedfs:datacenter1:rack1 adds child node-a I0824 22:18:35.447334 node.go:471 weedfs:datacenter1:rack1:node-a adds child I0824 22:18:35.447421 node.go:471 weedfs:datacenter1 adds child rack2 I0824 22:18:35.447490 node.go:471 weedfs:datacenter1:rack2 adds child node-b I0824 22:18:35.447574 volume_layout.go:815 Volume 12 becomes unwritable I0824 22:18:35.447608 node.go:471 weedfs:datacenter1:rack2:node-b adds child --- PASS: TestPlanRackAwareGrowth_DefaultStepNotMultiplied (0.00s) === RUN TestPlanRackAwareGrowth_NotCrowdedNoGrowth I0824 22:18:35.448853 node.go:471 weedfs adds child datacenter1 I0824 22:18:35.448914 node.go:471 weedfs:datacenter1 adds child rack2 I0824 22:18:35.449009 node.go:471 weedfs:datacenter1:rack2 adds child node-b I0824 22:18:35.449076 node.go:471 weedfs:datacenter1:rack2:node-b adds child I0824 22:18:35.449150 node.go:471 weedfs:datacenter1 adds child rack1 I0824 22:18:35.449188 node.go:471 weedfs:datacenter1:rack1 adds child node-a I0824 22:18:35.449223 volume_layout.go:815 Volume 12 becomes unwritable I0824 22:18:35.449949 node.go:471 weedfs:datacenter1:rack1:node-a adds child --- PASS: TestPlanRackAwareGrowth_NotCrowdedNoGrowth (0.00s) === RUN TestPlanRackAwareGrowth_PerRackForNonRackSpanning I0824 22:18:35.450488 node.go:471 weedfs adds child datacenter1 I0824 22:18:35.450615 node.go:471 weedfs:datacenter1 adds child rack1 I0824 22:18:35.450684 node.go:471 weedfs:datacenter1:rack1 adds child node-a I0824 22:18:35.450758 node.go:471 weedfs:datacenter1:rack1:node-a adds child I0824 22:18:35.450823 node.go:471 weedfs:datacenter1 adds child rack2 I0824 22:18:35.450857 node.go:471 weedfs:datacenter1:rack2 adds child node-b I0824 22:18:35.450884 node.go:471 weedfs:datacenter1:rack2:node-b adds child --- PASS: TestPlanRackAwareGrowth_PerRackForNonRackSpanning (0.00s) === RUN TestPlanRackAwareGrowth_EvenDistributionAcrossUnevenDCs I0824 22:18:35.451559 node.go:471 weedfs adds child dc1 I0824 22:18:35.451656 node.go:471 weedfs:dc1 adds child rack1 I0824 22:18:35.451696 node.go:471 weedfs:dc1:rack1 adds child node-a I0824 22:18:35.451755 node.go:471 weedfs:dc1:rack1:node-a adds child I0824 22:18:35.451842 node.go:471 weedfs:dc1 adds child rack2 I0824 22:18:35.451893 node.go:471 weedfs:dc1:rack2 adds child node-b I0824 22:18:35.451931 node.go:471 weedfs:dc1:rack2:node-b adds child I0824 22:18:35.451991 node.go:471 weedfs adds child dc2 I0824 22:18:35.452025 node.go:471 weedfs:dc2 adds child rack3 I0824 22:18:35.452061 node.go:471 weedfs:dc2:rack3 adds child node-c I0824 22:18:35.452102 node.go:471 weedfs:dc2:rack3:node-c adds child I0824 22:18:35.452193 node.go:471 weedfs:dc2 adds child rack4 I0824 22:18:35.452308 node.go:471 weedfs:dc2:rack4 adds child node-d I0824 22:18:35.452367 node.go:471 weedfs:dc2:rack4:node-d adds child I0824 22:18:35.452423 node.go:471 weedfs:dc2 adds child rack5 I0824 22:18:35.452454 node.go:471 weedfs:dc2:rack5 adds child node-e I0824 22:18:35.452491 node.go:471 weedfs:dc2:rack5:node-e adds child --- PASS: TestPlanRackAwareGrowth_EvenDistributionAcrossUnevenDCs (0.00s) === RUN TestShouldGrowVolumes_UnwritableCrowdedVolumes I0824 22:18:35.453593 volume_layout.go:959 Volume 1 becomes crowded I0824 22:18:35.453640 volume_layout.go:815 Volume 1 becomes unwritable I0824 22:18:35.453663 volume_layout.go:944 Volume 1 reaches full capacity. I0824 22:18:35.453695 volume_layout.go:959 Volume 2 becomes crowded --- PASS: TestShouldGrowVolumes_UnwritableCrowdedVolumes (0.00s) === RUN TestUpdateOversizedStateKeepsOversizedVolumeUnwritable I0824 22:18:35.454245 node.go:471 weedfs adds child dc1 I0824 22:18:35.454298 node.go:471 weedfs:dc1 adds child rack1 I0824 22:18:35.454341 node.go:471 weedfs:dc1:rack1 adds child server1 I0824 22:18:35.454402 node.go:471 weedfs:dc1:rack1:server1 adds child I0824 22:18:35.454477 volume_layout.go:815 Volume 1 becomes unwritable I0824 22:18:35.454515 volume_layout.go:391 Volume 1 reaches full capacity (effective=13000, limit=10000). I0824 22:18:35.454555 volume_layout.go:959 Volume 1 becomes crowded --- PASS: TestUpdateOversizedStateKeepsOversizedVolumeUnwritable (0.00s) === RUN TestUpdateOversizedStateClearsMarkWhenVolumeShrinks I0824 22:18:35.455012 node.go:471 weedfs adds child dc1 I0824 22:18:35.455114 node.go:471 weedfs:dc1 adds child rack1 I0824 22:18:35.455882 node.go:471 weedfs:dc1:rack1 adds child server1 I0824 22:18:35.455999 node.go:471 weedfs:dc1:rack1:server1 adds child --- PASS: TestUpdateOversizedStateClearsMarkWhenVolumeShrinks (0.00s) === RUN TestEnsureCorrectWritablesHonorsRecoveryCooldown I0824 22:18:35.456490 node.go:471 weedfs adds child dc1 I0824 22:18:35.456626 node.go:471 weedfs:dc1 adds child rack1 I0824 22:18:35.456729 node.go:471 weedfs:dc1:rack1 adds child server1 I0824 22:18:35.456789 node.go:471 weedfs:dc1:rack1:server1 adds child I0824 22:18:35.456853 volume_layout.go:815 Volume 1 becomes unwritable I0824 22:18:35.456883 volume_layout.go:391 Volume 1 reaches full capacity (effective=13000, limit=10000). I0824 22:18:35.456935 volume_layout.go:268 Volume 1 recovered to writable (effective=6250, reported=4000, limit=10000). --- PASS: TestEnsureCorrectWritablesHonorsRecoveryCooldown (0.00s) === RUN TestEnsureCorrectWritablesDoesNotRestoreVolumeStillAtLimit I0824 22:18:35.457269 node.go:471 weedfs adds child dc1 I0824 22:18:35.457347 node.go:471 weedfs:dc1 adds child rack1 I0824 22:18:35.457406 node.go:471 weedfs:dc1:rack1 adds child server1 I0824 22:18:35.458457 node.go:471 weedfs:dc1:rack1:server1 adds child I0824 22:18:35.458584 volume_layout.go:815 Volume 1 becomes unwritable I0824 22:18:35.458662 volume_layout.go:391 Volume 1 reaches full capacity (effective=13000, limit=10000). I0824 22:18:35.458703 volume_layout.go:959 Volume 1 becomes crowded --- PASS: TestEnsureCorrectWritablesDoesNotRestoreVolumeStillAtLimit (0.00s) === RUN TestEnsureCorrectWritablesRestoresCrowdedVolumeAfterReplicaReturns I0824 22:18:35.459152 node.go:471 weedfs adds child dc1 I0824 22:18:35.459246 node.go:471 weedfs:dc1 adds child rack1 I0824 22:18:35.459295 node.go:471 weedfs:dc1:rack1 adds child server1 I0824 22:18:35.459357 node.go:471 weedfs:dc1:rack1:server1 adds child I0824 22:18:35.459471 node.go:471 weedfs:dc1:rack1 adds child server2 I0824 22:18:35.459592 volume_layout.go:815 Volume 1 becomes unwritable I0824 22:18:35.459625 node.go:471 weedfs:dc1:rack1:server2 adds child I0824 22:18:35.459691 volume_layout.go:959 Volume 1 becomes crowded I0824 22:18:35.459752 volume_layout.go:815 Volume 1 becomes unwritable I0824 22:18:35.459782 volume_layout.go:335 volume 1 does not have enough copies --- PASS: TestEnsureCorrectWritablesRestoresCrowdedVolumeAfterReplicaReturns (0.00s) === RUN TestIncrementalRegistrationKeepsOversizedMark I0824 22:18:35.459998 node.go:471 weedfs adds child dc1 I0824 22:18:35.460086 node.go:471 weedfs:dc1 adds child rack1 I0824 22:18:35.460188 node.go:471 weedfs:dc1:rack1 adds child 127.0.0.1:34534 I0824 22:18:35.460235 node.go:471 weedfs:dc1:rack1:127.0.0.1:34534 adds child I0824 22:18:35.460329 volume_layout.go:959 Volume 1 becomes crowded --- PASS: TestIncrementalRegistrationKeepsOversizedMark (0.00s) === RUN TestPickForWriteWeightedDistribution I0824 22:18:35.460853 node.go:471 weedfs adds child dc1 I0824 22:18:35.460946 node.go:471 weedfs:dc1 adds child rack1 I0824 22:18:35.460991 node.go:471 weedfs:dc1:rack1 adds child server1 I0824 22:18:35.461045 node.go:471 weedfs:dc1:rack1:server1 adds child --- PASS: TestPickForWriteWeightedDistribution (0.07s) === RUN TestPickForWriteWithPendingSize I0824 22:18:35.533991 node.go:471 weedfs adds child dc1 I0824 22:18:35.534067 node.go:471 weedfs:dc1 adds child rack1 I0824 22:18:35.534111 node.go:471 weedfs:dc1:rack1 adds child server1 I0824 22:18:35.534196 node.go:471 weedfs:dc1:rack1:server1 adds child --- PASS: TestPickForWriteWithPendingSize (0.01s) === RUN TestPickForWriteSingleWritable I0824 22:18:35.545532 node.go:471 weedfs adds child dc1 I0824 22:18:35.545625 node.go:471 weedfs:dc1 adds child rack1 I0824 22:18:35.545692 node.go:471 weedfs:dc1:rack1 adds child server1 I0824 22:18:35.545763 node.go:471 weedfs:dc1:rack1:server1 adds child --- PASS: TestPickForWriteSingleWritable (0.00s) === RUN TestPickForWriteAllNearFull I0824 22:18:35.546252 node.go:471 weedfs adds child dc1 I0824 22:18:35.546296 node.go:471 weedfs:dc1 adds child rack1 I0824 22:18:35.546340 node.go:471 weedfs:dc1:rack1 adds child server1 I0824 22:18:35.546392 node.go:471 weedfs:dc1:rack1:server1 adds child --- PASS: TestPickForWriteAllNearFull (0.00s) === RUN TestPickForWriteConstrainedWeighted I0824 22:18:35.547153 node.go:471 weedfs adds child dc1 I0824 22:18:35.547210 node.go:471 weedfs:dc1 adds child rack2 I0824 22:18:35.547277 node.go:471 weedfs:dc1:rack2 adds child server2 I0824 22:18:35.547374 node.go:471 weedfs:dc1:rack2:server2 adds child I0824 22:18:35.547470 node.go:471 weedfs:dc1 adds child rack1 I0824 22:18:35.547506 node.go:471 weedfs:dc1:rack1 adds child server1 I0824 22:18:35.547544 node.go:471 weedfs:dc1:rack1:server1 adds child --- PASS: TestPickForWriteConstrainedWeighted (0.03s) === RUN TestRecordAssignMarksCrowded I0824 22:18:35.578657 node.go:471 weedfs adds child dc1 I0824 22:18:35.578712 node.go:471 weedfs:dc1 adds child rack1 I0824 22:18:35.578788 node.go:471 weedfs:dc1:rack1 adds child server1 I0824 22:18:35.578857 node.go:471 weedfs:dc1:rack1:server1 adds child I0824 22:18:35.578942 volume_layout.go:959 Volume 1 becomes crowded --- PASS: TestRecordAssignMarksCrowded (0.00s) === RUN TestRecordAssignReachingCapacityRemovesFromWritable I0824 22:18:35.579268 node.go:471 weedfs adds child dc1 I0824 22:18:35.579307 node.go:471 weedfs:dc1 adds child rack1 I0824 22:18:35.579338 node.go:471 weedfs:dc1:rack1 adds child server1 I0824 22:18:35.579392 node.go:471 weedfs:dc1:rack1:server1 adds child I0824 22:18:35.579460 volume_layout.go:815 Volume 1 becomes unwritable I0824 22:18:35.579489 volume_layout.go:391 Volume 1 reaches full capacity (effective=10000, limit=10000). --- PASS: TestRecordAssignReachingCapacityRemovesFromWritable (0.00s) === RUN TestUpdateVolumeSizeRecoversEagerlyRemovedVolume I0824 22:18:35.579822 node.go:471 weedfs adds child dc1 I0824 22:18:35.579878 node.go:471 weedfs:dc1 adds child rack1 I0824 22:18:35.579931 node.go:471 weedfs:dc1:rack1 adds child server1 I0824 22:18:35.579978 node.go:471 weedfs:dc1:rack1:server1 adds child I0824 22:18:35.580039 volume_layout.go:815 Volume 1 becomes unwritable I0824 22:18:35.580063 volume_layout.go:391 Volume 1 reaches full capacity (effective=10000, limit=10000). I0824 22:18:35.580164 volume_layout.go:268 Volume 1 recovered to writable (effective=4375, reported=4000, limit=10000). --- PASS: TestUpdateVolumeSizeRecoversEagerlyRemovedVolume (0.00s) === RUN TestUpdateVolumeSizeNoRecoveryWhenDiskStillOversized I0824 22:18:35.580416 node.go:471 weedfs adds child dc1 I0824 22:18:35.580456 node.go:471 weedfs:dc1 adds child rack1 I0824 22:18:35.580491 node.go:471 weedfs:dc1:rack1 adds child server1 I0824 22:18:35.580535 node.go:471 weedfs:dc1:rack1:server1 adds child I0824 22:18:35.580603 volume_layout.go:815 Volume 1 becomes unwritable I0824 22:18:35.580627 volume_layout.go:391 Volume 1 reaches full capacity (effective=10000, limit=10000). I0824 22:18:35.580701 volume_layout.go:959 Volume 1 becomes crowded --- PASS: TestUpdateVolumeSizeNoRecoveryWhenDiskStillOversized (0.00s) === RUN TestHeartbeatDecaysPendingSize I0824 22:18:35.581056 node.go:471 weedfs adds child dc1 I0824 22:18:35.581098 node.go:471 weedfs:dc1 adds child rack1 I0824 22:18:35.581151 node.go:471 weedfs:dc1:rack1 adds child server1 I0824 22:18:35.581199 node.go:471 weedfs:dc1:rack1:server1 adds child --- PASS: TestHeartbeatDecaysPendingSize (0.01s) === RUN TestHeartbeatDecayDedupReplicas I0824 22:18:35.590740 node.go:471 weedfs adds child dc1 I0824 22:18:35.590853 node.go:471 weedfs:dc1 adds child rack1 I0824 22:18:35.590892 node.go:471 weedfs:dc1:rack1 adds child server1 I0824 22:18:35.590965 node.go:471 weedfs:dc1:rack1:server1 adds child I0824 22:18:35.591041 node.go:471 weedfs:dc1:rack1 adds child server2 I0824 22:18:35.591094 volume_layout.go:815 Volume 1 becomes unwritable I0824 22:18:35.591124 node.go:471 weedfs:dc1:rack1:server2 adds child --- PASS: TestHeartbeatDecayDedupReplicas (0.00s) === RUN TestUpdateVolumeSize_DecaysEvenWhenReportedSizeUnchanged I0824 22:18:35.591319 node.go:471 weedfs adds child dc1 I0824 22:18:35.591378 node.go:471 weedfs:dc1 adds child rack1 I0824 22:18:35.591430 node.go:471 weedfs:dc1:rack1 adds child server1 I0824 22:18:35.593761 node.go:471 weedfs:dc1:rack1:server1 adds child --- PASS: TestUpdateVolumeSize_DecaysEvenWhenReportedSizeUnchanged (0.00s) === RUN TestShouldGrowVolumesByDcAndRack_WithPendingSize I0824 22:18:35.599849 node.go:471 weedfs adds child dc1 I0824 22:18:35.599989 node.go:471 weedfs:dc1 adds child rack1 I0824 22:18:35.600089 node.go:471 weedfs:dc1:rack1 adds child server1 I0824 22:18:35.600228 node.go:471 weedfs:dc1:rack1:server1 adds child I0824 22:18:35.600349 volume_layout.go:959 Volume 1 becomes crowded --- PASS: TestShouldGrowVolumesByDcAndRack_WithPendingSize (0.01s) === RUN TestSizeTrackingSkipsReadOnlyVolumes I0824 22:18:35.600599 node.go:471 st adds child dc1 I0824 22:18:35.600707 node.go:471 st:dc1 adds child rack1 I0824 22:18:35.600773 node.go:471 st:dc1:rack1 adds child 127.0.0.1:8080 I0824 22:18:35.600821 node.go:471 st:dc1:rack1:127.0.0.1:8080 adds child --- PASS: TestSizeTrackingSkipsReadOnlyVolumes (0.00s) === RUN TestSizeTrackingReleasedWhenAVolumeGoesReadOnly I0824 22:18:35.602219 node.go:471 st adds child dc1 I0824 22:18:35.602266 node.go:471 st:dc1 adds child rack1 I0824 22:18:35.602327 node.go:471 st:dc1:rack1 adds child 127.0.0.1:8080 I0824 22:18:35.602402 node.go:471 st:dc1:rack1:127.0.0.1:8080 adds child I0824 22:18:35.603153 volume_layout.go:815 Volume 1 becomes unwritable I0824 22:18:35.603243 volume_layout.go:338 volume 1 is not fully writable --- PASS: TestSizeTrackingReleasedWhenAVolumeGoesReadOnly (0.00s) === RUN TestSizeTrackingReturnsWhenAVolumeBecomesWritable I0824 22:18:35.603799 node.go:471 st adds child dc1 I0824 22:18:35.603915 node.go:471 st:dc1 adds child rack1 I0824 22:18:35.604132 node.go:471 st:dc1:rack1 adds child 127.0.0.1:8080 I0824 22:18:35.604216 node.go:471 st:dc1:rack1:127.0.0.1:8080 adds child --- PASS: TestSizeTrackingReturnsWhenAVolumeBecomesWritable (0.00s) === RUN TestSizeTrackingSurvivesAFullVolume I0824 22:18:35.604960 node.go:471 st adds child dc1 I0824 22:18:35.605049 node.go:471 st:dc1 adds child rack1 I0824 22:18:35.605149 node.go:471 st:dc1:rack1 adds child 127.0.0.1:8080 I0824 22:18:35.605259 node.go:471 st:dc1:rack1:127.0.0.1:8080 adds child I0824 22:18:35.605385 volume_layout.go:815 Volume 1 becomes unwritable I0824 22:18:35.605427 volume_layout.go:391 Volume 1 reaches full capacity (effective=33768, limit=32768). --- PASS: TestSizeTrackingSurvivesAFullVolume (0.00s) === RUN TestSizeTrackingIgnoresReplicaReportOrder === RUN TestSizeTrackingIgnoresReplicaReportOrder/ReadOnlyFirst I0824 22:18:35.605868 node.go:471 st adds child dc1 I0824 22:18:35.605979 node.go:471 st:dc1 adds child rack1 I0824 22:18:35.606166 node.go:471 st:dc1:rack1 adds child a I0824 22:18:35.606257 node.go:471 st:dc1:rack1:a adds child I0824 22:18:35.606353 node.go:471 st:dc1:rack1 adds child b I0824 22:18:35.606486 node.go:471 st:dc1:rack1:b adds child === RUN TestSizeTrackingIgnoresReplicaReportOrder/WritableFirst I0824 22:18:35.607069 node.go:471 st adds child dc1 I0824 22:18:35.607150 node.go:471 st:dc1 adds child rack1 I0824 22:18:35.607227 node.go:471 st:dc1:rack1 adds child a I0824 22:18:35.607314 node.go:471 st:dc1:rack1:a adds child I0824 22:18:35.607447 node.go:471 st:dc1:rack1 adds child b I0824 22:18:35.607516 node.go:471 st:dc1:rack1:b adds child --- PASS: TestSizeTrackingIgnoresReplicaReportOrder (0.00s) --- PASS: TestSizeTrackingIgnoresReplicaReportOrder/ReadOnlyFirst (0.00s) --- PASS: TestSizeTrackingIgnoresReplicaReportOrder/WritableFirst (0.00s) === RUN TestCrowdedEntryReleasedWithSizeTracking I0824 22:18:35.607970 node.go:471 st adds child dc1 I0824 22:18:35.608066 node.go:471 st:dc1 adds child rack1 I0824 22:18:35.608173 node.go:471 st:dc1:rack1 adds child 127.0.0.1:8080 I0824 22:18:35.608309 node.go:471 st:dc1:rack1:127.0.0.1:8080 adds child I0824 22:18:35.608564 volume_layout.go:959 Volume 1 becomes crowded I0824 22:18:35.608655 volume_layout.go:951 Volume 1 becomes uncrowded I0824 22:18:35.608714 volume_layout.go:815 Volume 1 becomes unwritable I0824 22:18:35.608755 volume_layout.go:338 volume 1 is not fully writable --- PASS: TestCrowdedEntryReleasedWithSizeTracking (0.00s) === RUN TestLocationFlagsFollowTheirNode --- PASS: TestLocationFlagsFollowTheirNode (0.00s) === RUN TestLocationFlagsClearedWithTheirNode --- PASS: TestLocationFlagsClearedWithTheirNode (0.00s) === RUN TestLocationFlagsSurviveAReplacedNode --- PASS: TestLocationFlagsSurviveAReplacedNode (0.00s) === RUN TestLocationFlagsIgnoreUntrackableReplicaCounts --- PASS: TestLocationFlagsIgnoreUntrackableReplicaCounts (0.00s) === RUN TestLocationFlagsRebuiltByRefresh --- PASS: TestLocationFlagsRebuiltByRefresh (0.00s) === RUN TestStreamVolumesStaysInsideTheAnnouncedTopology I0824 22:18:35.610735 node.go:471 bounds adds child dc1 I0824 22:18:35.616161 node.go:471 bounds:dc1 adds child rack1 I0824 22:18:35.616262 node.go:471 bounds:dc1:rack1 adds child 10.0.0.1:8080 I0824 22:18:35.616373 node.go:471 bounds:dc1:rack1:10.0.0.1:8080 adds child I0824 22:18:35.616620 node.go:471 bounds:dc1:rack1 adds child 10.0.0.2:8080 I0824 22:18:35.616662 node.go:471 bounds:dc1:rack1:10.0.0.2:8080 adds child --- PASS: TestStreamVolumesStaysInsideTheAnnouncedTopology (0.01s) === RUN TestStreamedListingRebuildsToTheSameThing I0824 22:18:35.617083 node.go:471 stream adds child dc1 I0824 22:18:35.617139 node.go:471 stream:dc1 adds child rack1 I0824 22:18:35.617181 node.go:471 stream:dc1:rack1 adds child 10.0.1.1:8080 I0824 22:18:35.617246 node.go:471 stream:dc1:rack1:10.0.1.1:8080 adds child I0824 22:18:35.625547 node.go:471 stream:dc1:rack1 adds child 10.0.1.2:8080 I0824 22:18:35.625737 node.go:471 stream:dc1:rack1:10.0.1.2:8080 adds child I0824 22:18:35.632815 node.go:471 stream adds child dc2 I0824 22:18:35.632909 node.go:471 stream:dc2 adds child rack1 I0824 22:18:35.632979 node.go:471 stream:dc2:rack1 adds child 10.0.2.1:8080 I0824 22:18:35.633053 node.go:471 stream:dc2:rack1:10.0.2.1:8080 adds child I0824 22:18:35.640178 node.go:471 stream:dc2:rack1 adds child 10.0.2.2:8080 I0824 22:18:35.640336 node.go:471 stream:dc2:rack1:10.0.2.2:8080 adds child === RUN TestStreamedListingRebuildsToTheSameThing/batch=1 === RUN TestStreamedListingRebuildsToTheSameThing/batch=7 === RUN TestStreamedListingRebuildsToTheSameThing/batch=250 === RUN TestStreamedListingRebuildsToTheSameThing/batch=100000 --- PASS: TestStreamedListingRebuildsToTheSameThing (0.07s) --- PASS: TestStreamedListingRebuildsToTheSameThing/batch=1 (0.01s) --- PASS: TestStreamedListingRebuildsToTheSameThing/batch=7 (0.01s) --- PASS: TestStreamedListingRebuildsToTheSameThing/batch=250 (0.01s) --- PASS: TestStreamedListingRebuildsToTheSameThing/batch=100000 (0.01s) === RUN TestStreamedListingHonoursTheFilter I0824 22:18:35.689615 node.go:471 stream adds child dc1 I0824 22:18:35.689748 node.go:471 stream:dc1 adds child rack1 I0824 22:18:35.689814 node.go:471 stream:dc1:rack1 adds child 10.0.1.1:8080 I0824 22:18:35.689865 node.go:471 stream:dc1:rack1:10.0.1.1:8080 adds child I0824 22:18:35.696528 node.go:471 stream:dc1:rack1 adds child 10.0.1.2:8080 I0824 22:18:35.696645 node.go:471 stream:dc1:rack1:10.0.1.2:8080 adds child I0824 22:18:35.707980 node.go:471 stream adds child dc2 I0824 22:18:35.708183 node.go:471 stream:dc2 adds child rack1 I0824 22:18:35.708289 node.go:471 stream:dc2:rack1 adds child 10.0.2.1:8080 I0824 22:18:35.708412 node.go:471 stream:dc2:rack1:10.0.2.1:8080 adds child I0824 22:18:35.721786 node.go:471 stream:dc2:rack1 adds child 10.0.2.2:8080 I0824 22:18:35.722003 node.go:471 stream:dc2:rack1:10.0.2.2:8080 adds child --- PASS: TestStreamedListingHonoursTheFilter (0.04s) PASS ok github.com/seaweedfs/seaweedfs/weed/topology 26.281s === RUN TestVolumeDensity --- PASS: TestVolumeDensity (0.00s) === RUN TestDensityRatio --- PASS: TestDensityRatio (0.00s) === RUN TestDensityNextRatio --- PASS: TestDensityNextRatio (0.00s) === RUN TestDensityRatio_OverfullSortsFullest --- PASS: TestDensityRatio_OverfullSortsFullest (0.00s) === RUN TestDiskTooFullAfter === RUN TestDiskTooFullAfter/disabled_at_0 === RUN TestDiskTooFullAfter/disabled_at_100 === RUN TestDiskTooFullAfter/not_reported_(total_0) === RUN TestDiskTooFullAfter/below_mark === RUN TestDiskTooFullAfter/at_mark === RUN TestDiskTooFullAfter/above_mark === RUN TestDiskTooFullAfter/below_but_crosses_with_incoming === RUN TestDiskTooFullAfter/below_and_stays_below_with_incoming === RUN TestDiskTooFullAfter/free_exceeds_total_(defensive) --- PASS: TestDiskTooFullAfter (0.00s) --- PASS: TestDiskTooFullAfter/disabled_at_0 (0.00s) --- PASS: TestDiskTooFullAfter/disabled_at_100 (0.00s) --- PASS: TestDiskTooFullAfter/not_reported_(total_0) (0.00s) --- PASS: TestDiskTooFullAfter/below_mark (0.00s) --- PASS: TestDiskTooFullAfter/at_mark (0.00s) --- PASS: TestDiskTooFullAfter/above_mark (0.00s) --- PASS: TestDiskTooFullAfter/below_but_crosses_with_incoming (0.00s) --- PASS: TestDiskTooFullAfter/below_and_stays_below_with_incoming (0.00s) --- PASS: TestDiskTooFullAfter/free_exceeds_total_(defensive) (0.00s) === RUN TestIsGoodMove_OverReplicatedSourceKeepsOtherReplica --- PASS: TestIsGoodMove_OverReplicatedSourceKeepsOtherReplica (0.00s) PASS ok github.com/seaweedfs/seaweedfs/weed/topology/balancer 13.082s === RUN TestAsyncBatchWorkerProcessesInOrder --- PASS: TestAsyncBatchWorkerProcessesInOrder (0.00s) === RUN TestAsyncBatchWorkerDrainWaitsForEnqueued --- PASS: TestAsyncBatchWorkerDrainWaitsForEnqueued (0.00s) === RUN TestAsyncBatchWorkerEnqueueNeverBlocks --- PASS: TestAsyncBatchWorkerEnqueueNeverBlocks (0.02s) === RUN TestAsyncBatchWorkerDrainCompletesDuringShutdown --- PASS: TestAsyncBatchWorkerDrainCompletesDuringShutdown (0.11s) === RUN TestAsyncBatchWorkerShutdownDrainsThenStops --- PASS: TestAsyncBatchWorkerShutdownDrainsThenStops (0.00s) === RUN TestExtractBucketPath === RUN TestExtractBucketPath/child_paths_return_bucket === RUN TestExtractBucketPath/bucket_root_without_child_fails_when_required === RUN TestExtractBucketPath/bucket_root_allowed_when_not_required === RUN TestExtractBucketPath/path_outside_buckets_fails === RUN TestExtractBucketPath/trailing_slash_on_base_is_normalized --- PASS: TestExtractBucketPath (0.01s) --- PASS: TestExtractBucketPath/child_paths_return_bucket (0.00s) --- PASS: TestExtractBucketPath/bucket_root_without_child_fails_when_required (0.00s) --- PASS: TestExtractBucketPath/bucket_root_allowed_when_not_required (0.00s) --- PASS: TestExtractBucketPath/path_outside_buckets_fails (0.00s) --- PASS: TestExtractBucketPath/trailing_slash_on_base_is_normalized (0.00s) === RUN TestRandomUint64ReturnsFullRandomValue --- PASS: TestRandomUint64ReturnsFullRandomValue (0.00s) === RUN TestRandomUint64PanicsOnRandomReadError --- PASS: TestRandomUint64PanicsOnRandomReadError (0.09s) === RUN TestByteParsing --- PASS: TestByteParsing (0.00s) === RUN TestSameAsJavaImplementation Now we need to generate a 256-bit key for AES 256 GCM --- PASS: TestSameAsJavaImplementation (0.00s) === RUN TestExecuteParallelTaskGroups --- PASS: TestExecuteParallelTaskGroups (0.00s) === RUN TestExecuteParallelTaskGroupsStopsOnlyFailedGroup --- PASS: TestExecuteParallelTaskGroupsStopsOnlyFailedGroup (0.00s) === RUN TestDropOSPageCache --- PASS: TestDropOSPageCache (0.00s) === RUN TestToShortFileName --- PASS: TestToShortFileName (0.00s) === RUN TestResolvePath === RUN TestResolvePath/empty === RUN TestResolvePath/absolute === RUN TestResolvePath/relative === RUN TestResolvePath/tilde_mid_path_is_literal === RUN TestResolvePath/bare_tilde === RUN TestResolvePath/tilde_slash === RUN TestResolvePath/tilde_with_subpath === RUN TestResolvePath/tilde_with_current_user === RUN TestResolvePath/tilde_with_current_user_and_subpath === RUN TestResolvePath/tilde_unknown_user_falls_back_to_literal === RUN TestResolvePath/tilde_with_native_separator === RUN TestResolvePath/tilde_user_with_native_separator --- PASS: TestResolvePath (0.05s) --- PASS: TestResolvePath/empty (0.00s) --- PASS: TestResolvePath/absolute (0.00s) --- PASS: TestResolvePath/relative (0.00s) --- PASS: TestResolvePath/tilde_mid_path_is_literal (0.00s) --- PASS: TestResolvePath/bare_tilde (0.00s) --- PASS: TestResolvePath/tilde_slash (0.00s) --- PASS: TestResolvePath/tilde_with_subpath (0.00s) --- PASS: TestResolvePath/tilde_with_current_user (0.00s) --- PASS: TestResolvePath/tilde_with_current_user_and_subpath (0.00s) --- PASS: TestResolvePath/tilde_unknown_user_falls_back_to_literal (0.04s) --- PASS: TestResolvePath/tilde_with_native_separator (0.00s) --- PASS: TestResolvePath/tilde_user_with_native_separator (0.00s) === RUN TestResolveCommaSeparatedPaths === RUN TestResolveCommaSeparatedPaths/empty === RUN TestResolveCommaSeparatedPaths/single_absolute === RUN TestResolveCommaSeparatedPaths/single_tilde === RUN TestResolveCommaSeparatedPaths/two_absolutes === RUN TestResolveCommaSeparatedPaths/two_tildes === RUN TestResolveCommaSeparatedPaths/mixed === RUN TestResolveCommaSeparatedPaths/no_tilde_fast_path --- PASS: TestResolveCommaSeparatedPaths (0.00s) --- PASS: TestResolveCommaSeparatedPaths/empty (0.00s) --- PASS: TestResolveCommaSeparatedPaths/single_absolute (0.00s) --- PASS: TestResolveCommaSeparatedPaths/single_tilde (0.00s) --- PASS: TestResolveCommaSeparatedPaths/two_absolutes (0.00s) --- PASS: TestResolveCommaSeparatedPaths/two_tildes (0.00s) --- PASS: TestResolveCommaSeparatedPaths/mixed (0.00s) --- PASS: TestResolveCommaSeparatedPaths/no_tilde_fast_path (0.00s) === RUN TestSanitizeUTF8Name_ValidPassThrough --- PASS: TestSanitizeUTF8Name_ValidPassThrough (0.00s) === RUN TestSanitizeUTF8Name_InvalidBytes --- PASS: TestSanitizeUTF8Name_InvalidBytes (0.00s) === RUN TestFullPathSanitized_WholePath --- PASS: TestFullPathSanitized_WholePath (0.00s) === RUN TestFullPathDirAndName_OnlyNameSanitized --- PASS: TestFullPathDirAndName_OnlyNameSanitized (0.00s) === RUN TestHumanReadableIntsMax --- PASS: TestHumanReadableIntsMax (0.00s) === RUN TestHumanReadableInts --- PASS: TestHumanReadableInts (0.00s) === RUN TestOrderedLock ActiveLock 1 acquired lock 0 ActiveLock 1 released lock 0 ActiveLock 3 acquired lock 1 ActiveLock 3 released lock 1 ActiveLock 4 acquired lock 0 ActiveLock 5 acquired lock 0 ActiveLock 6 acquired lock 0 ActiveLock 7 acquired lock 0 ActiveLock 7 released lock 0 ActiveLock 6 released lock 0 ActiveLock 4 released lock 0 ActiveLock 5 released lock 0 ActiveLock 8 acquired lock 1 ActiveLock 8 released lock 1 ActiveLock 9 acquired lock 0 ActiveLock 10 acquired lock 0 ActiveLock 11 acquired lock 0 ActiveLock 12 acquired lock 0 ActiveLock 12 released lock 0 ActiveLock 10 released lock 0 ActiveLock 11 released lock 0 ActiveLock 9 released lock 0 ActiveLock 13 acquired lock 1 ActiveLock 13 released lock 1 ActiveLock 14 acquired lock 1 ActiveLock 14 released lock 1 ActiveLock 15 acquired lock 1 ActiveLock 15 released lock 1 ActiveLock 16 acquired lock 0 ActiveLock 17 acquired lock 0 ActiveLock 17 released lock 0 ActiveLock 18 acquired lock 0 ActiveLock 19 acquired lock 0 ActiveLock 19 released lock 0 ActiveLock 18 released lock 0 ActiveLock 16 released lock 0 ActiveLock 20 acquired lock 1 ActiveLock 20 released lock 1 ActiveLock 22 acquired lock 0 ActiveLock 22 released lock 0 ActiveLock 23 acquired lock 1 ActiveLock 23 released lock 1 ActiveLock 24 acquired lock 0 ActiveLock 21 acquired lock 0 ActiveLock 25 acquired lock 0 ActiveLock 21 released lock 0 ActiveLock 26 acquired lock 0 ActiveLock 28 acquired lock 0 ActiveLock 27 acquired lock 0 ActiveLock 29 acquired lock 0 ActiveLock 2 acquired lock 0 ActiveLock 29 released lock 0 ActiveLock 28 released lock 0 ActiveLock 2 released lock 0 ActiveLock 26 released lock 0 ActiveLock 27 released lock 0 ActiveLock 24 released lock 0 ActiveLock 25 released lock 0 ActiveLock 30 acquired lock 1 ActiveLock 30 released lock 1 ActiveLock 32 acquired lock 0 ActiveLock 31 acquired lock 0 ActiveLock 31 released lock 0 ActiveLock 32 released lock 0 ActiveLock 35 acquired lock 1 ActiveLock 35 released lock 1 ActiveLock 37 acquired lock 0 ActiveLock 37 released lock 0 ActiveLock 36 acquired lock 0 ActiveLock 38 acquired lock 0 ActiveLock 39 acquired lock 0 ActiveLock 41 acquired lock 0 ActiveLock 42 acquired lock 0 ActiveLock 33 acquired lock 0 ActiveLock 43 acquired lock 0 ActiveLock 44 acquired lock 0 ActiveLock 45 acquired lock 0 ActiveLock 46 acquired lock 0 ActiveLock 47 acquired lock 0 ActiveLock 41 released lock 0 ActiveLock 47 released lock 0 ActiveLock 33 released lock 0 ActiveLock 43 released lock 0 ActiveLock 45 released lock 0 ActiveLock 42 released lock 0 ActiveLock 44 released lock 0 ActiveLock 46 released lock 0 ActiveLock 38 released lock 0 ActiveLock 39 released lock 0 ActiveLock 36 released lock 0 ActiveLock 48 acquired lock 1 ActiveLock 48 released lock 1 ActiveLock 49 acquired lock 0 ActiveLock 50 acquired lock 0 ActiveLock 34 acquired lock 0 ActiveLock 40 acquired lock 0 ActiveLock 49 released lock 0 ActiveLock 40 released lock 0 ActiveLock 34 released lock 0 ActiveLock 50 released lock 0 --- PASS: TestOrderedLock (1.31s) === RUN TestShouldWaitForSharedLock --- PASS: TestShouldWaitForSharedLock (0.00s) === RUN TestShouldWaitForExclusiveLock --- PASS: TestShouldWaitForExclusiveLock (0.00s) === RUN TestParseMinFreeSpace --- PASS: TestParseMinFreeSpace (0.00s) === RUN TestSetOutboundLocalIP === RUN TestSetOutboundLocalIP/ipv4 === RUN TestSetOutboundLocalIP/ipv6 === RUN TestSetOutboundLocalIP/empty === RUN TestSetOutboundLocalIP/wildcard_v4 === RUN TestSetOutboundLocalIP/wildcard_v6 === RUN TestSetOutboundLocalIP/garbage --- PASS: TestSetOutboundLocalIP (0.01s) --- PASS: TestSetOutboundLocalIP/ipv4 (0.00s) --- PASS: TestSetOutboundLocalIP/ipv6 (0.00s) --- PASS: TestSetOutboundLocalIP/empty (0.00s) --- PASS: TestSetOutboundLocalIP/wildcard_v4 (0.00s) --- PASS: TestSetOutboundLocalIP/wildcard_v6 (0.00s) --- PASS: TestSetOutboundLocalIP/garbage (0.00s) === RUN TestSetOutboundLocalIPFirstWins --- PASS: TestSetOutboundLocalIPFirstWins (0.00s) === RUN TestOutboundLocalAddrForDial === RUN TestOutboundLocalAddrForDial/remote_tcp_binds === RUN TestOutboundLocalAddrForDial/tcp4_binds === RUN TestOutboundLocalAddrForDial/hostname_binds === RUN TestOutboundLocalAddrForDial/loopback_ip_skipped === RUN TestOutboundLocalAddrForDial/loopback_name_skipped === RUN TestOutboundLocalAddrForDial/ipv6_loopback_skipped === RUN TestOutboundLocalAddrForDial/ipv6_literal_target_skipped === RUN TestOutboundLocalAddrForDial/unix_network_skipped === RUN TestOutboundLocalAddrForDial/udp_network_skipped --- PASS: TestOutboundLocalAddrForDial (0.01s) --- PASS: TestOutboundLocalAddrForDial/remote_tcp_binds (0.00s) --- PASS: TestOutboundLocalAddrForDial/tcp4_binds (0.00s) --- PASS: TestOutboundLocalAddrForDial/hostname_binds (0.00s) --- PASS: TestOutboundLocalAddrForDial/loopback_ip_skipped (0.00s) --- PASS: TestOutboundLocalAddrForDial/loopback_name_skipped (0.00s) --- PASS: TestOutboundLocalAddrForDial/ipv6_loopback_skipped (0.00s) --- PASS: TestOutboundLocalAddrForDial/ipv6_literal_target_skipped (0.00s) --- PASS: TestOutboundLocalAddrForDial/unix_network_skipped (0.00s) --- PASS: TestOutboundLocalAddrForDial/udp_network_skipped (0.00s) === RUN TestOutboundLocalAddrFamilyMismatch --- PASS: TestOutboundLocalAddrFamilyMismatch (0.00s) === RUN TestParseHostPort === RUN TestParseHostPort/hostname === RUN TestParseHostPort/ipv4 === RUN TestParseHostPort/bracketed_ipv6 === RUN TestParseHostPort/missing_port === RUN TestParseHostPort/invalid_port --- PASS: TestParseHostPort (0.00s) --- PASS: TestParseHostPort/hostname (0.00s) --- PASS: TestParseHostPort/ipv4 (0.00s) --- PASS: TestParseHostPort/bracketed_ipv6 (0.00s) --- PASS: TestParseHostPort/missing_port (0.00s) --- PASS: TestParseHostPort/invalid_port (0.00s) === RUN TestPathPrefixFilter_Empty --- PASS: TestPathPrefixFilter_Empty (0.00s) === RUN TestPathPrefixFilter_ExcludeOnly --- PASS: TestPathPrefixFilter_ExcludeOnly (0.00s) === RUN TestPathPrefixFilter_IncludeOnly --- PASS: TestPathPrefixFilter_IncludeOnly (0.00s) === RUN TestPathPrefixFilter_DeeperPrefixWins --- PASS: TestPathPrefixFilter_DeeperPrefixWins (0.00s) === RUN TestPathPrefixFilter_DeeperExcludeWins --- PASS: TestPathPrefixFilter_DeeperExcludeWins (0.00s) === RUN TestPathPrefixFilter_MultipleOverlappingPrefixes --- PASS: TestPathPrefixFilter_MultipleOverlappingPrefixes (0.00s) === RUN TestPathPrefixFilter_InvalidPrefixes --- PASS: TestPathPrefixFilter_InvalidPrefixes (0.00s) === RUN TestPathPrefixFilter_TrailingSlashNormalization --- PASS: TestPathPrefixFilter_TrailingSlashNormalization (0.00s) === RUN TestPathPrefixFilter_BoundaryMatching --- PASS: TestPathPrefixFilter_BoundaryMatching (0.00s) === RUN TestNewQueue --- PASS: TestNewQueue (0.00s) === RUN TestEnqueueAndConsume 1 2 3 ----------------------- 4 5 6 7 ----------------------- --- PASS: TestEnqueueAndConsume (0.00s) === RUN TestIsTransientError --- PASS: TestIsTransientError (0.04s) === RUN TestIsTransientErrorMessage --- PASS: TestIsTransientErrorMessage (0.00s) === RUN TestRetryTransientError I0824 22:18:24.143157 retry.go:95 retry test: err: read: connection reset by peer I0824 22:18:25.154943 retry.go:88 retry test successfully --- PASS: TestRetryTransientError (1.01s) === RUN TestRetryUntil === RUN TestRetryUntil/SucceedsImmediately === RUN TestRetryUntil/SucceedsAfterRetry === RUN TestRetryUntil/FailsNonRetryable --- PASS: TestRetryUntil (2.52s) --- PASS: TestRetryUntil/SucceedsImmediately (0.00s) --- PASS: TestRetryUntil/SucceedsAfterRetry (2.52s) --- PASS: TestRetryUntil/FailsNonRetryable (0.00s) === RUN TestRetryWithBackoff === RUN TestRetryWithBackoff/SucceedsAfterRetries === RUN TestRetryWithBackoff/StopsOnNonRetryableError === RUN TestRetryWithBackoff/StopsOnContextCancel === RUN TestRetryWithBackoff/StopsOnMaxDuration I0824 22:18:35.208381 retry.go:193 retry test: giving up after 3s: unavailable --- PASS: TestRetryWithBackoff (7.53s) --- PASS: TestRetryWithBackoff/SucceedsAfterRetries (2.52s) --- PASS: TestRetryWithBackoff/StopsOnNonRetryableError (0.00s) --- PASS: TestRetryWithBackoff/StopsOnContextCancel (2.00s) --- PASS: TestRetryWithBackoff/StopsOnMaxDuration (3.00s) === RUN TestSingleFlightGroupDedup --- PASS: TestSingleFlightGroupDedup (0.00s) === RUN TestSingleFlightGroupDifferentKeys --- PASS: TestSingleFlightGroupDifferentKeys (0.00s) === RUN TestSingleFlightGroupErrorPropagation --- PASS: TestSingleFlightGroupErrorPropagation (0.00s) === RUN TestSingleFlightGroupFreshCallAfterCompletion --- PASS: TestSingleFlightGroupFreshCallAfterCompletion (0.00s) PASS ok github.com/seaweedfs/seaweedfs/weed/util 25.765s === RUN TestSyncPoolPutBuffer_DropsOversized --- PASS: TestSyncPoolPutBuffer_DropsOversized (0.42s) === RUN TestSyncPoolPutBuffer_KeepsRightSized --- PASS: TestSyncPoolPutBuffer_KeepsRightSized (0.08s) === RUN TestSyncPoolPutBuffer_NilSafe --- PASS: TestSyncPoolPutBuffer_NilSafe (0.00s) PASS ok github.com/seaweedfs/seaweedfs/weed/util/buffer_pool 1.745s === RUN TestJobQueue enqueued 5 items dequeue 1 dequeue 2 enqueue 6 enqueue 7 dequeue ... dequeued 3 dequeue ... dequeued 4 dequeue ... dequeued 5 dequeue ... dequeued 6 dequeue ... dequeued 7 enqueue 8 enqueue 9 enqueue 10 enqueue 11 enqueue 12 dequeued 8 dequeued 9 dequeued 10 dequeued 11 dequeued 12 --- PASS: TestJobQueue (0.00s) === RUN TestJobQueueClose dequeued 1 dequeued 2 dequeued 3 dequeued 4 dequeued 5 dequeued 6 dequeued 7 --- PASS: TestJobQueueClose (0.00s) PASS ok github.com/seaweedfs/seaweedfs/weed/util/buffered_queue 1.312s ? github.com/seaweedfs/seaweedfs/weed/util/buffered_writer [no test files] === RUN TestOnDisk chunk_cache_on_disk_test.go:76: cache miss for entry 2 (acceptable with size constraints) chunk_cache_on_disk_test.go:129: cache miss for entry 2 after restart (acceptable) --- PASS: TestOnDisk (1.37s) === RUN TestLoadOrCreateChunkCacheVolumeClosesFilesOnError --- PASS: TestLoadOrCreateChunkCacheVolumeClosesFilesOnError (0.01s) PASS ok github.com/seaweedfs/seaweedfs/weed/util/chunk_cache 18.804s ? github.com/seaweedfs/seaweedfs/weed/util/constants [no test files] ? github.com/seaweedfs/seaweedfs/weed/util/fla9 [no test files] ? github.com/seaweedfs/seaweedfs/weed/util/grace [no test files] === RUN TestAppendQueryParameter === RUN TestAppendQueryParameter/without_existing_query === RUN TestAppendQueryParameter/with_existing_query === RUN TestAppendQueryParameter/with_trailing_question_mark === RUN TestAppendQueryParameter/with_trailing_ampersand === RUN TestAppendQueryParameter/encodes_values === RUN TestAppendQueryParameter/preserves_fragment === RUN TestAppendQueryParameter/blank_url --- PASS: TestAppendQueryParameter (0.08s) --- PASS: TestAppendQueryParameter/without_existing_query (0.00s) --- PASS: TestAppendQueryParameter/with_existing_query (0.00s) --- PASS: TestAppendQueryParameter/with_trailing_question_mark (0.00s) --- PASS: TestAppendQueryParameter/with_trailing_ampersand (0.00s) --- PASS: TestAppendQueryParameter/encodes_values (0.00s) --- PASS: TestAppendQueryParameter/preserves_fragment (0.00s) --- PASS: TestAppendQueryParameter/blank_url (0.00s) === RUN TestReadUrlAsStreamReturnsGzipReaderError --- PASS: TestReadUrlAsStreamReturnsGzipReaderError (0.16s) === RUN TestDeleteReturnsInvalidRequestErrorBeforeAddingAuth --- PASS: TestDeleteReturnsInvalidRequestErrorBeforeAddingAuth (0.00s) === RUN TestDeleteTreatsNoContentAsSuccess --- PASS: TestDeleteTreatsNoContentAsSuccess (0.05s) === RUN TestDeleteProxiedReturnsInvalidRequestErrorBeforeAddingAuth --- PASS: TestDeleteProxiedReturnsInvalidRequestErrorBeforeAddingAuth (0.00s) === RUN TestRetriedFetchChunkDataRetriesFreshUrlsImmediately I0824 22:18:21.023962 http_global_client_util.go:703 read http://127.0.0.1:39913/3,abc failed, err: Get "http://127.0.0.1:39913/3,abc?readDeleted=true": dial tcp 127.0.0.1:39913: connect: connection refused I0824 22:18:21.032525 http_global_client_util.go:558 chunk 3,abc failed on every known location, retrying on 1 fresh ones --- PASS: TestRetriedFetchChunkDataRetriesFreshUrlsImmediately (0.03s) === RUN TestRetriedFetchChunkDataKeepsBackoffWhenLocationsAreUnchanged I0824 22:18:21.066725 http_global_client_util.go:703 read http://127.0.0.1:35815/3,abc failed, err: Get "http://127.0.0.1:35815/3,abc?readDeleted=true": dial tcp 127.0.0.1:35815: connect: connection refused I0824 22:18:21.067103 http_global_client_util.go:712 retry reading in 1s I0824 22:18:22.077735 http_global_client_util.go:703 read http://127.0.0.1:35815/3,abc failed, err: Get "http://127.0.0.1:35815/3,abc?readDeleted=true": dial tcp 127.0.0.1:35815: connect: connection refused I0824 22:18:22.078165 http_global_client_util.go:712 retry reading in 1.5s --- PASS: TestRetriedFetchChunkDataKeepsBackoffWhenLocationsAreUnchanged (2.00s) === RUN TestSameUrlsIgnoresOrder --- PASS: TestSameUrlsIgnoresOrder (0.00s) PASS ok github.com/seaweedfs/seaweedfs/weed/util/http 13.526s === RUN TestNewHttpClientSetsResponseTimeouts --- PASS: TestNewHttpClientSetsResponseTimeouts (0.06s) === RUN TestResponseHeaderTimeoutUnblocksUnresponsivePeer --- PASS: TestResponseHeaderTimeoutUnblocksUnresponsivePeer (0.31s) === RUN TestResponseHeaderTimeoutUnblocksUnresponsivePeerTLS --- PASS: TestResponseHeaderTimeoutUnblocksUnresponsivePeerTLS (0.43s) PASS ok github.com/seaweedfs/seaweedfs/weed/util/http/client 14.120s ? github.com/seaweedfs/seaweedfs/weed/util/httpdown [no test files] === RUN TestFlushBudgetBoundsQueuedBytes --- PASS: TestFlushBudgetBoundsQueuedBytes (0.11s) === RUN TestFlushBudgetAdmitsOversizedWindow --- PASS: TestFlushBudgetAdmitsOversizedWindow (0.06s) === RUN TestFlushBudgetAdmitsInSealOrder --- PASS: TestFlushBudgetAdmitsInSealOrder (0.30s) === RUN TestFlushBudgetCloseReleasesWaiters --- PASS: TestFlushBudgetCloseReleasesWaiters (0.06s) === RUN TestQueueFlushChargesTheSlabNotTheWindow --- PASS: TestQueueFlushChargesTheSlabNotTheWindow (0.14s) === RUN TestQueueFlushKeepsSealedWindowWhenQueueHasRoom --- PASS: TestQueueFlushKeepsSealedWindowWhenQueueHasRoom (0.11s) === RUN TestStalledFlushBlocksOversizedProducers --- PASS: TestStalledFlushBlocksOversizedProducers (2.13s) === RUN TestBlockedProducersDrainOnceFlushResumes --- PASS: TestBlockedProducersDrainOnceFlushResumes (0.60s) === RUN TestReadTsCorruptedBuffer log_buffer_corruption_test.go:46: Error message: corrupted log buffer at pos 0, size 10: bad field tag at 0 log_buffer_corruption_test.go:53: ✓ readTs correctly returned error for corrupted buffer: corrupted log buffer at pos 0, size 10: bad field tag at 0 --- PASS: TestReadTsCorruptedBuffer (0.01s) === RUN TestReadTsValidBuffer log_buffer_corruption_test.go:92: ✓ readTs correctly parsed valid buffer: size=15, ts=123456789 --- PASS: TestReadTsValidBuffer (0.14s) === RUN TestReadFromBufferCorruption E0824 22:18:27.132044 log_buffer.go:1197 ReadFromBuffer: buffer corruption at idx[0] pos 0: corrupted log buffer at pos 0, size 33: bad field tag at 0 log_buffer_corruption_test.go:134: ✓ ReadFromBuffer correctly detected corruption: log buffer is corrupted: corrupted log buffer at pos 0, size 33: bad field tag at 0 log_buffer_corruption_test.go:137: ReadFromBuffer result: buf=false, offset=-1, err=log buffer is corrupted: corrupted log buffer at pos 0, size 33: bad field tag at 0 --- PASS: TestReadFromBufferCorruption (0.11s) === RUN TestLocateByTsCorruption log_buffer_corruption_test.go:163: ✓ locateByTs correctly detected corruption: locateByTs: buffer corruption at pos 0: corrupted log buffer at pos 0, size 10: bad field tag at 0 --- PASS: TestLocateByTsCorruption (0.00s) === RUN TestErrorPropagationChain === RUN TestErrorPropagationChain/Corruption_in_readTs log_buffer_corruption_test.go:171: ✓ readTs error propagation tested === RUN TestErrorPropagationChain/Corruption_in_locateByTs log_buffer_corruption_test.go:176: ✓ locateByTs error propagation tested === RUN TestErrorPropagationChain/Corruption_in_ReadFromBuffer_binary_search log_buffer_corruption_test.go:181: ✓ ReadFromBuffer error propagation tested === NAME TestErrorPropagationChain log_buffer_corruption_test.go:184: ✓ Complete error propagation chain verified --- PASS: TestErrorPropagationChain (0.01s) --- PASS: TestErrorPropagationChain/Corruption_in_readTs (0.00s) --- PASS: TestErrorPropagationChain/Corruption_in_locateByTs (0.00s) --- PASS: TestErrorPropagationChain/Corruption_in_ReadFromBuffer_binary_search (0.00s) === RUN TestNoSilentCorruption === RUN TestNoSilentCorruption/Invalid_protobuf log_buffer_corruption_test.go:215: ✓ Correctly returned error instead of silent (0, 0): corrupted log buffer: entry size 167772160 at pos 0 exceeds buffer length 14 === RUN TestNoSilentCorruption/Truncated_data log_buffer_corruption_test.go:215: ✓ Correctly returned error instead of silent (0, 0): corrupted log buffer: entry size 1677721600 at pos 0 exceeds buffer length 7 --- PASS: TestNoSilentCorruption (0.00s) --- PASS: TestNoSilentCorruption/Invalid_protobuf (0.00s) --- PASS: TestNoSilentCorruption/Truncated_data (0.00s) === RUN TestEvictionWatermarkTracksSealedRing --- PASS: TestEvictionWatermarkTracksSealedRing (0.07s) === RUN TestGapResumeCursorReadsSingleEntrySealedWindow --- PASS: TestGapResumeCursorReadsSingleEntrySealedWindow (0.01s) === RUN TestGapResumeCursorReadsFromMemory --- PASS: TestGapResumeCursorReadsFromMemory (0.00s) === RUN TestFlushSubscriberContract --- PASS: TestFlushSubscriberContract (0.13s) === RUN TestEvictionGatedCursor --- PASS: TestEvictionGatedCursor (0.00s) === RUN TestMarkEvictedThroughGatesYoungRing --- PASS: TestMarkEvictedThroughGatesYoungRing (0.33s) === RUN TestEvictionOriginalWatermark --- PASS: TestEvictionOriginalWatermark (0.02s) === RUN TestOriginalWatermarkWindowAttribution --- PASS: TestOriginalWatermarkWindowAttribution (0.01s) === RUN TestFlushOffsetGap_ReproduceDataLoss log_buffer_flush_gap_test.go:69: Sending 100 messages... log_buffer_flush_gap_test.go:82: Forcing flush... log_buffer_flush_gap_test.go:33: FLUSH: minOffset=0 maxOffset=99 size=4434 bytes log_buffer_flush_gap_test.go:61: Parsed 100 messages from flush buffer log_buffer_flush_gap_test.go:33: FLUSH: minOffset=100 maxOffset=149 size=2352 bytes log_buffer_flush_gap_test.go:61: Parsed 50 messages from flush buffer log_buffer_flush_gap_test.go:117: BUFFER STATE AFTER FLUSH: log_buffer_flush_gap_test.go:118: bufferStartOffset: 150 log_buffer_flush_gap_test.go:119: currentOffset (HWM): 150 log_buffer_flush_gap_test.go:120: pos (bytes in buffer): 0 log_buffer_flush_gap_test.go:121: Messages sent: 150 (offsets 0-149) log_buffer_flush_gap_test.go:122: Messages flushed to disk: 150 (offsets 0-149) log_buffer_flush_gap_test.go:128: OFFSET CONTINUITY CHECK: log_buffer_flush_gap_test.go:129: Last flushed offset: 149 log_buffer_flush_gap_test.go:130: Buffer starts at: 150 log_buffer_flush_gap_test.go:131: Gap: 0 offsets log_buffer_flush_gap_test.go:143: PASS: No gap detected - offsets are continuous log_buffer_flush_gap_test.go:147: READABILITY CHECK: log_buffer_flush_gap_test.go:159: Offset 0: OK (buf=false, err=resumeFromDisk) log_buffer_flush_gap_test.go:159: Offset 10: OK (buf=true, err=) log_buffer_flush_gap_test.go:159: Offset 20: OK (buf=true, err=) log_buffer_flush_gap_test.go:159: Offset 30: OK (buf=true, err=) log_buffer_flush_gap_test.go:159: Offset 40: OK (buf=true, err=) log_buffer_flush_gap_test.go:159: Offset 50: OK (buf=true, err=) log_buffer_flush_gap_test.go:159: Offset 60: OK (buf=true, err=) log_buffer_flush_gap_test.go:159: Offset 70: OK (buf=true, err=) log_buffer_flush_gap_test.go:159: Offset 80: OK (buf=true, err=) log_buffer_flush_gap_test.go:159: Offset 90: OK (buf=true, err=) log_buffer_flush_gap_test.go:159: Offset 100: OK (buf=true, err=) log_buffer_flush_gap_test.go:159: Offset 110: OK (buf=true, err=) log_buffer_flush_gap_test.go:159: Offset 120: OK (buf=true, err=) log_buffer_flush_gap_test.go:159: Offset 130: OK (buf=true, err=) log_buffer_flush_gap_test.go:159: Offset 140: OK (buf=true, err=) log_buffer_flush_gap_test.go:177: MESSAGE ACCOUNTING: log_buffer_flush_gap_test.go:178: Expected: 150 messages log_buffer_flush_gap_test.go:179: Flushed to disk: 150 log_buffer_flush_gap_test.go:180: In memory buffer: 0 (offset range 150-149) log_buffer_flush_gap_test.go:181: Total accounted for: 150 log_buffer_flush_gap_test.go:182: Missing: 0 messages log_buffer_flush_gap_test.go:187: All messages accounted for --- PASS: TestFlushOffsetGap_ReproduceDataLoss (0.27s) === RUN TestFlushOffsetGap_CheckPrevBuffers log_buffer_flush_gap_test.go:211: Batch 0: log_buffer_flush_gap_test.go:203: FLUSH #1: minOffset=0 maxOffset=19 size=872 bytes log_buffer_flush_gap_test.go:242: Before flush: offset=20, bufferStartOffset=0 log_buffer_flush_gap_test.go:243: After flush: offset=20, bufferStartOffset=20, prevBuffers=4 log_buffer_flush_gap_test.go:249: prevBuffer[3]: offsets 0-19, size=872 bytes (NOT FLUSHED!) log_buffer_flush_gap_test.go:211: Batch 1: log_buffer_flush_gap_test.go:203: FLUSH #2: minOffset=20 maxOffset=39 size=874 bytes log_buffer_flush_gap_test.go:242: Before flush: offset=40, bufferStartOffset=20 log_buffer_flush_gap_test.go:243: After flush: offset=40, bufferStartOffset=40, prevBuffers=4 log_buffer_flush_gap_test.go:249: prevBuffer[2]: offsets 0-19, size=872 bytes (NOT FLUSHED!) log_buffer_flush_gap_test.go:249: prevBuffer[3]: offsets 20-39, size=874 bytes (NOT FLUSHED!) log_buffer_flush_gap_test.go:211: Batch 2: log_buffer_flush_gap_test.go:203: FLUSH #3: minOffset=40 maxOffset=59 size=879 bytes log_buffer_flush_gap_test.go:242: Before flush: offset=60, bufferStartOffset=40 log_buffer_flush_gap_test.go:243: After flush: offset=60, bufferStartOffset=60, prevBuffers=4 log_buffer_flush_gap_test.go:249: prevBuffer[1]: offsets 0-19, size=872 bytes (NOT FLUSHED!) log_buffer_flush_gap_test.go:249: prevBuffer[2]: offsets 20-39, size=874 bytes (NOT FLUSHED!) log_buffer_flush_gap_test.go:249: prevBuffer[3]: offsets 40-59, size=879 bytes (NOT FLUSHED!) log_buffer_flush_gap_test.go:211: Batch 3: log_buffer_flush_gap_test.go:203: FLUSH #4: minOffset=60 maxOffset=79 size=904 bytes log_buffer_flush_gap_test.go:242: Before flush: offset=80, bufferStartOffset=60 log_buffer_flush_gap_test.go:243: After flush: offset=80, bufferStartOffset=80, prevBuffers=4 log_buffer_flush_gap_test.go:249: prevBuffer[0]: offsets 0-19, size=872 bytes (NOT FLUSHED!) log_buffer_flush_gap_test.go:249: prevBuffer[1]: offsets 20-39, size=874 bytes (NOT FLUSHED!) log_buffer_flush_gap_test.go:249: prevBuffer[2]: offsets 40-59, size=879 bytes (NOT FLUSHED!) log_buffer_flush_gap_test.go:249: prevBuffer[3]: offsets 60-79, size=904 bytes (NOT FLUSHED!) log_buffer_flush_gap_test.go:211: Batch 4: log_buffer_flush_gap_test.go:203: FLUSH #5: minOffset=80 maxOffset=99 size=905 bytes log_buffer_flush_gap_test.go:242: Before flush: offset=100, bufferStartOffset=80 log_buffer_flush_gap_test.go:243: After flush: offset=100, bufferStartOffset=100, prevBuffers=4 log_buffer_flush_gap_test.go:249: prevBuffer[0]: offsets 20-39, size=874 bytes (NOT FLUSHED!) log_buffer_flush_gap_test.go:249: prevBuffer[1]: offsets 40-59, size=879 bytes (NOT FLUSHED!) log_buffer_flush_gap_test.go:249: prevBuffer[2]: offsets 60-79, size=904 bytes (NOT FLUSHED!) log_buffer_flush_gap_test.go:249: prevBuffer[3]: offsets 80-99, size=905 bytes (NOT FLUSHED!) --- PASS: TestFlushOffsetGap_CheckPrevBuffers (0.44s) === RUN TestFlushOffsetGap_ConcurrentWriteAndFlush log_buffer_flush_gap_test.go:273: FLUSH: offsets 0-52 (2317 bytes) log_buffer_flush_gap_test.go:273: FLUSH: offsets 53-100 (2161 bytes) log_buffer_flush_gap_test.go:273: FLUSH: offsets 101-150 (2353 bytes) log_buffer_flush_gap_test.go:273: FLUSH: offsets 151-199 (2316 bytes) log_buffer_flush_gap_test.go:335: FINAL STATE: log_buffer_flush_gap_test.go:336: Total messages sent: 200 (offsets 0-199) log_buffer_flush_gap_test.go:337: Flushed to disk: 200 log_buffer_flush_gap_test.go:338: In memory: 0 (offsets 200-199) log_buffer_flush_gap_test.go:339: Total accounted: 200 log_buffer_flush_gap_test.go:340: Missing: 0 --- PASS: TestFlushOffsetGap_ConcurrentWriteAndFlush (0.48s) === RUN TestFlushOffsetGap_ProductionScenario log_buffer_flush_gap_test.go:393: === ROUND 1: Adding messages 0-49 === log_buffer_flush_gap_test.go:412: Before flush: logBuffer.offset=50, bufferStartOffset=0, nextKafkaOffset=50 log_buffer_flush_gap_test.go:382: FLUSH: minOffset=0 maxOffset=49, parsed 50 messages log_buffer_flush_gap_test.go:424: After flush: logBuffer.offset=50, bufferStartOffset=50 log_buffer_flush_gap_test.go:428: === ROUND 2: Adding messages 50-99 === log_buffer_flush_gap_test.go:382: FLUSH: minOffset=50 maxOffset=99, parsed 50 messages log_buffer_flush_gap_test.go:447: === VERIFICATION === log_buffer_flush_gap_test.go:448: Expected Kafka offsets: 0-99 log_buffer_flush_gap_test.go:452: Flush #0: minOffset=0, maxOffset=49, messages=50 log_buffer_flush_gap_test.go:452: Flush #1: minOffset=50, maxOffset=99, messages=50 log_buffer_flush_gap_test.go:481: SUCCESS: All 100 Kafka offsets accounted for (0-99) log_buffer_flush_gap_test.go:490: Final buffer state: log_buffer_flush_gap_test.go:491: logBuffer.offset: 100 log_buffer_flush_gap_test.go:492: bufferStartOffset: 100 log_buffer_flush_gap_test.go:493: Expected (nextKafkaOffset): 100 --- PASS: TestFlushOffsetGap_ProductionScenario (0.29s) === RUN TestFlushOffsetGap_ConcurrentReadDuringFlush log_buffer_flush_gap_test.go:554: Adding 100 messages... log_buffer_flush_gap_test.go:568: Flushing... log_buffer_flush_gap_test.go:546: FLUSH: Stored 100 offsets to disk (minOffset=0, maxOffset=99) log_buffer_flush_gap_test.go:573: Reading messages from offset 0... E0824 22:18:29.654509 log_read_stateless.go:169 [StatelessRead] CASE 1: Historical data - offset 0 < bufferStart 100 log_buffer_flush_gap_test.go:576: Read result: messages=100, nextOffset=100, hwm=100, endOfPartition=true, err= log_buffer_flush_gap_test.go:600: All 100 offsets can be read after flush --- PASS: TestFlushOffsetGap_ConcurrentReadDuringFlush (0.37s) === RUN TestFlushOffsetGap_ForceFlushAdvancesBuffer log_buffer_flush_gap_test.go:622: === ROUND 0 === log_buffer_flush_gap_test.go:630: Before adding: offset=0, bufferStartOffset=0 log_buffer_flush_gap_test.go:649: After adding: offset=10, bufferStartOffset=0 log_buffer_flush_gap_test.go:652: Forcing flush... log_buffer_flush_gap_test.go:614: FLUSH: offsets 0-9 log_buffer_flush_gap_test.go:662: After flush: offset=10, bufferStartOffset=10 log_buffer_flush_gap_test.go:671: bufferStartOffset correctly advanced to 10 log_buffer_flush_gap_test.go:622: === ROUND 1 === log_buffer_flush_gap_test.go:630: Before adding: offset=10, bufferStartOffset=10 log_buffer_flush_gap_test.go:649: After adding: offset=20, bufferStartOffset=10 log_buffer_flush_gap_test.go:652: Forcing flush... log_buffer_flush_gap_test.go:614: FLUSH: offsets 10-19 log_buffer_flush_gap_test.go:662: After flush: offset=20, bufferStartOffset=20 log_buffer_flush_gap_test.go:671: bufferStartOffset correctly advanced to 20 log_buffer_flush_gap_test.go:622: === ROUND 2 === log_buffer_flush_gap_test.go:630: Before adding: offset=20, bufferStartOffset=20 log_buffer_flush_gap_test.go:649: After adding: offset=30, bufferStartOffset=20 log_buffer_flush_gap_test.go:652: Forcing flush... log_buffer_flush_gap_test.go:614: FLUSH: offsets 20-29 log_buffer_flush_gap_test.go:662: After flush: offset=30, bufferStartOffset=30 log_buffer_flush_gap_test.go:671: bufferStartOffset correctly advanced to 30 log_buffer_flush_gap_test.go:677: === FLUSHED RANGES === log_buffer_flush_gap_test.go:679: Flush #0: offsets 0-9 log_buffer_flush_gap_test.go:679: Flush #1: offsets 10-19 log_buffer_flush_gap_test.go:692: Continuous with previous flush log_buffer_flush_gap_test.go:679: Flush #2: offsets 20-29 log_buffer_flush_gap_test.go:692: Continuous with previous flush --- PASS: TestFlushOffsetGap_ForceFlushAdvancesBuffer (0.40s) === RUN TestOversizedEntryDoesNotInflateBufferForever --- PASS: TestOversizedEntryDoesNotInflateBufferForever (2.53s) === RUN TestOversizedEntryStillFlushes --- PASS: TestOversizedEntryStillFlushes (0.24s) === RUN TestBufferQueryability log_buffer_queryability_test.go:104: Buffer queryability test passed - data is immediately readable --- PASS: TestBufferQueryability (0.13s) === RUN TestMultipleEntriesQueryability log_buffer_queryability_test.go:168: Entry 1: Key=test-key-1, Data=test-data-1, Offset=1 log_buffer_queryability_test.go:168: Entry 2: Key=test-key-2, Data=test-data-2, Offset=2 log_buffer_queryability_test.go:168: Entry 3: Key=test-key-3, Data=test-data-3, Offset=3 log_buffer_queryability_test.go:175: Multiple entries queryability test passed - found 3 entries --- PASS: TestMultipleEntriesQueryability (0.22s) === RUN TestSchemaRegistryScenario log_buffer_queryability_test.go:243: Schema registry scenario test passed - schema value preserved: 109 bytes --- PASS: TestSchemaRegistryScenario (0.05s) === RUN TestTimeBasedFirstReadBeforeEarliest --- PASS: TestTimeBasedFirstReadBeforeEarliest (0.04s) === RUN TestEarliestTimeExactRead --- PASS: TestEarliestTimeExactRead (0.02s) === RUN TestNewLogBufferFirstBuffer processed all messages E0824 22:18:33.716153 log_read.go:305 LoopProcessLogData: test process log entry 1 key:"" ts_ns:1787609913715716304 offset:4999 size:1024: EOF before flush: sent 5000 received 5000 lastProcessedTime {2026-08-24 22:18:33.715716304 +0000 UTC 5000 false} isDone true err: EOF --- PASS: TestNewLogBufferFirstBuffer (0.39s) === RUN TestReadFromBufferTimestampBased_AfterFlushReturnsNewerData --- PASS: TestReadFromBufferTimestampBased_AfterFlushReturnsNewerData (0.04s) === RUN TestReadFromBuffer_OldOffsetReturnsResumeFromDiskError === RUN TestReadFromBuffer_OldOffsetReturnsResumeFromDiskError/Request_offset_0_when_buffer_starts_at_4_(Schema_Registry_bug_scenario) log_buffer_test.go:211: ✓ When Schema Registry tries to read from offset 0, but data has been flushed to disk: correctly returned resumeFromDisk === RUN TestReadFromBuffer_OldOffsetReturnsResumeFromDiskError/Request_offset_before_buffer_start_with_empty_buffer log_buffer_test.go:211: ✓ Old offset with no data in memory should trigger disk read: correctly returned resumeFromDisk === RUN TestReadFromBuffer_OldOffsetReturnsResumeFromDiskError/Request_offset_before_buffer_start_with_data log_buffer_test.go:211: ✓ Old offset with current data in memory should still trigger disk read: correctly returned resumeFromDisk === RUN TestReadFromBuffer_OldOffsetReturnsResumeFromDiskError/Request_current_offset_(no_disk_read_needed) log_buffer_test.go:218: ✓ Current offset should return data from memory without error: correctly returned data without error === RUN TestReadFromBuffer_OldOffsetReturnsResumeFromDiskError/Request_offset_within_buffer_range log_buffer_test.go:218: ✓ Offset within buffer range should return data without error: correctly returned data without error --- PASS: TestReadFromBuffer_OldOffsetReturnsResumeFromDiskError (0.14s) --- PASS: TestReadFromBuffer_OldOffsetReturnsResumeFromDiskError/Request_offset_0_when_buffer_starts_at_4_(Schema_Registry_bug_scenario) (0.04s) --- PASS: TestReadFromBuffer_OldOffsetReturnsResumeFromDiskError/Request_offset_before_buffer_start_with_empty_buffer (0.02s) --- PASS: TestReadFromBuffer_OldOffsetReturnsResumeFromDiskError/Request_offset_before_buffer_start_with_data (0.02s) --- PASS: TestReadFromBuffer_OldOffsetReturnsResumeFromDiskError/Request_current_offset_(no_disk_read_needed) (0.02s) --- PASS: TestReadFromBuffer_OldOffsetReturnsResumeFromDiskError/Request_offset_within_buffer_range (0.05s) === RUN TestReadFromBuffer_OldOffsetWithNoPrevBuffers log_buffer_test.go:262: DEBUG: ReadFromBuffer returned: buf=false, batchIdx=-2, err=resumeFromDisk log_buffer_test.go:263: DEBUG: Buffer state: bufferStartOffset=10, offset=15, pos=100 log_buffer_test.go:265: DEBUG: Requested offset 5, prevBuffers[0] range: [20-25] log_buffer_test.go:274: ✓ BUG FIX VERIFIED: Correctly returns ResumeFromDiskError when requesting old offset 5 log_buffer_test.go:275: This allows the subscriber to read from disk instead of waiting forever --- PASS: TestReadFromBuffer_OldOffsetWithNoPrevBuffers (0.08s) === RUN TestReadFromBuffer_EmptyBufferAtCurrentOffset log_buffer_test.go:300: DEBUG: ReadFromBuffer returned: buf=false, batchIdx=-2, err=resumeFromDisk log_buffer_test.go:301: DEBUG: Buffer state: bufferStartOffset=0, offset=4, pos=0 log_buffer_test.go:311: ✓ BUG #2 FIX VERIFIED: Empty buffer correctly returns ResumeFromDiskError to check disk --- PASS: TestReadFromBuffer_EmptyBufferAtCurrentOffset (0.05s) === RUN TestReadFromBuffer_OffsetRanges === RUN TestReadFromBuffer_OffsetRanges/Before_buffer_start log_buffer_test.go:371: ✓ Offset 5 < bufferStartOffset 10 → read from disk === RUN TestReadFromBuffer_OffsetRanges/At_buffer_start log_buffer_test.go:379: ✓ Offset 10 == bufferStartOffset 10 → read from buffer === RUN TestReadFromBuffer_OffsetRanges/Within_buffer_range log_buffer_test.go:379: ✓ Offset 15 is within [10, 20] → read from buffer === RUN TestReadFromBuffer_OffsetRanges/At_buffer_end log_buffer_test.go:379: ✓ Offset 20 == offset 20 → read from buffer === RUN TestReadFromBuffer_OffsetRanges/After_buffer_end log_buffer_test.go:379: ✓ Offset 25 > offset 20 → future data, return nil without error --- PASS: TestReadFromBuffer_OffsetRanges (0.06s) --- PASS: TestReadFromBuffer_OffsetRanges/Before_buffer_start (0.00s) --- PASS: TestReadFromBuffer_OffsetRanges/At_buffer_start (0.00s) --- PASS: TestReadFromBuffer_OffsetRanges/Within_buffer_range (0.00s) --- PASS: TestReadFromBuffer_OffsetRanges/At_buffer_end (0.00s) --- PASS: TestReadFromBuffer_OffsetRanges/After_buffer_end (0.00s) === RUN TestReadFromBuffer_InitializedFromDisk log_buffer_test.go:410: After InitializeOffsetFromExistingData(highestOffset=3): log_buffer_test.go:411: offset=4 (should be 4), bufferStartOffset=4 (FIX: should be 4, not 0) log_buffer_test.go:429: After writing new message: log_buffer_test.go:430: bufferStartOffset=4, offset=5, pos=56 log_buffer_test.go:431: Requested offset 0, got: buf=false, batchIdx=-2, err=resumeFromDisk log_buffer_test.go:449: ✓ BUG #3 FIX VERIFIED: Reading old offset 0 correctly returns ResumeFromDiskError log_buffer_test.go:450: This ensures Schema Registry reads correct data from disk instead of getting new messages --- PASS: TestReadFromBuffer_InitializedFromDisk (0.00s) === RUN TestLoopProcessLogDataWithOffset_DiskReadRetry log_buffer_test.go:481: DISK READ #1: startOffset=0, dataReady=false log_buffer_test.go:484: → No data on disk yet log_buffer_test.go:545: ➕ Marking data as ready on disk flush from 2026-08-24 22:18:33.337328536 +0000 UTC m=+18.627632303 to 2026-08-24 22:18:33.715716304 +0000 UTC m=+19.006023331 5274870 bytes log_buffer_test.go:481: DISK READ #2: startOffset=0, dataReady=true log_buffer_test.go:516: ✉️ RECEIVED: offset=0 key=key-0 log_buffer_test.go:526: 📋 Reader finished: isDone=true, err= log_buffer_test.go:561: RESULTS: log_buffer_test.go:562: Disk reads: 2 log_buffer_test.go:563: Received messages: 1 log_buffer_test.go:564: Loop iterations: 1 log_buffer_test.go:576: ✓ SUCCESS: Message received after 2 disk read attempts --- PASS: TestLoopProcessLogDataWithOffset_DiskReadRetry (2.12s) === RUN TestShutdownStopsGoroutinesPromptly --- PASS: TestShutdownStopsGoroutinesPromptly (0.27s) === RUN TestConcurrentProducerConsumer log_read_integration_test.go:108: Consumer 0 consumed 500 messages log_read_integration_test.go:108: Consumer 1 consumed 500 messages --- PASS: TestConcurrentProducerConsumer (6.96s) === RUN TestBackwardSeeksWhileProducing log_read_integration_test.go:183: Seeking backward from 100 to 80 log_read_integration_test.go:183: Seeking backward from 100 to 80 log_read_integration_test.go:183: Seeking backward from 100 to 80 log_read_integration_test.go:183: Seeking backward from 100 to 80 log_read_integration_test.go:183: Seeking backward from 100 to 80 log_read_integration_test.go:183: Seeking backward from 100 to 80 log_read_integration_test.go:183: Seeking backward from 100 to 80 log_read_integration_test.go:183: Seeking backward from 100 to 80 log_read_integration_test.go:183: Seeking backward from 100 to 80 log_read_integration_test.go:183: Seeking backward from 100 to 80 log_read_integration_test.go:205: Total unique offsets read: 500 out of 500 --- PASS: TestBackwardSeeksWhileProducing (3.68s) === RUN TestHighConcurrencyReads --- PASS: TestHighConcurrencyReads (0.02s) === RUN TestRepeatedReadsAtSameOffset --- PASS: TestRepeatedReadsAtSameOffset (0.01s) === RUN TestEmptyPartitionPolling --- PASS: TestEmptyPartitionPolling (0.00s) === RUN TestReadMessagesAtOffset_EmptyBuffer --- PASS: TestReadMessagesAtOffset_EmptyBuffer (0.00s) === RUN TestReadMessagesAtOffset_SingleMessage --- PASS: TestReadMessagesAtOffset_SingleMessage (0.00s) === RUN TestReadMessagesAtOffset_MultipleMessages --- PASS: TestReadMessagesAtOffset_MultipleMessages (0.00s) === RUN TestReadMessagesAtOffset_StartFromMiddle --- PASS: TestReadMessagesAtOffset_StartFromMiddle (0.00s) === RUN TestReadMessagesAtOffset_MaxBytesLimit --- PASS: TestReadMessagesAtOffset_MaxBytesLimit (0.00s) === RUN TestReadMessagesAtOffset_ConcurrentReads --- PASS: TestReadMessagesAtOffset_ConcurrentReads (0.05s) === RUN TestReadMessagesAtOffset_FutureOffset --- PASS: TestReadMessagesAtOffset_FutureOffset (0.00s) === RUN TestWaitForDataWithTimeout_DataAvailable --- PASS: TestWaitForDataWithTimeout_DataAvailable (0.01s) === RUN TestWaitForDataWithTimeout_NoData log_read_stateless_test.go:314: Waited 55.43732ms for timeout --- PASS: TestWaitForDataWithTimeout_NoData (0.07s) === RUN TestWaitForDataWithTimeout_DataArrives --- PASS: TestWaitForDataWithTimeout_DataArrives (0.07s) === RUN TestGetHighWaterMark --- PASS: TestGetHighWaterMark (0.01s) === RUN TestGetLogStartOffset --- PASS: TestGetLogStartOffset (0.11s) === RUN TestLoopProcessLogDataWithOffset_ClientDisconnect log_read_test.go:53: Loop exited cleanly in 257.73447ms after client disconnect --- PASS: TestLoopProcessLogDataWithOffset_ClientDisconnect (0.32s) === RUN TestLoopProcessLogDataWithOffset_EmptyBuffer log_read_test.go:108: Loop exited cleanly in 772.18773ms after 4 iterations (no busy-waiting detected) --- PASS: TestLoopProcessLogDataWithOffset_EmptyBuffer (0.78s) === RUN TestLoopProcessLogDataWithOffset_NoDataResumeFromDisk log_read_test.go:154: Loop exited cleanly in 4.005644538s after 3 iterations (proper sleep detected) --- PASS: TestLoopProcessLogDataWithOffset_NoDataResumeFromDisk (4.01s) === RUN TestLoopFlush_NotifiesSubscribersAfterFlush log_read_test.go:205: flush notification delivered in 1.7883ms --- PASS: TestLoopFlush_NotifiesSubscribersAfterFlush (0.00s) === RUN TestLoopProcessLogDataWithOffset_WakesOnDataArrival I0824 22:18:52.674776 log_read.go:585 LoopProcessLogDataWithOffset: wake-test process log entry 1 log_read_test.go:274: reader processed the entry in 1.270719ms after AddToBuffer --- PASS: TestLoopProcessLogDataWithOffset_WakesOnDataArrival (0.06s) === RUN TestLoopProcessLogDataWithOffset_WakesOnShutdown log_read_test.go:335: reader exited in 238.28µs after ShutdownLogBuffer --- PASS: TestLoopProcessLogDataWithOffset_WakesOnShutdown (0.06s) === RUN TestLoopProcessLogDataWithOffset_WithData I0824 22:18:52.736404 log_read.go:585 LoopProcessLogDataWithOffset: test-client process log entry 1 log_read_test.go:401: Successfully processed 1 message(s) in 1.01542ms --- PASS: TestLoopProcessLogDataWithOffset_WithData (0.00s) === RUN TestLoopProcessLogDataWithOffset_ConcurrentDisconnect log_read_test.go:449: All 10 concurrent clients exited cleanly --- PASS: TestLoopProcessLogDataWithOffset_ConcurrentDisconnect (0.26s) === RUN TestLoopProcessLogDataWithOffset_StopTime log_read_test.go:488: Loop correctly exited for past stopTsNs in 255.88µs (waitForDataFn called 0 times) --- PASS: TestLoopProcessLogDataWithOffset_StopTime (0.01s) === RUN TestLoopProcessLogData_BoundedEmptyBufferTerminates --- PASS: TestLoopProcessLogData_BoundedEmptyBufferTerminates (0.01s) === RUN TestLoopProcessLogData_SlowConsumerFallsBehind --- PASS: TestLoopProcessLogData_SlowConsumerFallsBehind (0.03s) === RUN TestReadTsNsMatchesUnmarshal --- PASS: TestReadTsNsMatchesUnmarshal (0.00s) === RUN TestReadTsNsFuzz --- PASS: TestReadTsNsFuzz (0.07s) === RUN TestReadTsNsCorrupted --- PASS: TestReadTsNsCorrupted (0.00s) === RUN TestUnmarshalLogEntryAliasedMatchesUnmarshal --- PASS: TestUnmarshalLogEntryAliasedMatchesUnmarshal (0.14s) === RUN TestUnmarshalLogEntryAliasedAliases --- PASS: TestUnmarshalLogEntryAliasedAliases (0.00s) === RUN TestSealedReadsShareOneSnapshot --- PASS: TestSealedReadsShareOneSnapshot (0.01s) === RUN TestSharedSnapshotSurvivesRecycle --- PASS: TestSharedSnapshotSurvivesRecycle (0.01s) === RUN TestSharedSnapshotConcurrentIntegrity --- PASS: TestSharedSnapshotConcurrentIntegrity (2.24s) === RUN TestCurrentWindowPrefixSharing --- PASS: TestCurrentWindowPrefixSharing (0.00s) === RUN TestSealHandsOffCompleteSnapshot --- PASS: TestSealHandsOffCompleteSnapshot (0.00s) === RUN TestSharedSnapshotContentMatches --- PASS: TestSharedSnapshotContentMatches (0.01s) PASS ok github.com/seaweedfs/seaweedfs/weed/util/log_buffer 46.240s === RUN TestAllocateFree --- PASS: TestAllocateFree (0.01s) === RUN TestAllocateFreeEdgeCases --- PASS: TestAllocateFreeEdgeCases (0.01s) === RUN TestBitCount --- PASS: TestBitCount (0.00s) PASS ok github.com/seaweedfs/seaweedfs/weed/util/mem 4.013s === RUN TestQuoteDSNValue --- PASS: TestQuoteDSNValue (0.05s) === RUN TestBuildDSN_QuotesProblematicValues --- PASS: TestBuildDSN_QuotesProblematicValues (0.07s) PASS ok github.com/seaweedfs/seaweedfs/weed/util/pgxutil 4.790s === RUN TestNewUsesUppercaseHexFormat --- PASS: TestNewUsesUppercaseHexFormat (0.05s) === RUN TestNewIsUnique --- PASS: TestNewIsUnique (0.00s) === RUN TestEnsureIgnoresClientHeader --- PASS: TestEnsureIgnoresClientHeader (0.00s) === RUN TestEnsureReusesContextID --- PASS: TestEnsureReusesContextID (0.00s) PASS ok github.com/seaweedfs/seaweedfs/weed/util/request_id 3.637s === RUN TestReverseInsert --- PASS: TestReverseInsert (0.00s) === RUN TestDeleteRekeyedNode --- PASS: TestDeleteRekeyedNode (0.00s) === RUN TestInsertByKeyRecoversInconsistentEnds --- PASS: TestInsertByKeyRecoversInconsistentEnds (0.00s) === RUN TestInsertAndFind --- PASS: TestInsertAndFind (3.69s) === RUN TestDelete --- PASS: TestDelete (1.62s) === RUN TestNext --- PASS: TestNext (0.13s) === RUN TestPrev --- PASS: TestPrev (0.29s) === RUN TestFindGreaterOrEqual --- PASS: TestFindGreaterOrEqual (0.45s) === RUN TestChangeValue --- PASS: TestChangeValue (0.24s) PASS ok github.com/seaweedfs/seaweedfs/weed/util/skiplist 8.620s === RUN TestSplitStatements === RUN TestSplitStatements/Simple_single_statement === RUN TestSplitStatements/Multiple_statements === RUN TestSplitStatements/Semicolon_in_single_quotes === RUN TestSplitStatements/Semicolon_in_double_quotes === RUN TestSplitStatements/Escaped_quotes_in_strings === RUN TestSplitStatements/Escaped_quotes_in_identifiers === RUN TestSplitStatements/Single_line_comment === RUN TestSplitStatements/Single_line_comment_with_semicolon === RUN TestSplitStatements/Multi-line_comment === RUN TestSplitStatements/Multi-line_comment_with_semicolon === RUN TestSplitStatements/Complex_mixed_case === RUN TestSplitStatements/Empty_statements_filtered === RUN TestSplitStatements/Whitespace_handling === RUN TestSplitStatements/Single_statement_without_semicolon === RUN TestSplitStatements/Empty_query === RUN TestSplitStatements/Only_whitespace === RUN TestSplitStatements/Only_single_line_comment === RUN TestSplitStatements/Only_multi-line_comment --- PASS: TestSplitStatements (0.00s) --- PASS: TestSplitStatements/Simple_single_statement (0.00s) --- PASS: TestSplitStatements/Multiple_statements (0.00s) --- PASS: TestSplitStatements/Semicolon_in_single_quotes (0.00s) --- PASS: TestSplitStatements/Semicolon_in_double_quotes (0.00s) --- PASS: TestSplitStatements/Escaped_quotes_in_strings (0.00s) --- PASS: TestSplitStatements/Escaped_quotes_in_identifiers (0.00s) --- PASS: TestSplitStatements/Single_line_comment (0.00s) --- PASS: TestSplitStatements/Single_line_comment_with_semicolon (0.00s) --- PASS: TestSplitStatements/Multi-line_comment (0.00s) --- PASS: TestSplitStatements/Multi-line_comment_with_semicolon (0.00s) --- PASS: TestSplitStatements/Complex_mixed_case (0.00s) --- PASS: TestSplitStatements/Empty_statements_filtered (0.00s) --- PASS: TestSplitStatements/Whitespace_handling (0.00s) --- PASS: TestSplitStatements/Single_statement_without_semicolon (0.00s) --- PASS: TestSplitStatements/Empty_query (0.00s) --- PASS: TestSplitStatements/Only_whitespace (0.00s) --- PASS: TestSplitStatements/Only_single_line_comment (0.00s) --- PASS: TestSplitStatements/Only_multi-line_comment (0.00s) === RUN TestSplitStatements_EdgeCases === RUN TestSplitStatements_EdgeCases/Nested_comments_are_not_supported_but_handled_gracefully === RUN TestSplitStatements_EdgeCases/Unterminated_string_(malformed_SQL) === RUN TestSplitStatements_EdgeCases/Unterminated_comment_(malformed_SQL) === RUN TestSplitStatements_EdgeCases/Multiple_semicolons_in_quotes --- PASS: TestSplitStatements_EdgeCases (0.00s) --- PASS: TestSplitStatements_EdgeCases/Nested_comments_are_not_supported_but_handled_gracefully (0.00s) --- PASS: TestSplitStatements_EdgeCases/Unterminated_string_(malformed_SQL) (0.00s) --- PASS: TestSplitStatements_EdgeCases/Unterminated_comment_(malformed_SQL) (0.00s) --- PASS: TestSplitStatements_EdgeCases/Multiple_semicolons_in_quotes (0.00s) PASS ok github.com/seaweedfs/seaweedfs/weed/util/sqlutil 0.930s ? github.com/seaweedfs/seaweedfs/weed/util/version [no test files] === RUN TestMatchesWildcard === RUN TestMatchesWildcard/Exact_match === RUN TestMatchesWildcard/Single_wildcard === RUN TestMatchesWildcard/Empty_string_with_wildcard === RUN TestMatchesWildcard/Prefix_wildcard === RUN TestMatchesWildcard/Suffix_wildcard === RUN TestMatchesWildcard/Middle_wildcard === RUN TestMatchesWildcard/Multiple_wildcards === RUN TestMatchesWildcard/No_match === RUN TestMatchesWildcard/Single_question_mark === RUN TestMatchesWildcard/Multiple_question_marks === RUN TestMatchesWildcard/Question_mark_no_match === RUN TestMatchesWildcard/Mixed_wildcards === RUN TestMatchesWildcard/Empty_pattern === RUN TestMatchesWildcard/Empty_pattern_with_string === RUN TestMatchesWildcard/Pattern_with_string_empty === RUN TestMatchesWildcard/Pattern_with_regex_special_chars === RUN TestMatchesWildcard/Pattern_with_dots === RUN TestMatchesWildcard/Pattern_with_dots_and_wildcard --- PASS: TestMatchesWildcard (0.00s) --- PASS: TestMatchesWildcard/Exact_match (0.00s) --- PASS: TestMatchesWildcard/Single_wildcard (0.00s) --- PASS: TestMatchesWildcard/Empty_string_with_wildcard (0.00s) --- PASS: TestMatchesWildcard/Prefix_wildcard (0.00s) --- PASS: TestMatchesWildcard/Suffix_wildcard (0.00s) --- PASS: TestMatchesWildcard/Middle_wildcard (0.00s) --- PASS: TestMatchesWildcard/Multiple_wildcards (0.00s) --- PASS: TestMatchesWildcard/No_match (0.00s) --- PASS: TestMatchesWildcard/Single_question_mark (0.00s) --- PASS: TestMatchesWildcard/Multiple_question_marks (0.00s) --- PASS: TestMatchesWildcard/Question_mark_no_match (0.00s) --- PASS: TestMatchesWildcard/Mixed_wildcards (0.00s) --- PASS: TestMatchesWildcard/Empty_pattern (0.00s) --- PASS: TestMatchesWildcard/Empty_pattern_with_string (0.00s) --- PASS: TestMatchesWildcard/Pattern_with_string_empty (0.00s) --- PASS: TestMatchesWildcard/Pattern_with_regex_special_chars (0.00s) --- PASS: TestMatchesWildcard/Pattern_with_dots (0.00s) --- PASS: TestMatchesWildcard/Pattern_with_dots_and_wildcard (0.00s) === RUN TestWildcardMatcherMatch === RUN TestWildcardMatcherMatch/Simple_star === RUN TestWildcardMatcherMatch/Question_mark === RUN TestWildcardMatcherMatch/Extension_filter --- PASS: TestWildcardMatcherMatch (0.08s) --- PASS: TestWildcardMatcherMatch/Simple_star (0.00s) --- PASS: TestWildcardMatcherMatch/Question_mark (0.00s) --- PASS: TestWildcardMatcherMatch/Extension_filter (0.00s) === RUN TestCompileWildcardPattern === RUN TestCompileWildcardPattern/s3:Get* === RUN TestCompileWildcardPattern/s3:Get?bject === RUN TestCompileWildcardPattern/s3:*Object* --- PASS: TestCompileWildcardPattern (0.01s) --- PASS: TestCompileWildcardPattern/s3:Get* (0.00s) --- PASS: TestCompileWildcardPattern/s3:Get?bject (0.00s) --- PASS: TestCompileWildcardPattern/s3:*Object* (0.00s) === RUN TestFastMatchesWildcard === RUN TestFastMatchesWildcard/s3:Get*_s3:GetObject === RUN TestFastMatchesWildcard/s3:Put*_s3:GetObject === RUN TestFastMatchesWildcard/arn:aws:s3:::bucket/*_arn:aws:s3:::bucket/file.txt === RUN TestFastMatchesWildcard/user:admin-*_user:admin-john === RUN TestFastMatchesWildcard/user:admin-*_user:guest-john --- PASS: TestFastMatchesWildcard (0.00s) --- PASS: TestFastMatchesWildcard/s3:Get*_s3:GetObject (0.00s) --- PASS: TestFastMatchesWildcard/s3:Put*_s3:GetObject (0.00s) --- PASS: TestFastMatchesWildcard/arn:aws:s3:::bucket/*_arn:aws:s3:::bucket/file.txt (0.00s) --- PASS: TestFastMatchesWildcard/user:admin-*_user:admin-john (0.00s) --- PASS: TestFastMatchesWildcard/user:admin-*_user:guest-john (0.00s) === RUN TestWildcardMatcherCaching --- PASS: TestWildcardMatcherCaching (0.00s) === RUN TestWildcardMatcherCacheBounding --- PASS: TestWildcardMatcherCacheBounding (0.00s) === RUN TestWildcardMatcherCacheLRU --- PASS: TestWildcardMatcherCacheLRU (0.00s) === RUN TestWildcardMatcherCacheClear --- PASS: TestWildcardMatcherCacheClear (0.00s) PASS ok github.com/seaweedfs/seaweedfs/weed/util/wildcard 3.418s === RUN TestApplyDiscoveredFilersPrunesStaleAddress I0824 22:18:26.625756 filer_client.go:430 FilerClient: removed 1 filer(s) no longer in group '': [10.0.0.2:18888] I0824 22:18:26.639528 filer_client.go:433 FilerClient: discovered 1 new filer(s) in group '': [10.0.0.3:18888] --- PASS: TestApplyDiscoveredFilersPrunesStaleAddress (0.10s) === RUN TestApplyDiscoveredFilersKeepsIndexOnSurvivor I0824 22:18:26.641474 filer_client.go:430 FilerClient: removed 1 filer(s) no longer in group '': [10.0.0.1:18888] I0824 22:18:26.641815 filer_client.go:433 FilerClient: discovered 1 new filer(s) in group '': [10.0.0.3:18888] --- PASS: TestApplyDiscoveredFilersKeepsIndexOnSurvivor (0.00s) === RUN TestApplyDiscoveredFilersNoChangeIsNoop --- PASS: TestApplyDiscoveredFilersNoChangeIsNoop (0.00s) === RUN TestJitterBounds --- PASS: TestJitterBounds (0.00s) === RUN TestJitterZeroAndNegative --- PASS: TestJitterZeroAndNegative (0.00s) === RUN TestJitterTinyDuration --- PASS: TestJitterTinyDuration (0.00s) === RUN TestJitterDistribution --- PASS: TestJitterDistribution (0.00s) === RUN TestMovedVolumeKeepsItsLocation --- PASS: TestMovedVolumeKeepsItsLocation (0.00s) === RUN TestRemovedVolumeLosesItsLocation --- PASS: TestRemovedVolumeLosesItsLocation (0.00s) === RUN TestRemovalsAlongsideAMoveStillApply --- PASS: TestRemovalsAlongsideAMoveStillApply (0.00s) === RUN TestEcVolumeMovedKeepsItsLocation --- PASS: TestEcVolumeMovedKeepsItsLocation (0.00s) === RUN TestEcVolumeRemovedLosesItsLocation --- PASS: TestEcVolumeRemovedLosesItsLocation (0.00s) === RUN TestLookupVolumeIdsRetriesOnUnavailable --- PASS: TestLookupVolumeIdsRetriesOnUnavailable (5.12s) === RUN TestLookupVolumeIdsStopsOnContextCancel --- PASS: TestLookupVolumeIdsStopsOnContextCancel (2.01s) === RUN TestWaitUntilConnectedWithoutKeepConnected --- PASS: TestWaitUntilConnectedWithoutKeepConnected (0.11s) === RUN TestWaitUntilConnectedReturnsImmediatelyWhenConnected --- PASS: TestWaitUntilConnectedReturnsImmediatelyWhenConnected (0.00s) === RUN TestGetMasterRespectsContextCancellation --- PASS: TestGetMasterRespectsContextCancellation (0.11s) === RUN TestMasterClientFilerGroupLogging --- PASS: TestMasterClientFilerGroupLogging (0.00s) === RUN TestAddLocationReplacesEarlierGeneration --- PASS: TestAddLocationReplacesEarlierGeneration (0.00s) === RUN TestAddLocationMergesWithinGeneration --- PASS: TestAddLocationMergesWithinGeneration (0.00s) === RUN TestResetRetainsUntilWindowExpires --- PASS: TestResetRetainsUntilWindowExpires (0.00s) === RUN TestResetWindowRestartsOnRelearn --- PASS: TestResetWindowRestartsOnRelearn (0.00s) === RUN TestDeleteLocationDoesNotRefreshGeneration --- PASS: TestDeleteLocationDoesNotRefreshGeneration (0.00s) === RUN TestNewestGenerationWinsAcrossEcTransition === RUN TestNewestGenerationWinsAcrossEcTransition/encoded === RUN TestNewestGenerationWinsAcrossEcTransition/decoded === RUN TestNewestGenerationWinsAcrossEcTransition/same_generation_prefers_regular --- PASS: TestNewestGenerationWinsAcrossEcTransition (0.00s) --- PASS: TestNewestGenerationWinsAcrossEcTransition/encoded (0.00s) --- PASS: TestNewestGenerationWinsAcrossEcTransition/decoded (0.00s) --- PASS: TestNewestGenerationWinsAcrossEcTransition/same_generation_prefers_regular (0.00s) === RUN TestDeleteLastLocationDropsEntry --- PASS: TestDeleteLastLocationDropsEntry (0.00s) === RUN TestEcLocationsFollowGenerationRules --- PASS: TestEcLocationsFollowGenerationRules (0.00s) === RUN TestGetLocationsResultIsStable --- PASS: TestGetLocationsResultIsStable (0.00s) === RUN TestNewVidMapDefaultsRetention --- PASS: TestNewVidMapDefaultsRetention (0.00s) === RUN TestConcurrentResetAndUpdates --- PASS: TestConcurrentResetAndUpdates (0.01s) === RUN TestLookupFileId --- PASS: TestLookupFileId (0.02s) === RUN TestConcurrentGetLocations --- PASS: TestConcurrentGetLocations (0.06s) === RUN TestHasVolumeServer --- PASS: TestHasVolumeServer (0.00s) === RUN TestInvalidateCacheValidFileId --- PASS: TestInvalidateCacheValidFileId (0.00s) === RUN TestInvalidateCacheInvalidFileId === RUN TestInvalidateCacheInvalidFileId/empty_file_ID === RUN TestInvalidateCacheInvalidFileId/no_comma_separator === RUN TestInvalidateCacheInvalidFileId/non-numeric_vid === RUN TestInvalidateCacheInvalidFileId/negative_vid === RUN TestInvalidateCacheInvalidFileId/oversized_vid --- PASS: TestInvalidateCacheInvalidFileId (0.01s) --- PASS: TestInvalidateCacheInvalidFileId/empty_file_ID (0.00s) --- PASS: TestInvalidateCacheInvalidFileId/no_comma_separator (0.00s) --- PASS: TestInvalidateCacheInvalidFileId/non-numeric_vid (0.00s) --- PASS: TestInvalidateCacheInvalidFileId/negative_vid (0.00s) --- PASS: TestInvalidateCacheInvalidFileId/oversized_vid (0.00s) === RUN TestInvalidateCacheWithHistory --- PASS: TestInvalidateCacheWithHistory (0.00s) === RUN TestDeleteVidDropsRetainedGenerations --- PASS: TestDeleteVidDropsRetainedGenerations (0.00s) === RUN TestGetLocationsEmptyArrayNoFallback --- PASS: TestGetLocationsEmptyArrayNoFallback (0.00s) === RUN TestGetLocationsRetainedAcrossReset --- PASS: TestGetLocationsRetainedAcrossReset (0.00s) PASS ok github.com/seaweedfs/seaweedfs/weed/wdclient 23.503s ? github.com/seaweedfs/seaweedfs/weed/wdclient/exclusive_locks [no test files] === RUN TestManagerLoopSendersAbortAfterStop I0824 22:18:27.839097 worker.go:148 Generated new worker ID: w-arch-n-537d I0824 22:18:27.888339 worker.go:155 Saved worker ID to /tmp/TestManagerLoopSendersAbortAfterStop3030354470/001/worker.id I0824 22:18:27.891808 worker.go:517 Worker w-arch-n-537d stopped I0824 22:18:27.893111 worker.go:938 Worker w-arch-n-537d received task cancellation for task t1 I0824 22:18:27.893611 worker.go:517 Worker w-arch-n-537d stopped --- PASS: TestManagerLoopSendersAbortAfterStop (0.15s) PASS ok github.com/seaweedfs/seaweedfs/weed/worker 17.244s ? github.com/seaweedfs/seaweedfs/weed/worker/tasks [no test files] === RUN TestIntegration_DCFilterWithVolumeState I0824 22:18:28.838585 detection.go:289 BALANCE [hdd]: Created 13 task(s), cluster now balanced. Imbalance=16.0% (threshold=20.0%) detection_integration_test.go:147: DC+state filter: 13 tasks, effective=map[node-a:27 node-b:23] --- PASS: TestIntegration_DCFilterWithVolumeState (0.05s) === RUN TestIntegration_NodeFilterWithCollections I0824 22:18:28.855379 detection.go:289 BALANCE [hdd]: Created 18 task(s), cluster now balanced. Imbalance=13.3% (threshold=20.0%) detection_integration_test.go:206: Node filter with mixed collections: 18 tasks --- PASS: TestIntegration_NodeFilterWithCollections (0.01s) === RUN TestIntegration_ReplicaPlacementWithDCFilter I0824 22:18:28.861447 detection.go:289 BALANCE [hdd]: Created 11 task(s), cluster now balanced. Imbalance=17.1% (threshold=20.0%) detection_integration_test.go:287: Replica placement + DC filter: 11 tasks --- PASS: TestIntegration_ReplicaPlacementWithDCFilter (0.01s) === RUN TestIntegration_RackFilterWithReplicaPlacement I0824 22:18:28.869465 detection.go:289 BALANCE [hdd]: Created 11 task(s), cluster now balanced. Imbalance=17.1% (threshold=20.0%) detection_integration_test.go:361: Rack filter + replica placement: 11 tasks --- PASS: TestIntegration_RackFilterWithReplicaPlacement (0.01s) === RUN TestIntegration_AllFactors I0824 22:18:28.872979 detection.go:289 BALANCE [hdd]: Created 9 task(s), cluster now balanced. Imbalance=13.3% (threshold=20.0%) detection_integration_test.go:469: All factors combined: 9 tasks, effective=map[node-a:16 node-b:14] --- PASS: TestIntegration_AllFactors (0.00s) === RUN TestIntegration_FullVolumesOnlyBalancing I0824 22:18:28.894475 detection.go:289 BALANCE [hdd]: Created 11 task(s), cluster now balanced. Imbalance=17.1% (threshold=20.0%) detection_integration_test.go:512: Full-only balancing: 11 tasks --- PASS: TestIntegration_FullVolumesOnlyBalancing (0.02s) === RUN TestDetection_MixedDiskTypes I0824 22:18:28.912607 detection.go:286 BALANCE [hdd]: No tasks created - cluster well balanced. Imbalance=0.0% (threshold=20.0%). MaxUtil=10.0% on hdd-server-1, MinUtil=10.0% on hdd-server-1 I0824 22:18:28.912946 detection.go:286 BALANCE [ssd]: No tasks created - cluster well balanced. Imbalance=0.0% (threshold=20.0%). MaxUtil=1.0% on ssd-server-1, MinUtil=1.0% on ssd-server-1 --- PASS: TestDetection_MixedDiskTypes (0.02s) === RUN TestDetection_ImbalancedDiskType I0824 22:18:28.936206 detection.go:289 BALANCE [ssd]: Created 40 task(s), cluster now balanced. Imbalance=18.2% (threshold=20.0%) --- PASS: TestDetection_ImbalancedDiskType (0.02s) === RUN TestDetection_SkipsPhysicallyFullDestination I0824 22:18:28.954801 detection.go:347 BALANCE [hdd]: Created 7 task(s), no more beneficial moves available. After move, dest disk-empty util would be 0.8% vs source src util 0.0% --- PASS: TestDetection_SkipsPhysicallyFullDestination (0.02s) === RUN TestDetection_NoDestinationWhenAllDisksFull --- PASS: TestDetection_NoDestinationWhenAllDisksFull (0.00s) === RUN TestDetection_ProjectsVolumeOntoNearFullDestination I0824 22:18:28.956824 detection.go:286 BALANCE [hdd]: No tasks created - cluster well balanced. Imbalance=0.0% (threshold=20.0%). MaxUtil=80.0% on src, MinUtil=80.0% on src --- PASS: TestDetection_ProjectsVolumeOntoNearFullDestination (0.00s) === RUN TestDetection_SkipsRemoteTieredVolumes --- PASS: TestDetection_SkipsRemoteTieredVolumes (0.00s) === RUN TestDetection_RespectsMaxResults I0824 22:18:28.969484 detection.go:289 BALANCE [ssd]: Created 40 task(s), cluster now balanced. Imbalance=18.2% (threshold=20.0%) --- PASS: TestDetection_RespectsMaxResults (0.01s) === RUN TestDetection_ThreeServers_ConvergesToBalance I0824 22:18:28.974736 detection.go:289 BALANCE [hdd]: Created 23 task(s), cluster now balanced. Imbalance=17.6% (threshold=20.0%) --- PASS: TestDetection_ThreeServers_ConvergesToBalance (0.00s) === RUN TestDetection_SkipsPreExistingPendingTasks I0824 22:18:28.978050 detection.go:289 BALANCE [hdd]: Created 5 task(s), cluster now balanced. Imbalance=0.0% (threshold=20.0%) --- PASS: TestDetection_SkipsPreExistingPendingTasks (0.00s) === RUN TestDetection_NoDuplicateVolumesAcrossIterations I0824 22:18:28.982797 detection.go:289 BALANCE [ssd]: Created 18 task(s), cluster now balanced. Imbalance=13.3% (threshold=20.0%) --- PASS: TestDetection_NoDuplicateVolumesAcrossIterations (0.00s) === RUN TestDetection_ThreeServers_MaxServerShifts I0824 22:18:29.005513 detection.go:289 BALANCE [hdd]: Created 16 task(s), cluster now balanced. Imbalance=17.5% (threshold=20.0%) --- PASS: TestDetection_ThreeServers_MaxServerShifts (0.02s) === RUN TestDetection_FourServers_DestinationSpreading I0824 22:18:29.030807 detection.go:289 BALANCE [ssd]: Created 54 task(s), cluster now balanced. Imbalance=12.2% (threshold=20.0%) --- PASS: TestDetection_FourServers_DestinationSpreading (0.02s) === RUN TestDetection_ConvergenceVerification === RUN TestDetection_ConvergenceVerification/2-server-big-gap I0824 22:18:29.038495 detection.go:289 BALANCE [hdd]: Created 40 task(s), cluster now balanced. Imbalance=18.2% (threshold=20.0%) detection_test.go:915: 2-server-big-gap: 40 moves, effective=map[node-0:60 node-1:50], imbalance=18.2% === RUN TestDetection_ConvergenceVerification/3-server-staircase I0824 22:18:29.044861 detection.go:289 BALANCE [hdd]: Created 36 task(s), cluster now balanced. Imbalance=16.0% (threshold=20.0%) detection_test.go:915: 3-server-staircase: 36 moves, effective=map[node-0:54 node-1:50 node-2:46], imbalance=16.0% === RUN TestDetection_ConvergenceVerification/4-server-one-hot I0824 22:18:29.066663 detection.go:289 BALANCE [hdd]: Created 126 task(s), cluster now balanced. Imbalance=17.6% (threshold=20.0%) detection_test.go:915: 4-server-one-hot: 126 moves, effective=map[node-0:74 node-1:62 node-2:62 node-3:62], imbalance=18.5% === RUN TestDetection_ConvergenceVerification/3-server-tight-threshold I0824 22:18:29.072279 detection.go:289 BALANCE [hdd]: Created 10 task(s), cluster now balanced. Imbalance=0.0% (threshold=10.0%) detection_test.go:915: 3-server-tight-threshold: 10 moves, effective=map[node-0:20 node-1:20 node-2:20], imbalance=0.0% --- PASS: TestDetection_ConvergenceVerification (0.04s) --- PASS: TestDetection_ConvergenceVerification/2-server-big-gap (0.01s) --- PASS: TestDetection_ConvergenceVerification/3-server-staircase (0.01s) --- PASS: TestDetection_ConvergenceVerification/4-server-one-hot (0.02s) --- PASS: TestDetection_ConvergenceVerification/3-server-tight-threshold (0.00s) === RUN TestDetection_ExhaustedServerFallsThrough I0824 22:18:29.080849 detection.go:289 BALANCE [hdd]: Created 23 task(s), cluster now balanced. Imbalance=16.0% (threshold=20.0%) detection_test.go:987: Created 23 tasks from node-b after node-a exhausted --- PASS: TestDetection_ExhaustedServerFallsThrough (0.01s) === RUN TestDetection_HeterogeneousCapacity detection_test.go:1052: First balance task: move from server-3 (correct: highest utilization) --- PASS: TestDetection_HeterogeneousCapacity (0.02s) === RUN TestDetection_ZeroVolumeServerIncludedInBalance I0824 22:18:29.104136 detection.go:316 BALANCE [hdd]: Created 5 task(s), cluster as balanced as possible. Imbalance=40.0% (threshold=20.0%), max-min diff=1 detection_test.go:1134: Distribution 8/2/1/0 → map[node-a:3 node-b:3 node-c:3 node-d:2] after 5 moves (imbalance=36.4%) --- PASS: TestDetection_ZeroVolumeServerIncludedInBalance (0.00s) === RUN TestDetection_DataCenterFilter I0824 22:18:29.139884 detection.go:289 BALANCE [hdd]: Created 18 task(s), cluster now balanced. Imbalance=13.3% (threshold=20.0%) detection_test.go:1191: Created 18 tasks within dc1 scope --- PASS: TestDetection_DataCenterFilter (0.03s) === RUN TestDetection_NodeFilter I0824 22:18:29.144951 detection.go:289 BALANCE [hdd]: Created 18 task(s), cluster now balanced. Imbalance=13.3% (threshold=20.0%) detection_test.go:1244: Created 18 tasks within node-a,node-b scope --- PASS: TestDetection_NodeFilter (0.00s) === RUN TestDetection_HeterogeneousMax_NoOvershootNoOscillation I0824 22:18:29.146625 detection.go:344 BALANCE [hdd]: No tasks created - no beneficial move available. After move, dest node-b util would be 60.0% vs source node-a util 50.0% detection_test.go:1353: tasks=0, effective=map[node-a:11 node-b:5], final diff=0.050 (initial=0.050) --- PASS: TestDetection_HeterogeneousMax_NoOvershootNoOscillation (0.00s) === RUN TestDetection_RespectsClusterIdealUtilization I0824 22:18:29.148602 detection.go:289 BALANCE [hdd]: Created 3 task(s), cluster now balanced. Imbalance=11.8% (threshold=20.0%) detection_test.go:1411: tasks=3, effective=map[node-a:18 node-b:9 node-c:4], ideal=0.443 --- PASS: TestDetection_RespectsClusterIdealUtilization (0.00s) === RUN TestResolveBalanceDestination_UsesEffectiveCapacity --- PASS: TestResolveBalanceDestination_UsesEffectiveCapacity (0.00s) === RUN TestDecodeVolumeBalanceTaskParamsFromPayload --- PASS: TestDecodeVolumeBalanceTaskParamsFromPayload (0.03s) === RUN TestDecodeVolumeBalanceTaskParamsFallback --- PASS: TestDecodeVolumeBalanceTaskParamsFallback (0.00s) === RUN TestDeriveBalanceWorkerConfig --- PASS: TestDeriveBalanceWorkerConfig (0.00s) === RUN TestDeriveBalanceWorkerConfigBatchFields --- PASS: TestDeriveBalanceWorkerConfigBatchFields (0.00s) === RUN TestDeriveBalanceWorkerConfigBatchClamping --- PASS: TestDeriveBalanceWorkerConfigBatchClamping (0.00s) === RUN TestBuildBatchVolumeBalanceProposals_SingleBatch --- PASS: TestBuildBatchVolumeBalanceProposals_SingleBatch (0.02s) === RUN TestBuildBatchVolumeBalanceProposals_MultipleBatches --- PASS: TestBuildBatchVolumeBalanceProposals_MultipleBatches (0.00s) === RUN TestBuildBatchVolumeBalanceProposals_BatchSizeOne --- PASS: TestBuildBatchVolumeBalanceProposals_BatchSizeOne (0.02s) === RUN TestVolumeBalanceDescriptorHasBatchFields --- PASS: TestVolumeBalanceDescriptorHasBatchFields (0.00s) === RUN TestBuildVolumeBalanceProposal --- PASS: TestBuildVolumeBalanceProposal (0.00s) === RUN TestVolumeBalanceHandlerRejectsUnsupportedJobType --- PASS: TestVolumeBalanceHandlerRejectsUnsupportedJobType (0.00s) === RUN TestEmitVolumeBalanceDetectionDecisionTraceNoTasks --- PASS: TestEmitVolumeBalanceDetectionDecisionTraceNoTasks (0.00s) === RUN TestVolumeBalanceDescriptorOmitsExecutionTuningFields --- PASS: TestVolumeBalanceDescriptorOmitsExecutionTuningFields (0.00s) === RUN TestBuildMoveTaskParams --- PASS: TestBuildMoveTaskParams (0.00s) === RUN TestBuildMoveTaskParamsDefaultTimeout --- PASS: TestBuildMoveTaskParamsDefaultTimeout (0.00s) === RUN TestExecuteDispatchesBatchPath I0824 22:18:29.218819 plugin_handler.go:871 batch volume balance: 2 moves, max concurrent 2 W0824 22:18:29.224117 plugin_handler.go:797 volume balance: no master addresses in cluster context, skipping pre-move check for volume 2 I0824 22:18:29.228583 task.go:63 Starting balance task - moving volume W0824 22:18:29.228874 plugin_handler.go:797 volume balance: no master addresses in cluster context, skipping pre-move check for volume 1 I0824 22:18:29.229100 task.go:63 Starting balance task - moving volume I0824 22:18:29.230206 task.go:61 move stage: marking volume 2 readonly on s2:8080 I0824 22:18:29.230253 task.go:61 move stage: marking volume 1 readonly on s1:8080 W0824 22:18:29.231728 plugin_handler.go:984 batch balance move 1 failed: volume 2 s2:8080→t2:8080: move volume 2 from s2:8080 to t2:8080: mark volume 2 readonly on s2:8080: getOrCreateConnection s2:18080: fail to dial s2:18080: grpc: no transport security set (use grpc.WithTransportCredentials(insecure.NewCredentials()) explicitly or set credentials) W0824 22:18:29.232604 plugin_handler.go:984 batch balance move 0 failed: volume 1 s1:8080→t1:8080: move volume 1 from s1:8080 to t1:8080: mark volume 1 readonly on s1:8080: getOrCreateConnection s1:18080: fail to dial s1:18080: grpc: no transport security set (use grpc.WithTransportCredentials(insecure.NewCredentials()) explicitly or set credentials) --- PASS: TestExecuteDispatchesBatchPath (0.02s) === RUN TestExecuteSingleMovePathUnchanged W0824 22:18:29.236919 plugin_handler.go:797 volume balance: no master addresses in cluster context, skipping pre-move check for volume 99 I0824 22:18:29.237223 task.go:63 Starting balance task - moving volume I0824 22:18:29.237545 task.go:61 move stage: marking volume 99 readonly on src:8080 --- PASS: TestExecuteSingleMovePathUnchanged (0.00s) === RUN TestFilterMetricsByLocation --- PASS: TestFilterMetricsByLocation (0.00s) === RUN TestFilterMetricsByVolumeState === RUN TestFilterMetricsByVolumeState/ALL_returns_everything === RUN TestFilterMetricsByVolumeState/empty_string_returns_everything === RUN TestFilterMetricsByVolumeState/ACTIVE_keeps_FullnessRatio_below_1.01 === RUN TestFilterMetricsByVolumeState/FULL_keeps_FullnessRatio_at_or_above_1.01 === RUN TestFilterMetricsByVolumeState/unknown_value_returns_everything --- PASS: TestFilterMetricsByVolumeState (0.00s) --- PASS: TestFilterMetricsByVolumeState/ALL_returns_everything (0.00s) --- PASS: TestFilterMetricsByVolumeState/empty_string_returns_everything (0.00s) --- PASS: TestFilterMetricsByVolumeState/ACTIVE_keeps_FullnessRatio_below_1.01 (0.00s) --- PASS: TestFilterMetricsByVolumeState/FULL_keeps_FullnessRatio_at_or_above_1.01 (0.00s) --- PASS: TestFilterMetricsByVolumeState/unknown_value_returns_everything (0.00s) === RUN TestFilterMetricsByVolumeState_NilElement --- PASS: TestFilterMetricsByVolumeState_NilElement (0.00s) === RUN TestFilterMetricsByVolumeState_EmptyInput --- PASS: TestFilterMetricsByVolumeState_EmptyInput (0.00s) === RUN TestVolumeBalanceDescriptorHasVolumeStateField --- PASS: TestVolumeBalanceDescriptorHasVolumeStateField (0.00s) === RUN TestCheckMovePreconditions === RUN TestCheckMovePreconditions/valid_move === RUN TestCheckMovePreconditions/volume_left_the_source === RUN TestCheckMovePreconditions/volume_gone_entirely === RUN TestCheckMovePreconditions/target_already_has_a_replica === RUN TestCheckMovePreconditions/proposal_nodes_carry_grpc_suffix === RUN TestCheckMovePreconditions/grpc-suffixed_target_already_has_a_replica --- PASS: TestCheckMovePreconditions (0.00s) --- PASS: TestCheckMovePreconditions/valid_move (0.00s) --- PASS: TestCheckMovePreconditions/volume_left_the_source (0.00s) --- PASS: TestCheckMovePreconditions/volume_gone_entirely (0.00s) --- PASS: TestCheckMovePreconditions/target_already_has_a_replica (0.00s) --- PASS: TestCheckMovePreconditions/proposal_nodes_carry_grpc_suffix (0.00s) --- PASS: TestCheckMovePreconditions/grpc-suffixed_target_already_has_a_replica (0.00s) === RUN TestIsGoodMove_NoReplication --- PASS: TestIsGoodMove_NoReplication (0.00s) === RUN TestIsGoodMove_MachineAntiAffinity --- PASS: TestIsGoodMove_MachineAntiAffinity (0.00s) === RUN TestIsGoodMove_001_SameRack --- PASS: TestIsGoodMove_001_SameRack (0.00s) === RUN TestIsGoodMove_010_DiffRack --- PASS: TestIsGoodMove_010_DiffRack (0.00s) === RUN TestIsGoodMove_100_DiffDC --- PASS: TestIsGoodMove_100_DiffDC (0.00s) === RUN TestIsGoodMove_SameNode --- PASS: TestIsGoodMove_SameNode (0.00s) === RUN TestIsGoodMove_011_Composite --- PASS: TestIsGoodMove_011_Composite (0.00s) === RUN TestIsGoodMove_110_Composite --- PASS: TestIsGoodMove_110_Composite (0.00s) === RUN TestIsGoodMove_NilReplicaPlacement --- PASS: TestIsGoodMove_NilReplicaPlacement (0.00s) === RUN TestCalculateBalanceScore_ReplicationAware --- PASS: TestCalculateBalanceScore_ReplicationAware (0.00s) === RUN TestPlanBalanceDestination_ChoosesBestValidCompositeDestination --- PASS: TestPlanBalanceDestination_ChoosesBestValidCompositeDestination (0.00s) === RUN TestCreateBalanceTask_FallbackSelectsValidCompositeDestination --- PASS: TestCreateBalanceTask_FallbackSelectsValidCompositeDestination (0.00s) PASS ok github.com/seaweedfs/seaweedfs/weed/worker/tasks/balance 18.461s ? github.com/seaweedfs/seaweedfs/weed/worker/tasks/base [no test files] === RUN TestDedupRefusesToDeleteTheOnlyRealCopy I0824 22:18:30.137761 ec_balance_task.go:96 EC balance: moving shard(s) [0] of volume 42561 from 127.0.0.1:1.46561 to 127.0.0.1:1.46561 --- PASS: TestDedupRefusesToDeleteTheOnlyRealCopy (0.36s) === RUN TestDedupRemovesAGenuineDuplicate I0824 22:18:30.359894 ec_balance_task.go:96 EC balance: moving shard(s) [0] of volume 42561 from 127.0.0.1:1.39887 to 127.0.0.1:1.39887 I0824 22:18:30.411555 ec_balance_task.go:204 EC balance volume 42561: [25.00] Removing duplicate EC shard I0824 22:18:30.453723 ec_balance_task.go:204 EC balance volume 42561: [100.00] Duplicate shard removed --- PASS: TestDedupRemovesAGenuineDuplicate (0.13s) === RUN TestDedupRefusesWhenThePeerCannotBeQueried I0824 22:18:30.466126 ec_balance_task.go:96 EC balance: moving shard(s) [0] of volume 42561 from 127.0.0.1:1.33311 to 127.0.0.1:1.33311 --- PASS: TestDedupRefusesWhenThePeerCannotBeQueried (0.02s) === RUN TestDedupRefusesWhenTheKeepNodeHoldsAnotherCollection I0824 22:18:30.482222 ec_balance_task.go:96 EC balance: moving shard(s) [0] of volume 42561 from 127.0.0.1:1.46253 to 127.0.0.1:1.46253 --- PASS: TestDedupRefusesWhenTheKeepNodeHoldsAnotherCollection (0.04s) === RUN TestBuildBalancerTopology --- PASS: TestBuildBalancerTopology (0.01s) === RUN TestBuildBalancerTopologyGroupsByHost --- PASS: TestBuildBalancerTopologyGroupsByHost (0.01s) === RUN TestBuildBalancerTopologyCollectionFilter --- PASS: TestBuildBalancerTopologyCollectionFilter (0.01s) === RUN TestDetectionDisabled --- PASS: TestDetectionDisabled (0.00s) === RUN TestDetectionNilTopology --- PASS: TestDetectionNilTopology (0.00s) === RUN TestMovePhasePriority --- PASS: TestMovePhasePriority (0.00s) === RUN TestECBalanceTaskRejectsOutOfRangeShardIds --- PASS: TestECBalanceTaskRejectsOutOfRangeShardIds (0.00s) === RUN TestMultiDiskBalanceNeverLosesShards === RUN TestMultiDiskBalanceNeverLosesShards/balanced_across_disks === RUN TestMultiDiskBalanceNeverLosesShards/concentrated_on_one_node === RUN TestMultiDiskBalanceNeverLosesShards/spread_across_racks --- PASS: TestMultiDiskBalanceNeverLosesShards (0.07s) --- PASS: TestMultiDiskBalanceNeverLosesShards/balanced_across_disks (0.03s) --- PASS: TestMultiDiskBalanceNeverLosesShards/concentrated_on_one_node (0.01s) --- PASS: TestMultiDiskBalanceNeverLosesShards/spread_across_racks (0.02s) === RUN TestConcentratedVolumeSpreadsAcrossNodesAndDisks --- PASS: TestConcentratedVolumeSpreadsAcrossNodesAndDisks (0.01s) === RUN TestBuildBalancerTopologyNormalizesHddDiskType --- PASS: TestBuildBalancerTopologyNormalizesHddDiskType (0.00s) === RUN TestResolveReplicaPlacementFallsBackToMasterDefault === RUN TestResolveReplicaPlacementFallsBackToMasterDefault/explicit_config_used === RUN TestResolveReplicaPlacementFallsBackToMasterDefault/explicit_config_wins_over_default === RUN TestResolveReplicaPlacementFallsBackToMasterDefault/falls_back_to_master_default === RUN TestResolveReplicaPlacementFallsBackToMasterDefault/zero_master_default_=_no_constraint === RUN TestResolveReplicaPlacementFallsBackToMasterDefault/empty_everywhere_=_no_constraint === RUN TestResolveReplicaPlacementFallsBackToMasterDefault/invalid_value_ignored W0824 22:18:30.663981 replica_placement.go:119 ignoring invalid replica placement "nonsense": unexpected replication type: nonsense --- PASS: TestResolveReplicaPlacementFallsBackToMasterDefault (0.03s) --- PASS: TestResolveReplicaPlacementFallsBackToMasterDefault/explicit_config_used (0.00s) --- PASS: TestResolveReplicaPlacementFallsBackToMasterDefault/explicit_config_wins_over_default (0.00s) --- PASS: TestResolveReplicaPlacementFallsBackToMasterDefault/falls_back_to_master_default (0.00s) --- PASS: TestResolveReplicaPlacementFallsBackToMasterDefault/zero_master_default_=_no_constraint (0.00s) --- PASS: TestResolveReplicaPlacementFallsBackToMasterDefault/empty_everywhere_=_no_constraint (0.00s) --- PASS: TestResolveReplicaPlacementFallsBackToMasterDefault/invalid_value_ignored (0.03s) === RUN TestValidateRejectsSameNodeCrossDiskMove --- PASS: TestValidateRejectsSameNodeCrossDiskMove (0.00s) === RUN TestIsDedupPhaseRequiresSameDisk --- PASS: TestIsDedupPhaseRequiresSameDisk (0.00s) === RUN TestDeriveECBalanceWorkerConfig === RUN TestDeriveECBalanceWorkerConfig/nil_values_uses_defaults === RUN TestDeriveECBalanceWorkerConfig/custom_threshold === RUN TestDeriveECBalanceWorkerConfig/threshold_clamped_to_min === RUN TestDeriveECBalanceWorkerConfig/threshold_clamped_to_max === RUN TestDeriveECBalanceWorkerConfig/min_server_count_clamped_to_2 --- PASS: TestDeriveECBalanceWorkerConfig (0.01s) --- PASS: TestDeriveECBalanceWorkerConfig/nil_values_uses_defaults (0.00s) --- PASS: TestDeriveECBalanceWorkerConfig/custom_threshold (0.00s) --- PASS: TestDeriveECBalanceWorkerConfig/threshold_clamped_to_min (0.00s) --- PASS: TestDeriveECBalanceWorkerConfig/threshold_clamped_to_max (0.00s) --- PASS: TestDeriveECBalanceWorkerConfig/min_server_count_clamped_to_2 (0.00s) === RUN TestBuildECBalanceProposal --- PASS: TestBuildECBalanceProposal (0.03s) === RUN TestBuildECBalanceProposalNilResult --- PASS: TestBuildECBalanceProposalNilResult (0.00s) === RUN TestDecodeECBalanceTaskParamsFromProtobuf --- PASS: TestDecodeECBalanceTaskParamsFromProtobuf (0.02s) === RUN TestDecodeECBalanceTaskParamsFallback --- PASS: TestDecodeECBalanceTaskParamsFallback (0.00s) === RUN TestDecodeECBalanceTaskParamsProtobufValidation --- PASS: TestDecodeECBalanceTaskParamsProtobufValidation (0.00s) === RUN TestDecodeECBalanceTaskParamsMissingShardID --- PASS: TestDecodeECBalanceTaskParamsMissingShardID (0.00s) === RUN TestECBalanceHandlerCapability --- PASS: TestECBalanceHandlerCapability (0.00s) === RUN TestECBalanceConfigRoundTrip --- PASS: TestECBalanceConfigRoundTrip (0.00s) PASS ok github.com/seaweedfs/seaweedfs/weed/worker/tasks/ec_balance 19.593s === RUN TestPlanECDestinationsPrefersSourceDiskType_FullCluster --- PASS: TestPlanECDestinationsPrefersSourceDiskType_FullCluster (0.12s) === RUN TestPlanECDestinationsSpillsToOtherDiskType_WhenPreferredScarce W0824 22:18:30.602905 detection.go:521 EC volume 2: placed shards outside preferred disk type "ssd" --- PASS: TestPlanECDestinationsSpillsToOtherDiskType_WhenPreferredScarce (0.03s) === RUN TestCountExistingEcShards_DoesNotUnionAcrossGenerations --- PASS: TestCountExistingEcShards_DoesNotUnionAcrossGenerations (0.00s) === RUN TestCountExistingEcShards_SingleCompleteGeneration --- PASS: TestCountExistingEcShards_SingleCompleteGeneration (0.00s) === RUN TestCountExistingEcShards_PreUpgradeZeroGeneration --- PASS: TestCountExistingEcShards_PreUpgradeZeroGeneration (0.00s) === RUN TestPlanECDestinationsUsesPlanner --- PASS: TestPlanECDestinationsUsesPlanner (0.06s) === RUN TestDetectionSkipsWhenECShardsAlreadyExist W0824 22:18:30.686460 detection.go:156 EC Detection: Volume 42 has all 14 EC shards in topology; source replica on 127.0.0.1:8080 is orphaned (#9448). W0824 22:18:30.686795 detection.go:174 EC Detection: no gRPC dial option available to auto-clean orphaned source for volume 42; to clean up by hand, send a targeted VolumeDelete RPC to 127.0.0.1:8080 only — DO NOT use the cluster-wide `volume.delete` shell command, which would also delete the EC shards. --- PASS: TestDetectionSkipsWhenECShardsAlreadyExist (0.02s) === RUN TestDetectionAllowsRegularReplicaWhenShardsPartial I0824 22:18:30.690948 detection.go:217 EC Detection: Volume 43 meets all criteria, attempting to create task I0824 22:18:30.691248 detection.go:243 EC Detection: ActiveTopology available, planning destinations for volume 43 I0824 22:18:30.693670 detection.go:259 EC Detection: Successfully planned 14 destinations for volume 43 I0824 22:18:30.694364 detection.go:276 EC Detection: Looking for replica locations for volume 43 I0824 22:18:30.694686 detection.go:282 EC Detection: Found 1 replica locations for volume 43 I0824 22:18:30.696661 detection.go:401 EC Detection: Successfully created EC task for volume 43, adding to results --- PASS: TestDetectionAllowsRegularReplicaWhenShardsPartial (0.01s) === RUN TestCountExistingEcShardsForVolume --- PASS: TestCountExistingEcShardsForVolume (0.00s) === RUN TestDetectionContextCancellation --- PASS: TestDetectionContextCancellation (0.01s) === RUN TestDetectionMaxResultsHonorsLimit I0824 22:18:30.723034 detection.go:217 EC Detection: Volume 1 meets all criteria, attempting to create task I0824 22:18:30.723291 detection.go:243 EC Detection: ActiveTopology available, planning destinations for volume 1 I0824 22:18:30.725392 detection.go:259 EC Detection: Successfully planned 14 destinations for volume 1 I0824 22:18:30.725807 detection.go:276 EC Detection: Looking for replica locations for volume 1 I0824 22:18:30.727138 detection.go:282 EC Detection: Found 14 replica locations for volume 1 I0824 22:18:30.728196 detection.go:401 EC Detection: Successfully created EC task for volume 1, adding to results --- PASS: TestDetectionMaxResultsHonorsLimit (0.02s) === RUN TestPlanECDestinationsSpreadsAcrossPhysicalDisks --- PASS: TestPlanECDestinationsSpreadsAcrossPhysicalDisks (0.02s) === RUN TestPlanECDestinationsFailsWithInsufficientCapacity --- PASS: TestPlanECDestinationsFailsWithInsufficientCapacity (0.00s) === RUN TestPlanECDestinationsPacksWhenFewerDisksThanShards --- PASS: TestPlanECDestinationsPacksWhenFewerDisksThanShards (0.00s) === RUN TestDeleteOriginalVolumeSurfacesReplicaFailures I0824 22:21:51.990313 task.go:63 Starting original volume deletion from replica servers I0824 22:21:51.990467 task.go:63 Deleting volume from replica server I0824 22:21:52.012965 task.go:63 Successfully deleted volume from replica server I0824 22:21:52.013130 task.go:63 Deleting volume from replica server E0824 22:21:52.021402 task.go:79 Failed to delete volume from replica server E0824 22:21:52.025973 task.go:79 Failed to delete some original volume replicas after EC encoding --- PASS: TestDeleteOriginalVolumeSurfacesReplicaFailures (211.41s) === RUN TestDeleteOriginalVolumeSucceedsWhenAllReplicasReachable I0824 22:22:05.923893 task.go:63 Starting original volume deletion from replica servers I0824 22:22:05.924282 task.go:63 Deleting volume from replica server I0824 22:22:05.941345 task.go:63 Successfully deleted volume from replica server I0824 22:22:05.941757 task.go:63 Successfully deleted volume from all replica servers --- PASS: TestDeleteOriginalVolumeSucceedsWhenAllReplicasReachable (13.90s) === RUN TestReplicasPendingDelete --- PASS: TestReplicasPendingDelete (0.00s) === RUN TestIsExpectedSweepSkip --- PASS: TestIsExpectedSweepSkip (0.00s) === RUN TestSweepEmptyReplicasDeletesStubKeepsData I0824 22:22:19.928525 ec_task.go:905 EC volume 9479: removed empty stub replica on 127.0.0.1:35229.40595 before distribute --- PASS: TestSweepEmptyReplicasDeletesStubKeepsData (14.01s) === RUN TestEcEncodeLeavesRightFilesAndRemovesStubAndSource I0824 22:22:35.120765 task.go:63 Starting erasure coding task I0824 22:22:35.121066 task.go:63 Target server shard assignment I0824 22:22:35.121133 task.go:63 Target server shard assignment I0824 22:22:35.121200 task.go:63 Source server information I0824 22:22:35.121242 task.go:63 Source server information I0824 22:22:35.121599 task.go:63 Verifying preconditions and clearing stale EC state I0824 22:22:35.121760 task.go:63 Clearing stale EC shards on destination before re-distribute I0824 22:22:35.143433 task.go:63 Clearing stale EC shards on destination before re-distribute I0824 22:22:35.163667 task.go:63 Marking volume readonly I0824 22:22:35.491671 ec_task.go:392 EC encode replica sync for volume 9490: volume 9490: replicas are inconsistent, building union... volume 9490: selected 127.0.0.1:37117.46229 as best replica (file count: 8) I0824 22:22:35.491943 task.go:63 Copying volume files to worker I0824 22:22:35.495328 task.go:63 Starting volume file copy from source server I0824 22:22:35.501904 task.go:63 Volume index file copied successfully I0824 22:22:35.508895 task.go:63 Volume data file copied successfully I0824 22:22:35.509067 task.go:63 Generating EC shards locally I0824 22:22:36.068358 ec_encoder.go:138 encodeDatFile /tmp/TestEcEncodeLeavesRightFilesAndRemovesStubAndSource1004316427/001/vol_9490_1787610155/9490.dat size:33480 with EC context 10+4 (total: 14) I0824 22:22:36.473677 task.go:63 EC shard generated I0824 22:22:36.474407 task.go:63 EC shard generated I0824 22:22:36.474821 task.go:63 EC shard generated I0824 22:22:36.475002 task.go:63 EC shard generated I0824 22:22:36.475158 task.go:63 EC shard generated I0824 22:22:36.475432 task.go:63 EC shard generated I0824 22:22:36.475604 task.go:63 EC shard generated I0824 22:22:36.475663 task.go:63 EC shard generated I0824 22:22:36.475758 task.go:63 EC shard generated I0824 22:22:36.475917 task.go:63 EC shard generated I0824 22:22:36.475974 task.go:63 EC shard generated I0824 22:22:36.476028 task.go:63 EC shard generated I0824 22:22:36.476079 task.go:63 EC shard generated I0824 22:22:36.476191 task.go:63 EC shard generated I0824 22:22:36.477122 task.go:63 EC index file generated I0824 22:22:36.481215 task.go:63 Volume info file generated I0824 22:22:36.481523 task.go:63 EC bitrot checksum sidecar generated I0824 22:22:36.481663 task.go:63 EC shard generation completed I0824 22:22:36.481711 task.go:63 Removing empty stub replicas before distribute I0824 22:22:36.504554 ec_task.go:905 EC volume 9490: removed empty stub replica on 127.0.0.1:36485.32775 before distribute I0824 22:22:36.505811 task.go:63 Distributing EC shards to destinations I0824 22:22:36.507125 task.go:63 Starting shard distribution to destination server I0824 22:22:36.508263 task.go:63 Starting shard file transfer I0824 22:22:36.547477 task.go:63 Shard file transfer completed I0824 22:22:36.548377 task.go:63 Starting shard file transfer I0824 22:22:36.604070 task.go:63 Shard file transfer completed I0824 22:22:36.605325 task.go:63 Starting shard file transfer I0824 22:22:36.959978 task.go:63 Shard file transfer completed I0824 22:22:36.960285 task.go:63 Starting shard file transfer I0824 22:22:37.013177 task.go:63 Shard file transfer completed I0824 22:22:37.013444 task.go:63 Starting shard file transfer I0824 22:22:37.047076 task.go:63 Shard file transfer completed I0824 22:22:37.047592 task.go:63 Starting shard file transfer I0824 22:22:37.083433 task.go:63 Shard file transfer completed I0824 22:22:37.083939 task.go:63 Starting shard file transfer I0824 22:22:37.124211 task.go:63 Shard file transfer completed I0824 22:22:37.124976 task.go:63 Starting shard file transfer I0824 22:22:37.160979 task.go:63 Shard file transfer completed I0824 22:22:37.161448 task.go:63 Starting shard file transfer I0824 22:22:37.197589 task.go:63 Shard file transfer completed I0824 22:22:37.198115 task.go:63 Starting shard file transfer I0824 22:22:37.232552 task.go:63 Shard file transfer completed I0824 22:22:37.232781 task.go:63 Starting shard file transfer I0824 22:22:37.242087 task.go:63 Shard file transfer completed I0824 22:22:37.242723 task.go:63 Starting shard file transfer I0824 22:22:37.254004 task.go:63 Shard file transfer completed I0824 22:22:37.254725 task.go:63 Starting shard file transfer I0824 22:22:37.269027 task.go:63 Shard file transfer completed I0824 22:22:37.269337 task.go:63 All shards distributed to destination server I0824 22:22:37.269435 task.go:63 Starting shard distribution to destination server I0824 22:22:37.269690 task.go:63 Starting shard file transfer I0824 22:22:37.305597 task.go:63 Shard file transfer completed I0824 22:22:37.306136 task.go:63 Starting shard file transfer I0824 22:22:37.461801 task.go:63 Shard file transfer completed I0824 22:22:37.462448 task.go:63 Starting shard file transfer I0824 22:22:37.535633 task.go:63 Shard file transfer completed I0824 22:22:37.536141 task.go:63 Starting shard file transfer I0824 22:22:37.580558 task.go:63 Shard file transfer completed I0824 22:22:37.581063 task.go:63 Starting shard file transfer I0824 22:22:37.597285 task.go:63 Shard file transfer completed I0824 22:22:37.597819 task.go:63 Starting shard file transfer I0824 22:22:37.615087 task.go:63 Shard file transfer completed I0824 22:22:37.615756 task.go:63 Starting shard file transfer I0824 22:22:37.638925 task.go:63 Shard file transfer completed I0824 22:22:37.639267 task.go:63 All shards distributed to destination server I0824 22:22:37.639464 task.go:63 Mounting EC shards I0824 22:22:37.639767 task.go:63 Starting EC shard mount operation I0824 22:22:37.660577 task.go:63 Successfully mounted EC shards I0824 22:22:37.660968 task.go:63 Starting EC shard mount operation I0824 22:22:37.711723 task.go:63 Successfully mounted EC shards I0824 22:22:37.711991 task.go:63 Verifying EC shards across destinations I0824 22:22:37.736911 task.go:63 EC shard inventory before source deletion I0824 22:22:37.737201 task.go:63 Deleting original volume I0824 22:22:37.737287 task.go:63 Starting original volume deletion from replica servers I0824 22:22:37.737417 task.go:63 Deleting volume from replica server I0824 22:22:37.745531 task.go:63 Successfully deleted volume from replica server I0824 22:22:37.745838 task.go:63 Successfully deleted volume from all replica servers I0824 22:22:37.746007 ec_task.go:272 EC task completed successfully: volume 9490 from 127.0.0.1:37117.46229 with 17 shards distributed --- PASS: TestEcEncodeLeavesRightFilesAndRemovesStubAndSource (27.90s) === RUN TestEcEncodeJulorLayoutConverges I0824 22:23:05.973573 task.go:63 Starting erasure coding task I0824 22:23:05.973752 task.go:63 Target server shard assignment I0824 22:23:05.973799 task.go:63 Target server shard assignment I0824 22:23:05.973828 task.go:63 Target server shard assignment I0824 22:23:05.973869 task.go:63 Target server shard assignment I0824 22:23:05.973911 task.go:63 Target server shard assignment I0824 22:23:05.973935 task.go:63 Target server shard assignment I0824 22:23:05.973960 task.go:63 Source server information I0824 22:23:05.973983 task.go:63 Source server information I0824 22:23:05.974019 task.go:63 Source server information I0824 22:23:05.974612 task.go:63 Verifying preconditions and clearing stale EC state I0824 22:23:05.974764 task.go:63 Clearing stale EC shards on destination before re-distribute I0824 22:23:05.995592 task.go:63 Clearing stale EC shards on destination before re-distribute I0824 22:23:06.015291 task.go:63 Clearing stale EC shards on destination before re-distribute I0824 22:23:06.033651 task.go:63 Marking volume readonly I0824 22:23:06.128146 ec_task.go:392 EC encode replica sync for volume 9500: volume 9500: replicas are inconsistent, building union... volume 9500: selected 127.0.0.1:38683.34395 as best replica (file count: 8) I0824 22:23:06.128453 task.go:63 Copying volume files to worker I0824 22:23:06.132075 task.go:63 Starting volume file copy from source server I0824 22:23:06.145970 task.go:63 Volume index file copied successfully I0824 22:23:06.154743 task.go:63 Volume data file copied successfully I0824 22:23:06.154920 task.go:63 Generating EC shards locally I0824 22:23:06.163879 ec_encoder.go:138 encodeDatFile /tmp/TestEcEncodeJulorLayoutConverges2599294502/001/vol_9500_1787610185/9500.dat size:33480 with EC context 10+4 (total: 14) I0824 22:23:06.285845 task.go:63 EC shard generated I0824 22:23:06.286253 task.go:63 EC shard generated I0824 22:23:06.286481 task.go:63 EC shard generated I0824 22:23:06.286651 task.go:63 EC shard generated I0824 22:23:06.286736 task.go:63 EC shard generated I0824 22:23:06.286883 task.go:63 EC shard generated I0824 22:23:06.286941 task.go:63 EC shard generated I0824 22:23:06.286995 task.go:63 EC shard generated I0824 22:23:06.287100 task.go:63 EC shard generated I0824 22:23:06.287412 task.go:63 EC shard generated I0824 22:23:06.287569 task.go:63 EC shard generated I0824 22:23:06.287630 task.go:63 EC shard generated I0824 22:23:06.287681 task.go:63 EC shard generated I0824 22:23:06.287733 task.go:63 EC shard generated I0824 22:23:06.289887 task.go:63 EC index file generated I0824 22:23:06.293527 task.go:63 Volume info file generated I0824 22:23:06.293820 task.go:63 EC bitrot checksum sidecar generated I0824 22:23:06.293962 task.go:63 EC shard generation completed I0824 22:23:06.294004 task.go:63 Removing empty stub replicas before distribute I0824 22:23:06.306746 ec_task.go:905 EC volume 9500: removed empty stub replica on 127.0.0.1:35397.32769 before distribute I0824 22:23:06.307032 task.go:63 Distributing EC shards to destinations I0824 22:23:06.307469 task.go:63 Starting shard distribution to destination server I0824 22:23:06.307956 task.go:63 Starting shard file transfer I0824 22:23:06.343555 task.go:63 Shard file transfer completed I0824 22:23:06.344041 task.go:63 Starting shard file transfer I0824 22:23:06.379043 task.go:63 Shard file transfer completed I0824 22:23:06.381685 task.go:63 Starting shard file transfer I0824 22:23:06.420215 task.go:63 Shard file transfer completed I0824 22:23:06.420711 task.go:63 Starting shard file transfer I0824 22:23:06.429384 task.go:63 Shard file transfer completed I0824 22:23:06.429980 task.go:63 Starting shard file transfer I0824 22:23:06.436915 task.go:63 Shard file transfer completed I0824 22:23:06.439051 task.go:63 Starting shard file transfer I0824 22:23:06.444181 task.go:63 Shard file transfer completed I0824 22:23:06.444853 task.go:63 Starting shard file transfer I0824 22:23:06.486511 task.go:63 Shard file transfer completed I0824 22:23:06.486970 task.go:63 Starting shard file transfer I0824 22:23:06.519011 task.go:63 Shard file transfer completed I0824 22:23:06.519323 task.go:63 All shards distributed to destination server I0824 22:23:06.519713 task.go:63 Starting shard distribution to destination server I0824 22:23:06.520035 task.go:63 Starting shard file transfer I0824 22:23:06.550716 task.go:63 Shard file transfer completed I0824 22:23:06.551184 task.go:63 Starting shard file transfer I0824 22:23:06.575850 task.go:63 Shard file transfer completed I0824 22:23:06.576359 task.go:63 Starting shard file transfer I0824 22:23:06.880742 task.go:63 Shard file transfer completed I0824 22:23:06.881233 task.go:63 Starting shard file transfer I0824 22:23:06.888313 task.go:63 Shard file transfer completed I0824 22:23:06.888800 task.go:63 Starting shard file transfer I0824 22:23:06.892955 task.go:63 Shard file transfer completed I0824 22:23:06.893472 task.go:63 Starting shard file transfer I0824 22:23:06.900421 task.go:63 Shard file transfer completed I0824 22:23:06.900898 task.go:63 Starting shard file transfer I0824 22:23:06.929745 task.go:63 Shard file transfer completed I0824 22:23:06.930226 task.go:63 Starting shard file transfer I0824 22:23:06.960250 task.go:63 Shard file transfer completed I0824 22:23:06.960550 task.go:63 All shards distributed to destination server I0824 22:23:06.960728 task.go:63 Starting shard distribution to destination server I0824 22:23:06.960981 task.go:63 Starting shard file transfer I0824 22:23:07.022192 task.go:63 Shard file transfer completed I0824 22:23:07.022722 task.go:63 Starting shard file transfer I0824 22:23:07.061582 task.go:63 Shard file transfer completed I0824 22:23:07.062104 task.go:63 Starting shard file transfer I0824 22:23:07.067235 task.go:63 Shard file transfer completed I0824 22:23:07.067744 task.go:63 Starting shard file transfer I0824 22:23:07.071101 task.go:63 Shard file transfer completed I0824 22:23:07.071602 task.go:63 Starting shard file transfer I0824 22:23:07.075576 task.go:63 Shard file transfer completed I0824 22:23:07.076144 task.go:63 Starting shard file transfer I0824 22:23:07.106857 task.go:63 Shard file transfer completed I0824 22:23:07.107347 task.go:63 Starting shard file transfer I0824 22:23:07.139697 task.go:63 Shard file transfer completed I0824 22:23:07.140298 task.go:63 All shards distributed to destination server I0824 22:23:07.140486 task.go:63 Mounting EC shards I0824 22:23:07.140637 task.go:63 Starting EC shard mount operation I0824 22:23:07.155751 task.go:63 Successfully mounted EC shards I0824 22:23:07.156102 task.go:63 Starting EC shard mount operation I0824 22:23:07.176170 task.go:63 Successfully mounted EC shards I0824 22:23:07.176501 task.go:63 Starting EC shard mount operation I0824 22:23:07.183361 task.go:63 Successfully mounted EC shards I0824 22:23:07.183574 task.go:63 Verifying EC shards across destinations I0824 22:23:07.204017 task.go:63 EC shard inventory before source deletion I0824 22:23:07.204304 task.go:63 Deleting original volume I0824 22:23:07.204423 task.go:63 Starting original volume deletion from replica servers I0824 22:23:07.204564 task.go:63 Deleting volume from replica server I0824 22:23:07.214398 task.go:63 Successfully deleted volume from replica server I0824 22:23:07.215131 task.go:63 Successfully deleted volume from all replica servers I0824 22:23:07.215323 ec_task.go:272 EC task completed successfully: volume 9500 from 127.0.0.1:38683.34395 with 17 shards distributed --- PASS: TestEcEncodeJulorLayoutConverges (39.51s) === RUN TestCleanupOrphanSkipsWritableSourceReplica W0824 22:23:41.545055 detection.go:748 EC Detection: source replica for volume 955 on 127.0.0.1:33197.32923 is writable; not deleting (may hold writes the EC shards lack) --- PASS: TestCleanupOrphanSkipsWritableSourceReplica (14.16s) === RUN TestEnsureCleanEcStartClearsStaleShardsBeforeEncode I0824 22:23:54.749256 task.go:63 Clearing stale EC shards on destination before re-distribute --- PASS: TestEnsureCleanEcStartClearsStaleShardsBeforeEncode (13.21s) === RUN TestRollbackDistributeClearsShardsAndKeepsSource I0824 22:24:09.136924 task.go:63 Clearing stale EC shards on destination before re-distribute I0824 22:24:09.185450 ec_task.go:423 restored volume 94783 to writable on 127.0.0.1:33979 after EC task failure --- PASS: TestRollbackDistributeClearsShardsAndKeepsSource (14.46s) === RUN TestEnsureCleanEcStartRejectsMissingTargets --- PASS: TestEnsureCleanEcStartRejectsMissingTargets (0.00s) === RUN TestEnsureCleanEcStartRejectsMalformedTarget === RUN TestEnsureCleanEcStartRejectsMalformedTarget/nil_target === RUN TestEnsureCleanEcStartRejectsMalformedTarget/empty_node === RUN TestEnsureCleanEcStartRejectsMalformedTarget/no_shards === RUN TestEnsureCleanEcStartRejectsMalformedTarget/one_good_one_bad --- PASS: TestEnsureCleanEcStartRejectsMalformedTarget (0.00s) --- PASS: TestEnsureCleanEcStartRejectsMalformedTarget/nil_target (0.00s) --- PASS: TestEnsureCleanEcStartRejectsMalformedTarget/empty_node (0.00s) --- PASS: TestEnsureCleanEcStartRejectsMalformedTarget/no_shards (0.00s) --- PASS: TestEnsureCleanEcStartRejectsMalformedTarget/one_good_one_bad (0.00s) === RUN TestEnsureCleanEcStartRejectsMissingSource --- PASS: TestEnsureCleanEcStartRejectsMissingSource (0.00s) === RUN TestCleanupStaleEcShardsBeforeDistribute I0824 22:24:22.485520 task.go:63 Clearing stale EC shards on destination before re-distribute --- PASS: TestCleanupStaleEcShardsBeforeDistribute (13.26s) === RUN TestCleanupStaleEcShardsClearsShardsBeyondSources I0824 22:24:36.978311 task.go:63 Clearing stale EC shards on destination before re-distribute --- PASS: TestCleanupStaleEcShardsClearsShardsBeyondSources (14.53s) === RUN TestCleanupStaleEcShardsCoversTargetsWithoutSources I0824 22:24:50.703410 task.go:63 Clearing stale EC shards on destination before re-distribute --- PASS: TestCleanupStaleEcShardsCoversTargetsWithoutSources (13.68s) === RUN TestCleanupStaleEcShardsSkipsRegularReplicas --- PASS: TestCleanupStaleEcShardsSkipsRegularReplicas (14.27s) === RUN TestCopyVolumeFilesToWorkerUsesCurrentCompactionRevision I0824 22:25:18.668405 task.go:63 Starting volume file copy from source server I0824 22:25:18.676816 task.go:63 Volume index file copied successfully I0824 22:25:18.685936 task.go:63 Volume data file copied successfully --- PASS: TestCopyVolumeFilesToWorkerUsesCurrentCompactionRevision (13.76s) === RUN TestMarkReplicasReadonlyPersists --- PASS: TestMarkReplicasReadonlyPersists (14.61s) === RUN TestGenerateEcShardsLocallyStampsEncodeIdentity ec_task_test.go:155: Error Trace: /build/seaweedfs/src/seaweedfs-4.44/weed/worker/tasks/erasure_coding/ec_task_test.go:155 Error: Received unexpected error: mark volume 952 readonly on 127.0.0.1:38563: rpc error: code = Unknown desc = grpc VolumeMarkReadonly with master : set volume 952 to read only on master: rpc error: code = Unavailable desc = connection error: desc = "transport: Error while dialing: dial tcp: missing address" Test: TestGenerateEcShardsLocallyStampsEncodeIdentity cluster.go:194: volume server integration logs kept at /tmp/seaweedfs_volume_server_it_390402757 --- FAIL: TestGenerateEcShardsLocallyStampsEncodeIdentity (12.73s) === RUN TestGenerateEcShardsLocallyUsesAdminGeneration I0824 22:26:00.919166 task.go:63 Starting volume file copy from source server I0824 22:26:00.926351 task.go:63 Volume index file copied successfully I0824 22:26:00.932004 task.go:63 Volume data file copied successfully I0824 22:26:00.941470 ec_encoder.go:138 encodeDatFile /tmp/TestGenerateEcShardsLocallyUsesAdminGeneration3800818229/001/957.dat size:120 with EC context 10+4 (total: 14) I0824 22:26:01.707794 task.go:63 EC shard generated I0824 22:26:01.708185 task.go:63 EC shard generated I0824 22:26:01.709429 task.go:63 EC shard generated I0824 22:26:01.709680 task.go:63 EC shard generated I0824 22:26:01.709851 task.go:63 EC shard generated I0824 22:26:01.710055 task.go:63 EC shard generated I0824 22:26:01.710278 task.go:63 EC shard generated I0824 22:26:01.710475 task.go:63 EC shard generated I0824 22:26:01.710659 task.go:63 EC shard generated I0824 22:26:01.710842 task.go:63 EC shard generated I0824 22:26:01.710998 task.go:63 EC shard generated I0824 22:26:01.711066 task.go:63 EC shard generated I0824 22:26:01.711222 task.go:63 EC shard generated I0824 22:26:01.711285 task.go:63 EC shard generated I0824 22:26:01.713809 task.go:63 EC index file generated I0824 22:26:01.717812 task.go:63 Volume info file generated I0824 22:26:01.718257 task.go:63 EC bitrot checksum sidecar generated I0824 22:26:01.718646 task.go:63 EC shard generation completed --- PASS: TestGenerateEcShardsLocallyUsesAdminGeneration (15.67s) === RUN TestIsStubReplica --- PASS: TestIsStubReplica (0.00s) === RUN TestSelectCanonicalMetricPrefersCredibleOverLowServerStub --- PASS: TestSelectCanonicalMetricPrefersCredibleOverLowServerStub (0.00s) === RUN TestSelectCanonicalMetricSkipsECMetrics --- PASS: TestSelectCanonicalMetricSkipsECMetrics (0.00s) === RUN TestSelectCanonicalMetricAllStubsFallsBackToLowestServer --- PASS: TestSelectCanonicalMetricAllStubsFallsBackToLowestServer (0.00s) === RUN TestSelectCanonicalMetricTieBreaksByServerAmongCredible --- PASS: TestSelectCanonicalMetricTieBreaksByServerAmongCredible (0.00s) === RUN TestSelectCanonicalMetricEmpty --- PASS: TestSelectCanonicalMetricEmpty (0.00s) === RUN TestDetectionEncodesDespiteLowServerStub I0824 22:26:11.862163 detection.go:217 EC Detection: Volume 13 meets all criteria, attempting to create task I0824 22:26:11.862798 detection.go:243 EC Detection: ActiveTopology available, planning destinations for volume 13 I0824 22:26:11.866215 detection.go:259 EC Detection: Successfully planned 14 destinations for volume 13 I0824 22:26:11.866384 detection.go:276 EC Detection: Looking for replica locations for volume 13 I0824 22:26:11.867258 detection.go:282 EC Detection: Found 2 replica locations for volume 13 I0824 22:26:11.867665 detection.go:401 EC Detection: Successfully created EC task for volume 13, adding to results --- PASS: TestDetectionEncodesDespiteLowServerStub (0.01s) === RUN TestDecodeErasureCodingTaskParamsFromPayload --- PASS: TestDecodeErasureCodingTaskParamsFromPayload (0.01s) === RUN TestDecodeErasureCodingTaskParamsFallback --- PASS: TestDecodeErasureCodingTaskParamsFallback (0.00s) === RUN TestDeriveErasureCodingWorkerConfig --- PASS: TestDeriveErasureCodingWorkerConfig (0.00s) === RUN TestBuildErasureCodingProposal --- PASS: TestBuildErasureCodingProposal (0.00s) === RUN TestErasureCodingHandlerRejectsUnsupportedJobType --- PASS: TestErasureCodingHandlerRejectsUnsupportedJobType (0.00s) === RUN TestEmitErasureCodingDetectionDecisionTraceNoTasks --- PASS: TestEmitErasureCodingDetectionDecisionTraceNoTasks (0.00s) === RUN TestErasureCodingDescriptorOmitsLocalExecutionFields --- PASS: TestErasureCodingDescriptorOmitsLocalExecutionFields (0.00s) === RUN TestApplyErasureCodingExecutionDefaultsForcesLocalFields --- PASS: TestApplyErasureCodingExecutionDefaultsForcesLocalFields (0.00s) FAIL FAIL github.com/seaweedfs/seaweedfs/weed/worker/tasks/erasure_coding 480.636s === RUN TestExpireSnapshotsExecution exec_test.go:589: expireSnapshots result: expired 2 snapshot(s), deleted 6 unreferenced file(s) --- PASS: TestExpireSnapshotsExecution (2.33s) === RUN TestExpireSnapshotsKeepsTaggedSnapshot --- PASS: TestExpireSnapshotsKeepsTaggedSnapshot (0.71s) === RUN TestExpireSnapshotsHonorsBranchRetention --- PASS: TestExpireSnapshotsHonorsBranchRetention (0.42s) === RUN TestExpireSnapshotsHonorsBranchMaxSnapshotAge --- PASS: TestExpireSnapshotsHonorsBranchMaxSnapshotAge (0.33s) === RUN TestExpireSnapshotsRejectsPlanWhenARefAppears --- PASS: TestExpireSnapshotsRejectsPlanWhenARefAppears (0.51s) === RUN TestExpireSnapshotsNothingToExpire --- PASS: TestExpireSnapshotsNothingToExpire (0.07s) === RUN TestMaintenanceOnExternalTableLocation --- PASS: TestMaintenanceOnExternalTableLocation (0.30s) === RUN TestCompactDataFilesOnExternalTableLocation --- PASS: TestCompactDataFilesOnExternalTableLocation (0.33s) === RUN TestRemoveOrphansExecution --- PASS: TestRemoveOrphansExecution (0.14s) === RUN TestRemoveOrphansPreservesReferencedFiles --- PASS: TestRemoveOrphansPreservesReferencedFiles (0.12s) === RUN TestMaintenanceOnManifestListWithoutFormatVersion --- PASS: TestMaintenanceOnManifestListWithoutFormatVersion (0.07s) === RUN TestRewriteManifestsExecution exec_test.go:1284: rewriteManifests result: rewrote 5 manifests into 1 (5 entries) --- PASS: TestRewriteManifestsExecution (1.18s) === RUN TestRewriteManifestsBelowThreshold --- PASS: TestRewriteManifestsBelowThreshold (0.57s) === RUN TestFullExecuteFlow exec_test.go:1465: Full execution results: expire_snapshots: no snapshots expired; remove_orphans: removed 1 orphan file(s) --- PASS: TestFullExecuteFlow (0.41s) === RUN TestDetectWithFakeFiler --- PASS: TestDetectWithFakeFiler (0.17s) === RUN TestDetectWithFilters --- PASS: TestDetectWithFilters (1.72s) === RUN TestConnectToFilerSkipsUnreachableAddresses --- PASS: TestConnectToFilerSkipsUnreachableAddresses (0.06s) === RUN TestConnectToFilerFailsWhenAllAddressesAreUnreachable --- PASS: TestConnectToFilerFailsWhenAllAddressesAreUnreachable (0.03s) === RUN TestDetectSchedulesCompactionWithoutSnapshotPressure --- PASS: TestDetectSchedulesCompactionWithoutSnapshotPressure (0.52s) === RUN TestDetectSchedulesCompactionWithDeleteManifestPresent --- PASS: TestDetectSchedulesCompactionWithDeleteManifestPresent (0.50s) === RUN TestDetectSchedulesSnapshotExpiryDespiteCompactionEvaluationError --- PASS: TestDetectSchedulesSnapshotExpiryDespiteCompactionEvaluationError (0.30s) === RUN TestDetectSchedulesManifestRewriteWithoutSnapshotPressure --- PASS: TestDetectSchedulesManifestRewriteWithoutSnapshotPressure (1.01s) === RUN TestDetectUsesPlanningIndexForRepeatedCompactionScans --- PASS: TestDetectUsesPlanningIndexForRepeatedCompactionScans (0.15s) === RUN TestDetectInvalidatesPlanningIndexWhenCompactionConfigChanges --- PASS: TestDetectInvalidatesPlanningIndexWhenCompactionConfigChanges (0.35s) === RUN TestDetectPlanningIndexPreservesUnscannedSections --- PASS: TestDetectPlanningIndexPreservesUnscannedSections (0.24s) === RUN TestTableNeedsMaintenanceCachesPlanningIndexBuildError --- PASS: TestTableNeedsMaintenanceCachesPlanningIndexBuildError (0.06s) === RUN TestTableNeedsMaintenanceScopesPlanningIndexBuildErrorsPerOperation --- PASS: TestTableNeedsMaintenanceScopesPlanningIndexBuildErrorsPerOperation (0.16s) === RUN TestPersistPlanningIndexUsesMetadataXattrCASGuard --- PASS: TestPersistPlanningIndexUsesMetadataXattrCASGuard (0.09s) === RUN TestDetectDoesNotScheduleManifestRewriteFromDeleteManifestsOnly --- PASS: TestDetectDoesNotScheduleManifestRewriteFromDeleteManifestsOnly (0.09s) === RUN TestDetectSchedulesOrphanCleanupWithoutSnapshotPressure --- PASS: TestDetectSchedulesOrphanCleanupWithoutSnapshotPressure (0.13s) === RUN TestDetectSchedulesOrphanCleanupWithoutSnapshots --- PASS: TestDetectSchedulesOrphanCleanupWithoutSnapshots (0.04s) === RUN TestStalePlanGuard --- PASS: TestStalePlanGuard (0.04s) === RUN TestMetadataVersionCAS --- PASS: TestMetadataVersionCAS (0.06s) === RUN TestMetadataVersionCASDetectsConcurrentUpdate --- PASS: TestMetadataVersionCASDetectsConcurrentUpdate (0.32s) === RUN TestCompactDataFilesMetrics --- PASS: TestCompactDataFilesMetrics (0.43s) === RUN TestCompactDataFilesRecordsSnapshotTotals --- PASS: TestCompactDataFilesRecordsSnapshotTotals (0.20s) === RUN TestCompactDataFilesLeavesOutTotalsParentNeverRecorded --- PASS: TestCompactDataFilesLeavesOutTotalsParentNeverRecorded (0.43s) === RUN TestRewritePositionDeleteFilesRecordsSnapshotTotals --- PASS: TestRewritePositionDeleteFilesRecordsSnapshotTotals (0.37s) === RUN TestExpireSnapshotsMetrics --- PASS: TestExpireSnapshotsMetrics (0.57s) === RUN TestExecuteCompletionOutputValues --- PASS: TestExecuteCompletionOutputValues (0.38s) === RUN TestCompactDataFilesWithPositionDeletes exec_test.go:3418: result: compacted 2 files into 1 (across 1 bins) --- PASS: TestCompactDataFilesWithPositionDeletes (0.60s) === RUN TestCompactDataFilesWithEqualityDeletes exec_test.go:3518: result: compacted 2 files into 1 (across 1 bins) --- PASS: TestCompactDataFilesWithEqualityDeletes (0.36s) === RUN TestCompactDataFilesApplyDeletesDisabled --- PASS: TestCompactDataFilesApplyDeletesDisabled (0.22s) === RUN TestCompactDataFilesWithMixedDeletes --- PASS: TestCompactDataFilesWithMixedDeletes (0.47s) === RUN TestCompactDataFilesSortStrategyUsesAscendingTableSortOrder --- PASS: TestCompactDataFilesSortStrategyUsesAscendingTableSortOrder (0.28s) === RUN TestCompactDataFilesSortStrategyUsesTableSortOrder --- PASS: TestCompactDataFilesSortStrategyUsesTableSortOrder (0.29s) === RUN TestDetectSkipsSortCompactionBinsAboveCap --- PASS: TestDetectSkipsSortCompactionBinsAboveCap (0.12s) === RUN TestDetectSplitsSortCompactionBinsByCap --- PASS: TestDetectSplitsSortCompactionBinsByCap (0.20s) === RUN TestCompactDataFilesSortStrategyRequiresTableSortOrder --- PASS: TestCompactDataFilesSortStrategyRequiresTableSortOrder (0.07s) === RUN TestRewritePositionDeleteFilesExecution --- PASS: TestRewritePositionDeleteFilesExecution (0.28s) === RUN TestRewritePositionDeleteFilesDetection --- PASS: TestRewritePositionDeleteFilesDetection (0.11s) === RUN TestRewritePositionDeleteFilesSkipsSingleFile --- PASS: TestRewritePositionDeleteFilesSkipsSingleFile (0.14s) === RUN TestRewritePositionDeleteFilesRespectsMinInputFiles --- PASS: TestRewritePositionDeleteFilesRespectsMinInputFiles (0.17s) === RUN TestRewritePositionDeleteFilesPreservesUnsupportedMultiTargetDeletes --- PASS: TestRewritePositionDeleteFilesPreservesUnsupportedMultiTargetDeletes (0.30s) === RUN TestRewritePositionDeleteFilesRebuildsMixedDeleteManifests --- PASS: TestRewritePositionDeleteFilesRebuildsMixedDeleteManifests (0.43s) === RUN TestResolveCompactionRewritePlanFallsBackForUnsupportedSortTransform --- PASS: TestResolveCompactionRewritePlanFallsBackForUnsupportedSortTransform (0.00s) === RUN TestSaveFilerFileKeepsSmallContentInline --- PASS: TestSaveFilerFileKeepsSmallContentInline (0.03s) === RUN TestSaveFilerFileChunksLargeContent --- PASS: TestSaveFilerFileChunksLargeContent (2.02s) === RUN TestSaveFilerFileFailsWhenUploadFails E0824 22:18:59.130642 upload_chunked.go:281 chunked upload failed: assign volume: rpc error: code = Unavailable desc = no volume server (returning 0 partial chunks for cleanup) --- PASS: TestSaveFilerFileFailsWhenUploadFails (0.14s) === RUN TestOrphanCleanupWouldDeleteAnAdapterRegisteredLanceDataset --- PASS: TestOrphanCleanupWouldDeleteAnAdapterRegisteredLanceDataset (0.02s) === RUN TestScanSkipsAnAdapterRegisteredLanceTable --- PASS: TestScanSkipsAnAdapterRegisteredLanceTable (0.06s) === RUN TestIsIcebergTableEntry === RUN TestIsIcebergTableEntry/plain_table === RUN TestIsIcebergTableEntry/iceberg_table_type_is_honoured_case-insensitively === RUN TestIsIcebergTableEntry/lance_table_registered_through_the_iceberg_adapter === RUN TestIsIcebergTableEntry/view === RUN TestIsIcebergTableEntry/explicit_table_marker --- PASS: TestIsIcebergTableEntry (0.00s) --- PASS: TestIsIcebergTableEntry/plain_table (0.00s) --- PASS: TestIsIcebergTableEntry/iceberg_table_type_is_honoured_case-insensitively (0.00s) --- PASS: TestIsIcebergTableEntry/lance_table_registered_through_the_iceberg_adapter (0.00s) --- PASS: TestIsIcebergTableEntry/view (0.00s) --- PASS: TestIsIcebergTableEntry/explicit_table_marker (0.00s) === RUN TestParseTableMetadataEnvelopeRejectsForeignFormats --- PASS: TestParseTableMetadataEnvelopeRejectsForeignFormats (0.00s) === RUN TestParseConfig --- PASS: TestParseConfig (0.00s) === RUN TestParseOperations --- PASS: TestParseOperations (0.00s) === RUN TestExtractMetadataVersion --- PASS: TestExtractMetadataVersion (0.00s) === RUN TestNeedsMaintenanceNoSnapshots --- PASS: TestNeedsMaintenanceNoSnapshots (0.00s) === RUN TestNeedsMaintenanceExceedsMaxSnapshots --- PASS: TestNeedsMaintenanceExceedsMaxSnapshots (0.00s) === RUN TestNeedsMaintenanceExceedsMaxSnapshotsWithinRetention --- PASS: TestNeedsMaintenanceExceedsMaxSnapshotsWithinRetention (0.00s) === RUN TestNeedsMaintenanceSkipsRefPinnedSnapshots --- PASS: TestNeedsMaintenanceSkipsRefPinnedSnapshots (0.00s) === RUN TestNeedsMaintenanceWithinLimits --- PASS: TestNeedsMaintenanceWithinLimits (0.00s) === RUN TestNeedsMaintenanceOldSnapshot --- PASS: TestNeedsMaintenanceOldSnapshot (0.00s) === RUN TestNeedsMaintenanceSingleSnapshot --- PASS: TestNeedsMaintenanceSingleSnapshot (0.00s) === RUN TestCapabilityAndDescriptor --- PASS: TestCapabilityAndDescriptor (0.00s) === RUN TestBuildMaintenanceProposal --- PASS: TestBuildMaintenanceProposal (0.00s) === RUN TestReadResourceGroupConfig --- PASS: TestReadResourceGroupConfig (0.00s) === RUN TestSelectTablesByResourceGroupRoundRobin --- PASS: TestSelectTablesByResourceGroupRoundRobin (0.00s) === RUN TestSelectTablesByResourceGroupCap --- PASS: TestSelectTablesByResourceGroupCap (0.00s) === RUN TestManifestRewritePathConsistency --- PASS: TestManifestRewritePathConsistency (0.02s) === RUN TestManifestRewriteNestedPathConsistency === RUN TestManifestRewriteNestedPathConsistency/nested_two_levels === RUN TestManifestRewriteNestedPathConsistency/nested_one_level --- PASS: TestManifestRewriteNestedPathConsistency (0.04s) --- PASS: TestManifestRewriteNestedPathConsistency/nested_two_levels (0.02s) --- PASS: TestManifestRewriteNestedPathConsistency/nested_one_level (0.02s) === RUN TestNormalizeIcebergPath === RUN TestNormalizeIcebergPath/relative_metadata_path === RUN TestNormalizeIcebergPath/relative_data_path === RUN TestNormalizeIcebergPath/S3_URL === RUN TestNormalizeIcebergPath/absolute_filer_path === RUN TestNormalizeIcebergPath/nested_data_path === RUN TestNormalizeIcebergPath/S3_URL_nested === RUN TestNormalizeIcebergPath/S3_URL_outside_the_catalog_path === RUN TestNormalizeIcebergPath/S3_URL_wins_over_a_stale_data_path === RUN TestNormalizeIcebergPath/other_bucket === RUN TestNormalizeIcebergPath/traversal_out_of_the_bucket --- PASS: TestNormalizeIcebergPath (0.00s) --- PASS: TestNormalizeIcebergPath/relative_metadata_path (0.00s) --- PASS: TestNormalizeIcebergPath/relative_data_path (0.00s) --- PASS: TestNormalizeIcebergPath/S3_URL (0.00s) --- PASS: TestNormalizeIcebergPath/absolute_filer_path (0.00s) --- PASS: TestNormalizeIcebergPath/nested_data_path (0.00s) --- PASS: TestNormalizeIcebergPath/S3_URL_nested (0.00s) --- PASS: TestNormalizeIcebergPath/S3_URL_outside_the_catalog_path (0.00s) --- PASS: TestNormalizeIcebergPath/S3_URL_wins_over_a_stale_data_path (0.00s) --- PASS: TestNormalizeIcebergPath/other_bucket (0.00s) --- PASS: TestNormalizeIcebergPath/traversal_out_of_the_bucket (0.00s) === RUN TestTableDataPath === RUN TestTableDataPath/catalog_path === RUN TestTableDataPath/location_outside_the_catalog_path === RUN TestTableDataPath/relative_location === RUN TestTableDataPath/missing_location === RUN TestTableDataPath/location_in_another_bucket --- PASS: TestTableDataPath (0.00s) --- PASS: TestTableDataPath/catalog_path (0.00s) --- PASS: TestTableDataPath/location_outside_the_catalog_path (0.00s) --- PASS: TestTableDataPath/relative_location (0.00s) --- PASS: TestTableDataPath/missing_location (0.00s) --- PASS: TestTableDataPath/location_in_another_bucket (0.00s) === RUN TestPartitionKey === RUN TestPartitionKey/empty_partition === RUN TestPartitionKey/nil_partition === RUN TestPartitionKey/single_field === RUN TestPartitionKey/multiple_fields_sorted --- PASS: TestPartitionKey (0.00s) --- PASS: TestPartitionKey/empty_partition (0.00s) --- PASS: TestPartitionKey/nil_partition (0.00s) --- PASS: TestPartitionKey/single_field (0.00s) --- PASS: TestPartitionKey/multiple_fields_sorted (0.00s) === RUN TestBuildCompactionBins --- PASS: TestBuildCompactionBins (0.00s) === RUN TestBuildCompactionBinsFiltersLargeFiles --- PASS: TestBuildCompactionBinsFiltersLargeFiles (0.00s) === RUN TestBuildCompactionBinsLowercaseParquetFormat --- PASS: TestBuildCompactionBinsLowercaseParquetFormat (0.00s) === RUN TestBuildCompactionBinsMinFilesThreshold --- PASS: TestBuildCompactionBinsMinFilesThreshold (0.00s) === RUN TestBuildCompactionBinsMultiplePartitions --- PASS: TestBuildCompactionBinsMultiplePartitions (0.00s) === RUN TestSplitOversizedBinRespectsTargetSize --- PASS: TestSplitOversizedBinRespectsTargetSize (0.00s) === RUN TestSplitOversizedBinDropsImpossibleRunts --- PASS: TestSplitOversizedBinDropsImpossibleRunts (0.00s) === RUN TestBuildCompactionBinsMultipleSpecs --- PASS: TestBuildCompactionBinsMultipleSpecs (0.00s) === RUN TestBuildCompactionBinsSingleSpec --- PASS: TestBuildCompactionBinsSingleSpec (0.00s) === RUN TestParseConfigApplyDeletes --- PASS: TestParseConfigApplyDeletes (0.00s) === RUN TestNormalizeDetectionConfigUsesSharedDefaults --- PASS: TestNormalizeDetectionConfigUsesSharedDefaults (0.00s) === RUN TestParseConfigRewriteStrategy --- PASS: TestParseConfigRewriteStrategy (0.00s) === RUN TestCollectPositionDeletes --- PASS: TestCollectPositionDeletes (0.03s) === RUN TestCollectEqualityDeletes --- PASS: TestCollectEqualityDeletes (0.03s) === RUN TestMergeParquetFilesWithPositionDeletes --- PASS: TestMergeParquetFilesWithPositionDeletes (0.03s) === RUN TestMergeParquetFilesWithEqualityDeletes --- PASS: TestMergeParquetFilesWithEqualityDeletes (0.04s) === RUN TestMergeParquetFilesDictionaryEncodedInput --- PASS: TestMergeParquetFilesDictionaryEncodedInput (0.06s) === RUN TestDetectNilRequest --- PASS: TestDetectNilRequest (0.00s) === RUN TestExecuteNilRequest --- PASS: TestExecuteNilRequest (0.00s) === RUN TestResolveTableConfigNoProperties --- PASS: TestResolveTableConfigNoProperties (0.00s) === RUN TestResolveTableConfigPropertiesWin --- PASS: TestResolveTableConfigPropertiesWin (0.00s) === RUN TestResolveTableConfigSubHourRetention --- PASS: TestResolveTableConfigSubHourRetention (0.00s) === RUN TestResolveTableConfigIgnoresUnusableValues === RUN TestResolveTableConfigIgnoresUnusableValues/zero === RUN TestResolveTableConfigIgnoresUnusableValues/negative === RUN TestResolveTableConfigIgnoresUnusableValues/overflow === RUN TestResolveTableConfigIgnoresUnusableValues/not_a_number === RUN TestResolveTableConfigIgnoresUnusableValues/empty --- PASS: TestResolveTableConfigIgnoresUnusableValues (0.00s) --- PASS: TestResolveTableConfigIgnoresUnusableValues/zero (0.00s) --- PASS: TestResolveTableConfigIgnoresUnusableValues/negative (0.00s) --- PASS: TestResolveTableConfigIgnoresUnusableValues/overflow (0.00s) --- PASS: TestResolveTableConfigIgnoresUnusableValues/not_a_number (0.00s) --- PASS: TestResolveTableConfigIgnoresUnusableValues/empty (0.00s) === RUN TestResolveTableConfigTrimsWhitespace --- PASS: TestResolveTableConfigTrimsWhitespace (0.00s) === RUN TestResolveTableConfigLeavesOtherFields --- PASS: TestResolveTableConfigLeavesOtherFields (0.00s) === RUN TestApplyThresholdDefaultsClampsOrphanCutoff --- PASS: TestApplyThresholdDefaultsClampsOrphanCutoff (0.00s) === RUN TestResolveTableConfigAppliesMaintenanceConfig --- PASS: TestResolveTableConfigAppliesMaintenanceConfig (0.00s) === RUN TestResolveTableConfigPropertyBeatsMaintenanceConfig --- PASS: TestResolveTableConfigPropertyBeatsMaintenanceConfig (0.00s) === RUN TestResolveTableConfigIgnoresDisabledSettings --- PASS: TestResolveTableConfigIgnoresDisabledSettings (0.00s) === RUN TestFilterDisabledOperations --- PASS: TestFilterDisabledOperations (0.00s) === RUN TestMergeMaintenanceConfigurationTableWins --- PASS: TestMergeMaintenanceConfigurationTableWins (0.00s) === RUN TestParseMaintenanceConfiguration --- PASS: TestParseMaintenanceConfiguration (0.00s) === RUN TestResolveTableConfigClampsUnreferencedDays --- PASS: TestResolveTableConfigClampsUnreferencedDays (0.00s) === RUN TestResolveTableConfigAddsNonCurrentWindow --- PASS: TestResolveTableConfigAddsNonCurrentWindow (0.00s) === RUN TestBuildJobStatus --- PASS: TestBuildJobStatus (0.00s) === RUN TestBuildJobStatusFailureWins --- PASS: TestBuildJobStatusFailureWins (0.00s) === RUN TestBuildJobStatusEmpty --- PASS: TestBuildJobStatusEmpty (0.00s) === RUN TestResolveTableConfigAppliesCompactionStrategy --- PASS: TestResolveTableConfigAppliesCompactionStrategy (0.00s) === RUN TestResolveCompactionRewritePlanAuto --- PASS: TestResolveCompactionRewritePlanAuto (0.00s) === RUN TestValidateWhereOperations --- PASS: TestValidateWhereOperations (0.00s) === RUN TestSplitWhereConjunctionQuoteAware --- PASS: TestSplitWhereConjunctionQuoteAware (0.00s) === RUN TestPartitionPredicateMatchesUsesPartitionFieldIDs --- PASS: TestPartitionPredicateMatchesUsesPartitionFieldIDs (0.00s) === RUN TestCompactDataFilesWhereFilter --- PASS: TestCompactDataFilesWhereFilter (1.05s) PASS ok github.com/seaweedfs/seaweedfs/weed/worker/tasks/iceberg 48.695s === RUN TestBuildLimiterFromClusterContext_NilCC --- PASS: TestBuildLimiterFromClusterContext_NilCC (0.02s) === RUN TestBuildLimiterFromClusterContext_NoMetadata --- PASS: TestBuildLimiterFromClusterContext_NoMetadata (0.00s) === RUN TestBuildLimiterFromClusterContext_MissingRateKey --- PASS: TestBuildLimiterFromClusterContext_MissingRateKey (0.00s) === RUN TestBuildLimiterFromClusterContext_NonPositiveRate --- PASS: TestBuildLimiterFromClusterContext_NonPositiveRate (0.02s) === RUN TestBuildLimiterFromClusterContext_PositiveRateBuildsLimiter --- PASS: TestBuildLimiterFromClusterContext_PositiveRateBuildsLimiter (0.00s) === RUN TestBuildLimiterFromClusterContext_BurstMissingDefaultsTo2xRate --- PASS: TestBuildLimiterFromClusterContext_BurstMissingDefaultsTo2xRate (0.00s) === RUN TestBuildLimiterFromClusterContext_TinyRateClampsBurstToOne --- PASS: TestBuildLimiterFromClusterContext_TinyRateClampsBurstToOne (0.00s) === RUN TestParseConfigDefaults --- PASS: TestParseConfigDefaults (0.00s) === RUN TestParseConfigIgnoresWorkerValues --- PASS: TestParseConfigIgnoresWorkerValues (0.00s) === RUN TestParseConfigMetaLogRetentionDefaultsToZero --- PASS: TestParseConfigMetaLogRetentionDefaultsToZero (0.00s) === RUN TestParseConfigMetaLogRetentionDaysConvertsToDuration --- PASS: TestParseConfigMetaLogRetentionDaysConvertsToDuration (0.00s) === RUN TestParseConfigMetaLogRetentionNegativeStaysZero --- PASS: TestParseConfigMetaLogRetentionNegativeStaysZero (0.00s) === RUN TestParseConfigWalkerIntervalDefaultsToZero --- PASS: TestParseConfigWalkerIntervalDefaultsToZero (0.00s) === RUN TestParseConfigWalkerIntervalMinutesConvertsToDuration --- PASS: TestParseConfigWalkerIntervalMinutesConvertsToDuration (0.00s) === RUN TestParseConfigWalkerIntervalNegativeStaysZero --- PASS: TestParseConfigWalkerIntervalNegativeStaysZero (0.00s) === RUN TestClusterS3Endpoints_NilContext --- PASS: TestClusterS3Endpoints_NilContext (0.00s) === RUN TestClusterS3Endpoints_EmptyList --- PASS: TestClusterS3Endpoints_EmptyList (0.00s) === RUN TestClusterS3Endpoints_FiltersEmptyEntries --- PASS: TestClusterS3Endpoints_FiltersEmptyEntries (0.00s) === RUN TestClusterS3Endpoints_AllValid --- PASS: TestClusterS3Endpoints_AllValid (0.00s) === RUN TestReadString_MissingKeyReturnsFallback --- PASS: TestReadString_MissingKeyReturnsFallback (0.02s) === RUN TestReadString_NilValueReturnsFallback --- PASS: TestReadString_NilValueReturnsFallback (0.00s) === RUN TestReadString_WrongKindReturnsFallback --- PASS: TestReadString_WrongKindReturnsFallback (0.00s) === RUN TestReadString_StringValueReturned --- PASS: TestReadString_StringValueReturned (0.00s) === RUN TestCapability_AdvertisesJobType --- PASS: TestCapability_AdvertisesJobType (0.00s) === RUN TestDetect_NilRequestErrors --- PASS: TestDetect_NilRequestErrors (0.00s) === RUN TestDetect_NilSenderErrors --- PASS: TestDetect_NilSenderErrors (0.00s) === RUN TestDetect_WrongJobTypeErrors --- PASS: TestDetect_WrongJobTypeErrors (0.00s) === RUN TestDetect_NoS3EndpointsCompletesWithSkipActivity --- PASS: TestDetect_NoS3EndpointsCompletesWithSkipActivity (0.00s) === RUN TestDetect_NoFilerAddressesCompletesWithSkipActivity --- PASS: TestDetect_NoFilerAddressesCompletesWithSkipActivity (0.00s) === RUN TestDetect_HappyPathProposesOneJobWithFirstFilerAddress --- PASS: TestDetect_HappyPathProposesOneJobWithFirstFilerAddress (0.02s) === RUN TestDetect_EmptyJobTypeAccepted --- PASS: TestDetect_EmptyJobTypeAccepted (0.00s) === RUN TestDetect_PropagatesProposalsSendError --- PASS: TestDetect_PropagatesProposalsSendError (0.00s) === RUN TestDetect_PropagatesCompleteSendError --- PASS: TestDetect_PropagatesCompleteSendError (0.00s) === RUN TestDescriptor_BasicShape --- PASS: TestDescriptor_BasicShape (0.00s) === RUN TestDescriptor_AdminConfigFormHasNoWorkersField --- PASS: TestDescriptor_AdminConfigFormHasNoWorkersField (0.00s) === RUN TestDescriptor_WorkerConfigFormIsAbsent --- PASS: TestDescriptor_WorkerConfigFormIsAbsent (0.00s) === RUN TestDescriptor_AdminRuntimeDefaultsHaveNoTimeoutInPractice --- PASS: TestDescriptor_AdminRuntimeDefaultsHaveNoTimeoutInPractice (0.00s) === RUN TestDescriptor_AdminRuntimeDefaultsDailyCadence --- PASS: TestDescriptor_AdminRuntimeDefaultsDailyCadence (0.00s) === RUN TestDescriptor_AdminRuntimeDefaultsEnabledByDefault --- PASS: TestDescriptor_AdminRuntimeDefaultsEnabledByDefault (0.00s) === RUN TestExecute_NilRequestErrors --- PASS: TestExecute_NilRequestErrors (0.00s) === RUN TestExecute_NilJobErrors --- PASS: TestExecute_NilJobErrors (0.00s) === RUN TestExecute_NilSenderErrors --- PASS: TestExecute_NilSenderErrors (0.00s) === RUN TestExecute_WrongJobTypeErrors --- PASS: TestExecute_WrongJobTypeErrors (0.00s) === RUN TestExecute_NoS3EndpointsErrors --- PASS: TestExecute_NoS3EndpointsErrors (0.00s) === RUN TestExecute_MissingFilerAddressErrors --- PASS: TestExecute_MissingFilerAddressErrors (0.00s) === RUN TestExecute_EmptyJobTypeAccepted --- PASS: TestExecute_EmptyJobTypeAccepted (0.00s) === RUN TestSendSuccessCompletion_EmitsCanonicalCompletion --- PASS: TestSendSuccessCompletion_EmitsCanonicalCompletion (0.00s) === RUN TestSendSuccessCompletion_PropagatesSendError --- PASS: TestSendSuccessCompletion_PropagatesSendError (0.00s) === RUN TestLookupBucketsPath_PropagatesGRPCError --- PASS: TestLookupBucketsPath_PropagatesGRPCError (0.00s) === RUN TestLookupBucketsPath_UsesConfiguredDirBuckets --- PASS: TestLookupBucketsPath_UsesConfiguredDirBuckets (0.02s) === RUN TestLookupBucketsPath_EmptyDirBucketsFallsBackToDefault --- PASS: TestLookupBucketsPath_EmptyDirBucketsFallsBackToDefault (0.00s) PASS ok github.com/seaweedfs/seaweedfs/weed/worker/tasks/s3_lifecycle 15.794s ? github.com/seaweedfs/seaweedfs/weed/worker/tasks/util [no test files] === RUN TestDecodeVacuumTaskParamsFromPayload --- PASS: TestDecodeVacuumTaskParamsFromPayload (0.10s) === RUN TestDecodeVacuumTaskParamsFallback --- PASS: TestDecodeVacuumTaskParamsFallback (0.00s) === RUN TestDeriveVacuumConfigAllowsZeroValues --- PASS: TestDeriveVacuumConfigAllowsZeroValues (0.00s) === RUN TestMasterAddressCandidates --- PASS: TestMasterAddressCandidates (0.02s) === RUN TestShouldSkipDetectionByInterval --- PASS: TestShouldSkipDetectionByInterval (0.00s) === RUN TestVacuumHandlerRejectsUnsupportedJobType --- PASS: TestVacuumHandlerRejectsUnsupportedJobType (0.00s) === RUN TestBuildExecutorActivity --- PASS: TestBuildExecutorActivity (0.00s) === RUN TestEmitVacuumDetectionDecisionTraceNoTasks --- PASS: TestEmitVacuumDetectionDecisionTraceNoTasks (0.00s) === RUN TestVacuumDescriptorHasNewFields --- PASS: TestVacuumDescriptorHasNewFields (0.02s) === RUN TestVacuumFiltersVolumeState === RUN TestVacuumFiltersVolumeState/ALL_returns_all === RUN TestVacuumFiltersVolumeState/ACTIVE_returns_writable === RUN TestVacuumFiltersVolumeState/FULL_returns_read-only --- PASS: TestVacuumFiltersVolumeState (0.00s) --- PASS: TestVacuumFiltersVolumeState/ALL_returns_all (0.00s) --- PASS: TestVacuumFiltersVolumeState/ACTIVE_returns_writable (0.00s) --- PASS: TestVacuumFiltersVolumeState/FULL_returns_read-only (0.00s) === RUN TestVacuumFiltersLocation === RUN TestVacuumFiltersLocation/dc_filter === RUN TestVacuumFiltersLocation/rack_filter === RUN TestVacuumFiltersLocation/node_filter === RUN TestVacuumFiltersLocation/dc_+_rack --- PASS: TestVacuumFiltersLocation (0.00s) --- PASS: TestVacuumFiltersLocation/dc_filter (0.00s) --- PASS: TestVacuumFiltersLocation/rack_filter (0.00s) --- PASS: TestVacuumFiltersLocation/node_filter (0.00s) --- PASS: TestVacuumFiltersLocation/dc_+_rack (0.00s) PASS ok github.com/seaweedfs/seaweedfs/weed/worker/tasks/vacuum 16.308s ? github.com/seaweedfs/seaweedfs/weed/worker/types [no test files] ? github.com/seaweedfs/seaweedfs/weed/worker/types/base [no test files] FAIL ==> ERROR: A failure occurred in check().  Aborting... ==> ERROR: Build failed, check /var/lib/archbuild/extra-riscv64/felix-2/build receiving incremental file list seaweedfs-4.44-1-riscv64-build.log seaweedfs-4.44-1-riscv64-check.log seaweedfs-4.44-1-riscv64-prepare.log sent 81 bytes received 307,208 bytes 122,915.60 bytes/sec total size is 2,453,441 speedup is 7.98