Fix snapshots garbage collection by nvazquez · Pull Request #4188 · apache/cloudstack

nvazquez · 2020-06-29T16:11:26Z

Description

Cleanup orphan entries for primary storage on table snapshot_store_ref

Fixes: #4018 - Problem 1

Types of changes

Breaking change (fix or feature that would cause existing functionality to change)
New feature (non-breaking change which adds functionality)
Bug fix (non-breaking change which fixes an issue)
Enhancement (improves an existing feature and functionality)
Cleanup (Code refactoring and cleanup, that may add test cases)

Screenshots (if appropriate):

How Has This Been Tested?

nvazquez · 2020-06-30T13:30:36Z

@blueorangutan package

blueorangutan · 2020-06-30T13:31:44Z

@nvazquez a Jenkins job has been kicked to build packages. I'll keep you posted as I make progress.

blueorangutan · 2020-06-30T13:58:04Z

Packaging result: ✔centos7 ✔debian. JID-1493

nvazquez · 2020-07-04T19:14:08Z

@blueorangutan test matrix

blueorangutan · 2020-07-04T19:14:58Z

@nvazquez a Trillian-Jenkins matrix job (centos7 mgmt + xs71, centos7 mgmt + vmware67, centos7 mgmt + kvmcentos7) has been kicked to run smoke tests

blueorangutan · 2020-07-05T08:03:18Z

Trillian test result (tid-1988)
Environment: vmware-67u3 (x2), Advanced Networking with Mgmt server 7
Total time taken: 44639 seconds
Marvin logs: https://github.com/blueorangutan/acs-prs/releases/download/trillian/pr4188-t1988-vmware-67u3.zip
Intermittent failure detected: /marvin/tests/smoke/test_ssvm.py
Intermittent failure detected: /marvin/tests/smoke/test_vpc_redundant.py
Smoke tests completed. 78 look OK, 1 have error(s)
Only failed tests results shown below:

Test	Result	Time (s)	Test File
test_05_rvpc_multi_tiers	`Failure`	220.55	test_vpc_redundant.py
test_05_rvpc_multi_tiers	`Error`	220.57	test_vpc_redundant.py
ContextSuite context=TestVPCRedundancy>:teardown	`Error`	220.58	test_vpc_redundant.py

blueorangutan · 2020-07-05T09:40:03Z

Trillian test result (tid-1987)
Environment: kvm-centos7 (x2), Advanced Networking with Mgmt server 7
Total time taken: 50139 seconds
Marvin logs: https://github.com/blueorangutan/acs-prs/releases/download/trillian/pr4188-t1987-kvm-centos7.zip
Intermittent failure detected: /marvin/tests/smoke/test_kubernetes_clusters.py
Intermittent failure detected: /marvin/tests/smoke/test_vpc_redundant.py
Smoke tests completed. 82 look OK, 1 have error(s)
Only failed tests results shown below:

Test	Result	Time (s)	Test File
test_04_deploy_and_upgrade_kubernetes_cluster	`Failure`	873.79	test_kubernetes_clusters.py

blueorangutan · 2020-07-05T09:45:36Z

Trillian test result (tid-1986)
Environment: xenserver-71 (x2), Advanced Networking with Mgmt server 7
Total time taken: 50672 seconds
Marvin logs: https://github.com/blueorangutan/acs-prs/releases/download/trillian/pr4188-t1986-xenserver-71.zip
Intermittent failure detected: /marvin/tests/smoke/test_scale_vm.py
Intermittent failure detected: /marvin/tests/smoke/test_vpc_redundant.py
Smoke tests completed. 82 look OK, 1 have error(s)
Only failed tests results shown below:

Test	Result	Time (s)	Test File
test_01_scale_vm	`Failure`	14.40	test_scale_vm.py

borisstoyanov

LGTM. manually checked after GC passed there's no records in the table.

nvazquez · 2020-07-09T20:38:56Z

Thanks @borisstoyanov, as its a bug completing the fix on #3969 do you agree targeting this fix to branch 4.13 and forward merge to 4.14 and master? cc @rhtyd @DaanHoogland

nvazquez · 2020-07-14T12:52:14Z

@blueorangutan package

blueorangutan · 2020-07-14T12:52:46Z

@nvazquez a Jenkins job has been kicked to build packages. I'll keep you posted as I make progress.

blueorangutan · 2020-07-14T13:29:27Z

Packaging result: ✔centos7 ✔debian. JID-1578

nvazquez · 2020-07-14T13:49:15Z

@blueorangutan test

blueorangutan · 2020-07-14T13:50:40Z

@nvazquez a Trillian-Jenkins test job (centos7 mgmt + kvm-centos7) has been kicked to run smoke tests

yadvr · 2020-07-15T10:27:03Z

@blueorangutan test

blueorangutan · 2020-07-15T10:27:50Z

@rhtyd a Trillian-Jenkins test job (centos7 mgmt + kvm-centos7) has been kicked to run smoke tests

blueorangutan · 2020-07-16T04:07:26Z

Trillian test result (tid-2128)
Environment: kvm-centos7 (x2), Advanced Networking with Mgmt server 7
Total time taken: 61821 seconds
Marvin logs: https://github.com/blueorangutan/acs-prs/releases/download/trillian/pr4188-t2128-kvm-centos7.zip
Intermittent failure detected: /marvin/tests/smoke/test_internal_lb.py
Intermittent failure detected: /marvin/tests/smoke/test_privategw_acl.py
Intermittent failure detected: /marvin/tests/smoke/test_public_ip_range.py
Intermittent failure detected: /marvin/tests/smoke/test_reset_vm_on_reboot.py
Intermittent failure detected: /marvin/tests/smoke/test_resource_accounting.py
Intermittent failure detected: /marvin/tests/smoke/test_router_dhcphosts.py
Intermittent failure detected: /marvin/tests/smoke/test_router_dns.py
Intermittent failure detected: /marvin/tests/smoke/test_router_dnsservice.py
Intermittent failure detected: /marvin/tests/smoke/test_routers_iptables_default_policy.py
Intermittent failure detected: /marvin/tests/smoke/test_routers_network_ops.py
Intermittent failure detected: /marvin/tests/smoke/test_routers.py
Intermittent failure detected: /marvin/tests/smoke/test_secondary_storage.py
Intermittent failure detected: /marvin/tests/smoke/test_service_offerings.py
Intermittent failure detected: /marvin/tests/smoke/test_snapshots.py
Intermittent failure detected: /marvin/tests/smoke/test_ssvm.py
Intermittent failure detected: /marvin/tests/smoke/test_templates.py
Intermittent failure detected: /marvin/tests/smoke/test_usage.py
Intermittent failure detected: /marvin/tests/smoke/test_vm_life_cycle.py
Intermittent failure detected: /marvin/tests/smoke/test_vm_snapshots.py
Intermittent failure detected: /marvin/tests/smoke/test_volumes.py
Intermittent failure detected: /marvin/tests/smoke/test_vpc_redundant.py
Intermittent failure detected: /marvin/tests/smoke/test_vpc_router_nics.py
Intermittent failure detected: /marvin/tests/smoke/test_vpc_vpn.py
Intermittent failure detected: /marvin/tests/smoke/test_hostha_kvm.py
Smoke tests completed. 55 look OK, 22 have error(s)
Only failed tests results shown below:

Test	Result	Time (s)	Test File
test_02_vpc_privategw_static_routes	`Failure`	202.38	test_privategw_acl.py
test_03_vpc_privategw_restart_vpc_cleanup	`Failure`	208.31	test_privategw_acl.py
test_04_rvpc_privategw_static_routes	`Failure`	265.79	test_privategw_acl.py
ContextSuite context=TestResetVmOnReboot>:setup	`Error`	0.00	test_reset_vm_on_reboot.py
ContextSuite context=TestRAMCPUResourceAccounting>:setup	`Error`	0.00	test_resource_accounting.py
ContextSuite context=TestRouterDHCPHosts>:setup	`Error`	0.00	test_router_dhcphosts.py
ContextSuite context=TestRouterDHCPOpts>:setup	`Error`	0.00	test_router_dhcphosts.py
ContextSuite context=TestRouterDns>:setup	`Error`	0.00	test_router_dns.py
test_01_sys_vm_start	`Failure`	0.07	test_secondary_storage.py
ContextSuite context=TestRouterDnsService>:setup	`Error`	0.00	test_router_dnsservice.py
ContextSuite context=TestRouterIpTablesPolicies>:setup	`Error`	0.00	test_routers_iptables_default_policy.py
ContextSuite context=TestVPCIpTablesPolicies>:setup	`Error`	0.00	test_routers_iptables_default_policy.py
ContextSuite context=TestIsolatedNetworks>:setup	`Error`	0.00	test_routers_network_ops.py
ContextSuite context=TestRedundantIsolateNetworks>:setup	`Error`	0.00	test_routers_network_ops.py
ContextSuite context=TestRouterServices>:setup	`Error`	0.00	test_routers.py
ContextSuite context=TestCpuCapServiceOfferings>:setup	`Error`	0.00	test_service_offerings.py
ContextSuite context=TestServiceOfferings>:setup	`Error`	0.15	test_service_offerings.py
ContextSuite context=TestSnapshotRootDisk>:setup	`Error`	0.00	test_snapshots.py
test_01_list_sec_storage_vm	`Failure`	0.03	test_ssvm.py
test_02_list_cpvm_vm	`Failure`	0.03	test_ssvm.py
test_03_ssvm_internals	`Failure`	0.03	test_ssvm.py
test_04_cpvm_internals	`Failure`	0.02	test_ssvm.py
test_05_stop_ssvm	`Failure`	0.03	test_ssvm.py
test_06_stop_cpvm	`Failure`	0.03	test_ssvm.py
test_07_reboot_ssvm	`Failure`	0.03	test_ssvm.py
test_08_reboot_cpvm	`Failure`	0.03	test_ssvm.py
test_09_destroy_ssvm	`Failure`	0.03	test_ssvm.py
test_10_destroy_cpvm	`Failure`	0.03	test_ssvm.py
test_02_create_template_with_checksum_sha1	`Error`	65.34	test_templates.py
test_03_create_template_with_checksum_sha256	`Error`	65.37	test_templates.py
test_04_create_template_with_checksum_md5	`Error`	65.39	test_templates.py
test_05_create_template_with_no_checksum	`Error`	65.39	test_templates.py
test_02_deploy_vm_from_direct_download_template	`Error`	1.19	test_templates.py
test_03_deploy_vm_wrong_checksum	`Error`	1.25	test_templates.py
ContextSuite context=TestTemplates>:setup	`Error`	16.65	test_templates.py
ContextSuite context=TestISOUsage>:setup	`Error`	0.00	test_usage.py
ContextSuite context=TestLBRuleUsage>:setup	`Error`	0.00	test_usage.py
ContextSuite context=TestNatRuleUsage>:setup	`Error`	0.00	test_usage.py
ContextSuite context=TestPublicIPUsage>:setup	`Error`	0.00	test_usage.py
ContextSuite context=TestSnapshotUsage>:setup	`Error`	0.00	test_usage.py
ContextSuite context=TestVmUsage>:setup	`Error`	0.00	test_usage.py
ContextSuite context=TestVolumeUsage>:setup	`Error`	0.00	test_usage.py
ContextSuite context=TestVpnUsage>:setup	`Error`	0.00	test_usage.py
ContextSuite context=Test01DeployVM>:setup	`Error`	0.00	test_vm_life_cycle.py
ContextSuite context=Test02VMLifeCycle>:setup	`Error`	0.00	test_vm_life_cycle.py
test_14_secure_to_secure_vm_migration	`Error`	11.28	test_vm_life_cycle.py
test_15_secured_to_nonsecured_vm_migration	`Error`	74.11	test_vm_life_cycle.py
test_16_nonsecured_to_secured_vm_migration	`Error`	1.17	test_vm_life_cycle.py
ContextSuite context=TestVmSnapshot>:setup	`Error`	1.57	test_vm_snapshots.py
ContextSuite context=TestCreateVolume>:setup	`Error`	0.00	test_volumes.py
ContextSuite context=TestVolumes>:setup	`Error`	0.00	test_volumes.py
ContextSuite context=TestVPCRedundancy>:setup	`Error`	0.00	test_vpc_redundant.py
ContextSuite context=TestVPCNics>:setup	`Error`	0.00	test_vpc_router_nics.py
ContextSuite context=TestRVPCSite2SiteVpn>:setup	`Error`	0.00	test_vpc_vpn.py
ContextSuite context=TestVPCSite2SiteVPNMultipleOptions>:setup	`Error`	0.00	test_vpc_vpn.py
ContextSuite context=TestVpcRemoteAccessVpn>:setup	`Error`	0.00	test_vpc_vpn.py
ContextSuite context=TestVpcSite2SiteVpn>:setup	`Error`	0.00	test_vpc_vpn.py
test_disable_oobm_ha_state_ineligible	`Error`	1511.59	test_hostha_kvm.py

nvazquez · 2020-07-16T13:32:50Z

@blueorangutan package

blueorangutan · 2020-07-16T13:33:54Z

@nvazquez a Jenkins job has been kicked to build packages. I'll keep you posted as I make progress.

blueorangutan · 2020-07-16T13:54:55Z

Packaging result: ✔centos7 ✔debian. JID-1585

nvazquez · 2020-07-16T14:32:58Z

@blueorangutan test

blueorangutan · 2020-07-16T14:33:50Z

@nvazquez a Trillian-Jenkins test job (centos7 mgmt + kvm-centos7) has been kicked to run smoke tests

blueorangutan · 2020-07-16T23:23:52Z

Trillian test result (tid-2136)
Environment: kvm-centos7 (x2), Advanced Networking with Mgmt server 7
Total time taken: 30014 seconds
Marvin logs: https://github.com/blueorangutan/acs-prs/releases/download/trillian/pr4188-t2136-kvm-centos7.zip
Intermittent failure detected: /marvin/tests/smoke/test_privategw_acl.py
Smoke tests completed. 76 look OK, 1 have error(s)
Only failed tests results shown below:

Test	Result	Time (s)	Test File
test_02_vpc_privategw_static_routes	`Failure`	182.99	test_privategw_acl.py
test_03_vpc_privategw_restart_vpc_cleanup	`Failure`	183.11	test_privategw_acl.py
test_04_rvpc_privategw_static_routes	`Failure`	250.29	test_privategw_acl.py

nvazquez · 2020-07-17T15:26:59Z

@rhtyd can you please review this one?

GabrielBrascher

LGTM based on code review. I did not test it, though.

nvazquez · 2020-07-18T17:12:36Z

Thanks @GabrielBrascher, @borisstoyanov tested and approved as well. Merging it

* Cleanup orphan entries from snapshot store ref for primary storage * Add debug message

nvazquez changed the title ~~WIP: Fix snapshots garbage collection~~ Fix snapshots garbage collection Jul 4, 2020

nvazquez added the type:cleanup label Jul 4, 2020

nvazquez added this to the 4.15.0.0 milestone Jul 4, 2020

nvazquez requested review from andrijapanicsb, borisstoyanov and yadvr July 4, 2020 19:32

borisstoyanov approved these changes Jul 7, 2020

View reviewed changes

nvazquez changed the base branch from master to 4.14 July 9, 2020 20:33

nvazquez changed the base branch from 4.14 to master July 9, 2020 20:34

nvazquez marked this pull request as ready for review July 9, 2020 20:35

svenvogel added type:bug component:primary-storage labels Jul 10, 2020

nvazquez added 2 commits July 14, 2020 09:49

Cleanup orphan entries from snapshot store ref for primary storage

5454525

Add debug message

f04bf83

nvazquez force-pushed the fixgcsnaps branch from d2d3909 to f04bf83 Compare July 14, 2020 12:50

nvazquez changed the base branch from master to 4.13 July 14, 2020 12:50

apache deleted a comment from blueorangutan Jul 15, 2020

GabrielBrascher approved these changes Jul 17, 2020

View reviewed changes

nvazquez merged commit f843c53 into apache:4.13 Jul 18, 2020

nvazquez deleted the fixgcsnaps branch July 18, 2020 17:13

shwstppr pushed a commit to shapeblue/cloudstack that referenced this pull request Jul 20, 2020

Fix snapshots garbage collection (apache#4188)

395d33e

* Cleanup orphan entries from snapshot store ref for primary storage * Add debug message

Pearl1594 pushed a commit to shapeblue/cloudstack that referenced this pull request Jul 27, 2020

Fix snapshots garbage collection (apache#4188)

615a81c

* Cleanup orphan entries from snapshot store ref for primary storage * Add debug message

nvazquez mentioned this pull request Aug 4, 2020

Snapshots GC from DB and XenServer Primary Storage garbage #4018

Closed

PaulAngus removed the type:cleanup label Nov 8, 2020

DaanHoogland added the Severity:Major label Nov 10, 2020

Conversation

nvazquez commented Jun 29, 2020 • edited by PaulAngus Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description

Types of changes

Screenshots (if appropriate):

How Has This Been Tested?

Uh oh!

nvazquez commented Jun 30, 2020

Uh oh!

blueorangutan commented Jun 30, 2020

Uh oh!

blueorangutan commented Jun 30, 2020

Uh oh!

nvazquez commented Jul 4, 2020

Uh oh!

blueorangutan commented Jul 4, 2020

Uh oh!

blueorangutan commented Jul 5, 2020

Uh oh!

blueorangutan commented Jul 5, 2020

Uh oh!

blueorangutan commented Jul 5, 2020

Uh oh!

borisstoyanov left a comment

Choose a reason for hiding this comment

Uh oh!

nvazquez commented Jul 9, 2020

Uh oh!

nvazquez commented Jul 14, 2020

Uh oh!

blueorangutan commented Jul 14, 2020

Uh oh!

blueorangutan commented Jul 14, 2020

Uh oh!

nvazquez commented Jul 14, 2020

Uh oh!

blueorangutan commented Jul 14, 2020

Uh oh!

yadvr commented Jul 15, 2020

Uh oh!

blueorangutan commented Jul 15, 2020

Uh oh!

blueorangutan commented Jul 16, 2020

Uh oh!

nvazquez commented Jul 16, 2020

Uh oh!

blueorangutan commented Jul 16, 2020

Uh oh!

blueorangutan commented Jul 16, 2020

Uh oh!

nvazquez commented Jul 16, 2020

Uh oh!

blueorangutan commented Jul 16, 2020

Uh oh!

blueorangutan commented Jul 16, 2020

Uh oh!

nvazquez commented Jul 17, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

GabrielBrascher left a comment

Choose a reason for hiding this comment

Uh oh!

nvazquez commented Jul 18, 2020

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

8 participants

nvazquez commented Jun 29, 2020 •

edited by PaulAngus

Loading

nvazquez commented Jul 17, 2020 •

edited

Loading