RvR: Set up metadata/password/dhcp server on gateway IP instead of guest IP in RVR by ustcweizhou · Pull Request #3477 · apache/cloudstack

ustcweizhou · 2019-07-09T09:23:39Z

Description

When we create a vm in the network with redundant VRs, the lease file in the vm (for example /var/lib/dhcp/dhclient.eth0.leases) shows the dhcp-server-identifier is the guest ip (not vip/gateway) of master VR. That's the ip ipaddress where the vm fetch password and metadata from.
if we stop the master VR (then backup will be master) or restart the network with cleanup (VRs will be created), the guest ip of master VR changes so vm are not able to get metadata/ssh-key using the ips in dhcp lease file.

Setting up metadata/password/dhcp server on gateway instead of guest IP in redundant VRs will fix the issues.

FIxes: #3409

Types of changes

Breaking change (fix or feature that would cause existing functionality to change)
New feature (non-breaking change which adds functionality)
Bug fix (non-breaking change which fixes an issue)
Enhancement (improves an existing feature and functionality)
Cleanup (Code refactoring and cleanup, that may add test cases)

Screenshots (if appropriate):

How Has This Been Tested?

ustcweizhou · 2019-07-09T09:25:30Z

@rhtyd Here is the PR for issue #3409
I will not be able to respond to your comments in time because I will be on holiday in the coming weeks.

yadvr · 2019-07-09T09:32:30Z

Thanks @ustcweizhou I'll help review and test, if you're unavailable may extend and address any review comments myself.

yadvr · 2019-07-09T09:33:14Z

systemvm/debian/etc/apache2/vhost.template

@@ -1,4 +1,4 @@
-<VirtualHost 10.1.1.1:80>
+<VirtualHost 10.1.1.1:8180>


Why change the port?

these lines will be replaced with gateway IP and guest IP in CsApp.py
If we use :80 and :443 in this template, the first time it is ok, after that the configuration file gets mess up
if we use :8180 and :8443, the configuration file will be changed only once.

@ustcweizhou Thanks for explaining. I'll test it.

yadvr · 2019-07-09T09:33:34Z

systemvm/debian/etc/apache2/vhost.template


 <IfModule mod_ssl.c>
-<VirtualHost 10.1.1.1:443>
+<VirtualHost 10.1.1.1:8443>


Same as above - any reason to change the port or did it come from your internal branch?

yadvr · 2019-07-09T09:34:10Z

@blueorangutan package

blueorangutan · 2019-07-09T09:35:05Z

@rhtyd a Jenkins job has been kicked to build packages. I'll keep you posted as I make progress.

blueorangutan · 2019-07-09T09:57:31Z

Packaging result: ✔centos6 ✔centos7 ✔debian. JID-105

yadvr · 2019-07-09T09:59:22Z

@blueorangutan test

blueorangutan · 2019-07-09T10:00:06Z

@rhtyd a Trillian-Jenkins test job (centos7 mgmt + kvm-centos7) has been kicked to run smoke tests

yadvr · 2019-07-10T12:16:58Z

@blueorangutan test

blueorangutan · 2019-07-10T12:17:09Z

@rhtyd a Trillian-Jenkins test job (centos7 mgmt + kvm-centos7) has been kicked to run smoke tests

yadvr · 2019-07-10T15:37:22Z

Looks like on 4.11 branch there is some issue, I'll kick tests against master.
@blueorangutan package

blueorangutan · 2019-07-10T15:38:10Z

@rhtyd a Jenkins job has been kicked to build packages. I'll keep you posted as I make progress.

blueorangutan · 2019-07-10T15:58:07Z

Packaging result: ✔centos6 ✔centos7 ✔debian. JID-111

yadvr · 2019-07-10T16:01:10Z

@blueorangutan test

blueorangutan · 2019-07-10T16:02:09Z

@rhtyd a Trillian-Jenkins test job (centos7 mgmt + kvm-centos7) has been kicked to run smoke tests

blueorangutan · 2019-07-11T04:10:27Z

Trillian test result (tid-139)
Environment: kvm-centos7 (x2), Advanced Networking with Mgmt server 7
Total time taken: 41924 seconds
Marvin logs: https://github.com/blueorangutan/acs-prs/releases/download/trillian/pr3477-t139-kvm-centos7.zip
Intermittent failure detected: /marvin/tests/smoke/test_accounts.py
Intermittent failure detected: /marvin/tests/smoke/test_internal_lb.py
Intermittent failure detected: /marvin/tests/smoke/test_iso.py
Intermittent failure detected: /marvin/tests/smoke/test_templates.py
Intermittent failure detected: /marvin/tests/smoke/test_volumes.py
Intermittent failure detected: /marvin/tests/smoke/test_vpc_redundant.py
Smoke tests completed. 67 look OK, 5 have error(s)
Only failed tests results shown below:

Test	Result	Time (s)	Test File
test_04_rvpc_internallb_haproxy_stats_on_all_interfaces	`Error`	199.56	test_internal_lb.py
ContextSuite context=TestTemplateHierarchy>:setup	`Error`	1521.01	test_accounts.py
test_04_extract_Iso	`Failure`	1.09	test_iso.py
test_04_extract_template	`Failure`	1.11	test_templates.py
test_06_download_detached_volume	`Failure`	11.49	test_volumes.py

yadvr · 2019-07-11T06:34:14Z

@blueorangutan test

blueorangutan · 2019-07-11T06:35:12Z

@rhtyd a Trillian-Jenkins test job (centos7 mgmt + kvm-centos7) has been kicked to run smoke tests

yadvr · 2019-07-11T12:48:55Z

@blueorangutan test

blueorangutan · 2019-07-11T12:49:13Z

@rhtyd a Trillian-Jenkins test job (centos7 mgmt + kvm-centos7) has been kicked to run smoke tests

blueorangutan · 2019-07-12T00:39:53Z

Trillian test result (tid-149)
Environment: kvm-centos7 (x2), Advanced Networking with Mgmt server 7
Total time taken: 40829 seconds
Marvin logs: https://github.com/blueorangutan/acs-prs/releases/download/trillian/pr3477-t149-kvm-centos7.zip
Intermittent failure detected: /marvin/tests/smoke/test_accounts.py
Intermittent failure detected: /marvin/tests/smoke/test_iso.py
Intermittent failure detected: /marvin/tests/smoke/test_templates.py
Intermittent failure detected: /marvin/tests/smoke/test_volumes.py
Intermittent failure detected: /marvin/tests/smoke/test_vpc_redundant.py
Smoke tests completed. 67 look OK, 5 have error(s)
Only failed tests results shown below:

Test	Result	Time (s)	Test File
ContextSuite context=TestTemplateHierarchy>:setup	`Error`	1521.43	test_accounts.py
test_04_extract_Iso	`Failure`	1.13	test_iso.py
test_04_extract_template	`Failure`	1.09	test_templates.py
test_06_download_detached_volume	`Failure`	10.49	test_volumes.py
test_05_rvpc_multi_tiers	`Failure`	409.02	test_vpc_redundant.py
test_05_rvpc_multi_tiers	`Error`	437.98	test_vpc_redundant.py

yadvr · 2019-07-12T07:11:34Z

@blueorangutan package

blueorangutan · 2019-10-23T02:11:44Z

Trillian test result (tid-425)
Environment: vmware-65u2 (x2), Advanced Networking with Mgmt server 7
Total time taken: 53497 seconds
Marvin logs: https://github.com/blueorangutan/acs-prs/releases/download/trillian/pr3477-t425-vmware-65u2.zip
Intermittent failure detected: /marvin/tests/smoke/test_accounts.py
Intermittent failure detected: /marvin/tests/smoke/test_iso.py
Intermittent failure detected: /marvin/tests/smoke/test_templates.py
Intermittent failure detected: /marvin/tests/smoke/test_volumes.py
Smoke tests completed. 73 look OK, 4 have error(s)
Only failed tests results shown below:

Test	Result	Time (s)	Test File
ContextSuite context=TestTemplateHierarchy>:setup	`Error`	1517.79	test_accounts.py
test_04_extract_Iso	`Failure`	1.06	test_iso.py
test_04_extract_template	`Failure`	74.53	test_templates.py
test_06_download_detached_volume	`Failure`	89.92	test_volumes.py

ustcweizhou · 2019-10-23T07:20:39Z

@rhtyd I will look into the failures.
it seems ssvm is broken by this PR.

…late changes

ustcweizhou · 2019-10-23T08:16:04Z

@rhtyd
I have pushed a new commit to fix ssvm.
could you please kick off another test ? thanks.

DennisKonrad · 2019-11-07T14:32:17Z

Hi @ustcweizhou,

do you think this will solve #3179 ?
We have this problem with KVM+OvS with redundant VPC Offering where the wrong dev num is chosen. On current master.

I suspect the changes in this PR will at least change the behaviour of VPC also.

ustcweizhou · 2019-11-07T20:27:18Z

@DennisKonrad unfortunately I do not think this pr is helpful on fixing the issue you mentioned.

andrijapanicsb · 2020-01-03T20:39:26Z

@DaanHoogland all 3 envs failed, I wiped them to regain some resources on Trillian

DaanHoogland · 2020-01-05T14:42:14Z

build failures again "TASK [Remove previous SSH key from Project if it exists] ". Not sure if this is related to the PR, but looks like it.

yadvr · 2020-01-06T04:33:53Z

Trillian test failed, needs re-run @DaanHoogland cc @andrijapanicsb
@blueorangutan package

blueorangutan · 2020-01-06T04:34:37Z

@rhtyd a Jenkins job has been kicked to build packages. I'll keep you posted as I make progress.

blueorangutan · 2020-01-06T04:54:52Z

Packaging result: ✔centos6 ✔centos7 ✔debian. JID-536

andrijapanicsb · 2020-01-06T07:27:29Z

@blueorangutan test matrix

blueorangutan · 2020-01-06T07:27:38Z

@andrijapanicsb a Trillian-Jenkins matrix job (centos7 mgmt + xs71, centos7 mgmt + vmware65, centos7 mgmt + kvmcentos7) has been kicked to run smoke tests

blueorangutan · 2020-01-06T19:54:40Z

Trillian test result (tid-704)
Environment: kvm-centos7 (x2), Advanced Networking with Mgmt server 7
Total time taken: 42566 seconds
Marvin logs: https://github.com/blueorangutan/acs-prs/releases/download/trillian/pr3477-t704-kvm-centos7.zip
Intermittent failure detected: /marvin/tests/smoke/test_privategw_acl.py
Smoke tests completed. 76 look OK, 1 have error(s)
Only failed tests results shown below:

Test	Result	Time (s)	Test File
test_02_vpc_privategw_static_routes	`Failure`	267.01	test_privategw_acl.py
test_03_vpc_privategw_restart_vpc_cleanup	`Failure`	267.22	test_privategw_acl.py
test_04_rvpc_privategw_static_routes	`Failure`	412.07	test_privategw_acl.py

blueorangutan · 2020-01-06T21:07:25Z

Trillian test result (tid-703)
Environment: xenserver-71 (x2), Advanced Networking with Mgmt server 7
Total time taken: 47090 seconds
Marvin logs: https://github.com/blueorangutan/acs-prs/releases/download/trillian/pr3477-t703-xenserver-71.zip
Intermittent failure detected: /marvin/tests/smoke/test_scale_vm.py
Smoke tests completed. 76 look OK, 1 have error(s)
Only failed tests results shown below:

Test	Result	Time (s)	Test File
test_01_scale_vm	`Failure`	35.02	test_scale_vm.py

blueorangutan · 2020-01-07T00:05:13Z

Trillian test result (tid-705)
Environment: vmware-65u2 (x2), Advanced Networking with Mgmt server 7
Total time taken: 57883 seconds
Marvin logs: https://github.com/blueorangutan/acs-prs/releases/download/trillian/pr3477-t705-vmware-65u2.zip
Intermittent failure detected: /marvin/tests/smoke/test_deploy_vm_root_resize.py
Smoke tests completed. 76 look OK, 1 have error(s)
Only failed tests results shown below:

Test	Result	Time (s)	Test File
test_00_deploy_vm_root_resize	`Failure`	404.93	test_deploy_vm_root_resize.py

DaanHoogland · 2020-01-08T14:01:21Z

@rhtyd @andrijapanicsb differnet errors at different environments above. as this had been reviewed and tested before, do we spend more time investigating?

DaanHoogland · 2020-01-21T15:31:49Z

ping @rhtyd @andrijapanicsb ??

… guest IP in RVR (apache#3477) When we create a vm in the network with redundant VRs, the lease file in the vm (for example /var/lib/dhcp/dhclient.eth0.leases) shows the dhcp-server-identifier is the guest ip (not vip/gateway) of master VR. That's the ip ipaddress where the vm fetch password and metadata from. if we stop the master VR (then backup will be master) or restart the network with cleanup (VRs will be created), the guest ip of master VR changes so vm are not able to get metadata/ssh-key using the ips in dhcp lease file. Setting up metadata/password/dhcp server on gateway instead of guest IP in redundant VRs will fix the issues. FIxes apache#3409

yadvr added this to the 4.13.0.0 milestone Jul 9, 2019

yadvr reviewed Jul 9, 2019

View reviewed changes

yadvr added component:virtual-router type:bug labels Jul 9, 2019

ustcweizhou closed this Jul 9, 2019

ustcweizhou reopened this Jul 9, 2019

yadvr changed the base branch from 4.11 to master July 10, 2019 15:37

apache deleted a comment from blueorangutan Jul 11, 2019

secondary storage: fix apache2 misconfig in SSVM caused by vhost.temp…

9d6e00a

…late changes

DaanHoogland closed this Jan 3, 2020

DaanHoogland reopened this Jan 3, 2020

apache deleted a comment from blueorangutan Jan 6, 2020

yadvr approved these changes Jan 28, 2020

View reviewed changes

yadvr merged commit ff1c6e7 into apache:4.13 Jan 28, 2020

		@@ -1,4 +1,4 @@
		<VirtualHost 10.1.1.1:80>
		<VirtualHost 10.1.1.1:8180>

Conversation

ustcweizhou commented Jul 9, 2019

Description

Types of changes

Screenshots (if appropriate):

How Has This Been Tested?

Uh oh!

ustcweizhou commented Jul 9, 2019

Uh oh!

yadvr commented Jul 9, 2019

Uh oh!

yadvr Jul 9, 2019

Choose a reason for hiding this comment

Uh oh!

ustcweizhou Jul 9, 2019

Choose a reason for hiding this comment

Uh oh!

yadvr Jul 9, 2019

Choose a reason for hiding this comment

Uh oh!

yadvr Jul 9, 2019

Choose a reason for hiding this comment

Uh oh!

yadvr commented Jul 9, 2019

Uh oh!

blueorangutan commented Jul 9, 2019

Uh oh!

blueorangutan commented Jul 9, 2019

Uh oh!

yadvr commented Jul 9, 2019

Uh oh!

blueorangutan commented Jul 9, 2019

Uh oh!

yadvr commented Jul 10, 2019

Uh oh!

blueorangutan commented Jul 10, 2019

Uh oh!

yadvr commented Jul 10, 2019

Uh oh!

blueorangutan commented Jul 10, 2019

Uh oh!

blueorangutan commented Jul 10, 2019

Uh oh!

yadvr commented Jul 10, 2019

Uh oh!

blueorangutan commented Jul 10, 2019

Uh oh!

blueorangutan commented Jul 11, 2019

Uh oh!

yadvr commented Jul 11, 2019

Uh oh!

blueorangutan commented Jul 11, 2019

Uh oh!

yadvr commented Jul 11, 2019

Uh oh!

blueorangutan commented Jul 11, 2019

Uh oh!

blueorangutan commented Jul 12, 2019

Uh oh!

yadvr commented Jul 12, 2019

Uh oh!

blueorangutan commented Oct 23, 2019

Uh oh!

ustcweizhou commented Oct 23, 2019

Uh oh!

ustcweizhou commented Oct 23, 2019

Uh oh!

DennisKonrad commented Nov 7, 2019 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

ustcweizhou commented Nov 7, 2019

Uh oh!

andrijapanicsb commented Jan 3, 2020

Uh oh!

DaanHoogland commented Jan 5, 2020

Uh oh!

yadvr commented Jan 6, 2020

Uh oh!

blueorangutan commented Jan 6, 2020

Uh oh!

DennisKonrad commented Nov 7, 2019 •

edited

Loading