CloudStack 4.18: Error while adding new advanced core zone with VMware hypervisor.

[sptest232]

ISSUE TYPE

• Bug Report

COMPONENT NAME

UI

CLOUDSTACK VERSION

4.18

CONFIGURATION

A new installation without any zones

OS / ENVIRONMENT

Oracle Linux Server 8.8
mysql Ver 8.0.30
vSphere 6.7

SUMMARY

An error "Something went wrong; please correct the following: Unable to add the external cluster" when I try to add new zone with VMware hypervisor.

I think the main reason of this error is wrong processing cluster, pod, zone names which I enter in the ACS UI.
In this case my vSphere cluster is named "lab-cluster-1", datacenter name is "Lab-DC1".
When I add in ACS zone with name "z1" I fill "Clusert name" record in UI as "lab-cluster-1" I get the error:

2023-05-25 16:35:13,412 ERROR [c.c.h.v.VmwareServerDiscoverer] (qtp649329985-20:ctx-3035123f ctx-1802f527) (logid:02ad0493) Unable to find host or cluster based on url: /Lab-DC1/Cluster-z1

STEPS TO REPRODUCE

Install CloudStack 4.18, deploy vSphere 6.7, try to add a vmware-zone in the UI ACS.

EXPECTED RESULTS

New zone added without any errors.

ACTUAL RESULTS

Full error message from log file:

2023-05-25 16:35:13,410 ERROR [c.c.h.v.u.VmwareContext] (qtp649329985-20:ctx-3035123f ctx-1802f527) (logid:02ad0493) Path element points to an un-existing inventory entity
2023-05-25 16:35:13,411 ERROR [c.c.h.v.m.VmwareManagerImpl] (qtp649329985-20:ctx-3035123f ctx-1802f527) (logid:02ad0493) Unable to find host from inventory path: /Lab-DC1/Cluster-z1
2023-05-25 16:35:13,412 INFO [c.c.h.v.VmwareServerDiscoverer] (qtp649329985-20:ctx-3035123f ctx-1802f527) (logid:02ad0493) Found 0 hosts.
2023-05-25 16:35:13,412 ERROR [c.c.h.v.VmwareServerDiscoverer] (qtp649329985-20:ctx-3035123f ctx-1802f527) (logid:02ad0493) Unable to find host or cluster based on url: /Lab-DC1/Cluster-z1
2023-05-25 16:35:13,412 INFO [c.c.h.v.u.VmwareContext] (qtp649329985-20:ctx-3035123f ctx-1802f527) (logid:02ad0493) Disconnecting VMware session
2023-05-25 16:35:13,420 WARN [c.c.r.ResourceManagerImpl] (qtp649329985-20:ctx-3035123f ctx-1802f527) (logid:02ad0493) Unable to find the server resources at http://vcsa67.lab/Lab-DC1/Cluster-z1
2023-05-25 16:35:13,420 INFO [c.c.u.e.CSExceptionErrorCode] (qtp649329985-20:ctx-3035123f ctx-1802f527) (logid:02ad0493) Could not find exception: com.cloud.exception.DiscoveryException in error code list fo
r exceptions
2023-05-25 16:35:13,477 WARN [o.a.c.a.c.a.c.AddClusterCmd] (qtp649329985-20:ctx-3035123f ctx-1802f527) (logid:02ad0493) Exception:
com.cloud.exception.DiscoveryException: Unable to add the external cluster
at com.cloud.resource.ResourceManagerImpl.discoverCluster(ResourceManagerImpl.java:588)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.base/java.lang.reflect.Method.invoke(Method.java:566)
at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:344)
at org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:198)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:163)
at org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:97)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:186)
at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:215)
at com.sun.proxy.$Proxy199.discoverCluster(Unknown Source)
at org.apache.cloudstack.api.command.admin.cluster.AddClusterCmd.execute(AddClusterCmd.java:212)
at com.cloud.api.ApiDispatcher.dispatch(ApiDispatcher.java:163)
at com.cloud.api.ApiServer.queueCommand(ApiServer.java:777)
at com.cloud.api.ApiServer.handleRequest(ApiServer.java:601)
at com.cloud.api.ApiServlet.processRequestInContext(ApiServlet.java:347)
at com.cloud.api.ApiServlet$1.run(ApiServlet.java:154)
at org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:55)
at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:102)
at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:52)
at com.cloud.api.ApiServlet.processRequest(ApiServlet.java:151)
at com.cloud.api.ApiServlet.doGet(ApiServlet.java:105)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:645)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:750)
at org.eclipse.jetty.servlet.ServletHolder$NotAsync.service(ServletHolder.java:1450)
at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:799)
at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:554)
at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143)
at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:600)
at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:127)
at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:235)
at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:1624)
at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:233)
at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1440)
at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:188)
at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:505)
at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:1594)
at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:186)
at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1355)
at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
at org.eclipse.jetty.server.handler.gzip.GzipHandler.handle(GzipHandler.java:772)
at org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:146)
at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:127)
at org.eclipse.jetty.server.Server.handle(Server.java:516)
at org.eclipse.jetty.server.HttpChannel.lambda$handle$1(HttpChannel.java:487)
at org.eclipse.jetty.server.HttpChannel.dispatch(HttpChannel.java:732)
at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:479)
at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:277)
at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:311)
at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:105)
at org.eclipse.jetty.io.ChannelEndPoint$1.run(ChannelEndPoint.java:104)
at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.runTask(EatWhatYouKill.java:338)
at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:315)
at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:173)
at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run(EatWhatYouKill.java:131)
at org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:409)
at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:883)
at org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:1034)
at java.base/java.lang.Thread.run(Thread.java:829)
2023-05-25 16:35:13,479 INFO [c.c.a.ApiServer] (qtp649329985-20:ctx-3035123f ctx-1802f527) (logid:02ad0493) Unable to add the external cluster

[boring-cyborg]

Thanks for opening your first issue here! Be sure to follow the issue template!

[kiranchavala]

Hi @sptest232

Could you please let me know details you entered here in the screenshot

Is it in this order

lab-cluster-1
vcsa67.lab
Administrator@vpshere.local
password
Lab-DC1

[sptest232]

Hello kiranchavala!
Yes, you're absolutely right!
I enter:
Cluster name: lab-cluster-1
vCenter host: vcsa67.lab
vCenter username: administrator@vsphere.local
vCenter password: some password
vCenter datacenter: Lab-DC1

[kiranchavala]

@sptest232

Thanks for the update, let me check internally

Meanwhile,

1. Try manually adding the cluster from Infrastructure>Cluster > Add Cluster

2 Also , could you cross check, if you are able to ping "vcsa67.lab" from the management server

ping "vcsa67.lab"

Please add it to the /etc/hosts if there is a communication issue to the vcenter

[kiranchavala]

@sptest232 I am able to reproduce the issue as a workaround please try to add cluster from

Infrastructure>Cluster > Add Cluster

@DaanHoogland

This is a bug able to reproduce the issue, when we try to create zone from add zone wizard

We are adding the zone name in the "addcluster" api call

http://10.0.34.116:8080/client/api/?zoneId=9f24aed6-c9a5-4b83-a5bd-065520a3a404&hypervisor=VMware&clustertype=ExternalManaged&podId=ac9bbfa0-f28a-472f-b1e7-70eb694ed795&username=administrator@vsphere.local&password=P@ssword123&url=http://vcenter65.sblab2.local/Trillian/Cluster-kiran-zone1&clustername=vcenter65.sblab2.local/Trillian/Cluster-kiran-zone1&command=addCluster&response=json

---

The issue is not reproducible is the cluster is added from add cluster wizard form

"Infrastructure>Cluster > Add Cluster"

[sptest232]

Yes, kiranchavala. The problem is connected with add zone wizard.

Please notice following information (only in case of add zone wizard use ):

1. Incorrect pod and cluster naming while adding zone with other hypervisors (XenServer).

2. In case of adding vmware hypervisor zone:
   a) error when I try to add datastore cluster as a primary storage.
   If i try datastore cluster as a primary storage via "Infrastructure>Primary Storage" everything as OK.

Log:

2023-05-29 11:58:14,955 DEBUG [o.a.c.s.d.l.CloudStackPrimaryDataStoreLifeCycleImpl] (qtp649329985-19:ctx-ad16463d ctx-58b09641) (logid:b1970496) createPool Params @ scheme - datastorecluster storageHost - dum
my hostPath - /Lab-DC1/ds-cluster-type-1 port - -1
2023-05-29 11:58:14,961 DEBUG [c.c.a.t.Request] (qtp649329985-19:ctx-ad16463d ctx-58b09641) (logid:b1970496) Seq 1-3859303405679804420: Sending { Cmd , MgmtId: 172315270, via: 1(s21.lab), Ver: v1, Flags: 100
011, [{"com.cloud.agent.api.ValidateVcenterDetailsCommand":{"vCenterServerAddress":"dummy","wait":"0","bypassHostMaintenance":"false"}}] }
2023-05-29 11:58:14,961 DEBUG [c.c.a.t.Request] (qtp649329985-19:ctx-ad16463d ctx-58b09641) (logid:b1970496) Seq 1-3859303405679804420: Executing: { Cmd , MgmtId: 172315270, via: 1(s21.lab), Ver: v1, Flags:
100011, [{"com.cloud.agent.api.ValidateVcenterDetailsCommand":{"vCenterServerAddress":"dummy","wait":"0","bypassHostMaintenance":"false"}}] }
2023-05-29 11:58:14,966 DEBUG [c.c.a.m.DirectAgentAttache] (DirectAgent-3:ctx-c0eb1350) (logid:c540686b) Seq 1-3859303405679804420: Executing request
2023-05-29 11:58:14,966 DEBUG [c.c.h.v.r.VmwareResource] (DirectAgent-3:ctx-c0eb1350) (logid:b1970496) Executing resource command ValidateVcenterDetailsCommand: [{"vCenterServerAddress":"dummy","wait":0,"bypa
ssHostMaintenance":false}].
2023-05-29 11:58:14,975 DEBUG [c.c.a.m.DirectAgentAttache] (DirectAgent-3:ctx-c0eb1350) (logid:b1970496) Seq 1-3859303405679804420: Response Received:
2023-05-29 11:58:14,976 DEBUG [c.c.a.t.Request] (DirectAgent-3:ctx-c0eb1350) (logid:b1970496) Seq 1-3859303405679804420: Processing: { Ans: , MgmtId: 172315270, via: 1(s21.lab), Ver: v1, Flags: 10, [{"com.cl
oud.agent.api.Answer":{"result":"false","details":"Provided vCenter server address is invalid","wait":"0","bypassHostMaintenance":"false"}}] }
2023-05-29 11:58:14,976 DEBUG [c.c.a.t.Request] (qtp649329985-19:ctx-ad16463d ctx-58b09641) (logid:b1970496) Seq 1-3859303405679804420: Received: { Ans: , MgmtId: 172315270, via: 1(s21.lab), Ver: v1, Flags:
10, { Answer } }
2023-05-29 11:58:14,976 DEBUG [c.c.a.m.AgentManagerImpl] (qtp649329985-19:ctx-ad16463d ctx-58b09641) (logid:b1970496) Details from executing class com.cloud.agent.api.ValidateVcenterDetailsCommand: Provided v
Center server address is invalid
2023-05-29 11:58:14,978 DEBUG [c.c.s.StorageManagerImpl] (qtp649329985-19:ctx-ad16463d ctx-58b09641) (logid:b1970496) Failed to add data store: Provided vCenter server details does not match with the existing
vCenter in zone id: 1
com.cloud.exception.InvalidParameterValueException: Provided vCenter server details does not match with the existing vCenter in zone id: 1
at org.apache.cloudstack.storage.datastore.lifecycle.CloudStackPrimaryDataStoreLifeCycleImpl.validateVcenterDetails(CloudStackPrimaryDataStoreLifeCycleImpl.java:379)
at org.apache.cloudstack.storage.datastore.lifecycle.CloudStackPrimaryDataStoreLifeCycleImpl.initialize(CloudStackPrimaryDataStoreLifeCycleImpl.java:256)
at com.cloud.storage.StorageManagerImpl.createPool(StorageManagerImpl.java:823)
at com.cloud.storage.StorageManagerImpl.createPool(StorageManagerImpl.java:238)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.base/java.lang.reflect.Method.invoke(Method.java:566)
at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:344)
at org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:198)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:163)
at org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:97)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:186)
at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:215)
at com.sun.proxy.$Proxy101.createPool(Unknown Source)
at org.apache.cloudstack.api.command.admin.storage.CreateStoragePoolCmd.execute(CreateStoragePoolCmd.java:157)
at com.cloud.api.ApiDispatcher.dispatch(ApiDispatcher.java:163)
at com.cloud.api.ApiServer.queueCommand(ApiServer.java:777)
at com.cloud.api.ApiServer.handleRequest(ApiServer.java:601)
at com.cloud.api.ApiServlet.processRequestInContext(ApiServlet.java:347)
at com.cloud.api.ApiServlet$1.run(ApiServlet.java:154)
at org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:55)
at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:102)
at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:52)
at com.cloud.api.ApiServlet.processRequest(ApiServlet.java:151)
at com.cloud.api.ApiServlet.doGet(ApiServlet.java:105)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:645)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:750)
at org.eclipse.jetty.servlet.ServletHolder$NotAsync.service(ServletHolder.java:1450)
at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:799)
at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:554)
at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143)
at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:600)
at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:127)
at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:235)
at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:1624)
at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:233)
at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1440)
at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:188)
at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:505)
at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:1594)
at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:186)
at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1355)
at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
at org.eclipse.jetty.server.handler.gzip.GzipHandler.handle(GzipHandler.java:772)
at org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:146)
at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:127)
at org.eclipse.jetty.server.Server.handle(Server.java:516)
at org.eclipse.jetty.server.HttpChannel.lambda$handle$1(HttpChannel.java:487)
at org.eclipse.jetty.server.HttpChannel.dispatch(HttpChannel.java:732)
at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:479)
at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:277)
at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:311)
at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:105)
at org.eclipse.jetty.io.ChannelEndPoint$1.run(ChannelEndPoint.java:104)
at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.runTask(EatWhatYouKill.java:338)
at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:315)
at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:173)
at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run(EatWhatYouKill.java:131)
at org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:409)
at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:883)
at org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:1034)
at java.base/java.lang.Thread.run(Thread.java:829)
2023-05-29 11:58:14,979 INFO [c.c.a.ApiServer] (qtp649329985-19:ctx-ad16463d ctx-58b09641) (logid:b1970496) Failed to add data store: Provided vCenter server details does not match with the existing vCenter
in zone id: 1

b) when I add a zone physical network, entered name (label) of Management network doesn't apply during zone deploying, the wizard still use "Management Network" vmware vSwitch (vDS) portgroup which results in zone creation error. Maybe it's connected to the default global setting "Vmware management portgroup", but it's not obvious when creating the zone.