A valid host was not found
If your hosts are not ready or your environment does not fulfill all of
the requirements, you might see this message: AEVCOM039E: A valid host cannot be found. If you
cannot solve the problem, review the scheduler
log: /var/log/nova/nova-scheduler.log.
Note: This topic contains example logs. The filters
shown in your log might be slightly different.
Use the information
in these sections to understand the scheduler log:
Sample log
2014-02-26 15:05:29.690 87364 ERROR nova.scheduler.ibm.ego.ego_manager [req-dde88661-265a-43e1-9c86-c7fdd44974d4 0 5a2c2cae9e93461584d18d3516d483da]
The EGO resource manager detected insufficient resources for resource requirement (reserveon(group_all)consume(vcpu=160000:memSize=16384)
select(( hypervisor_type==powervm)&&( activeCompute==1))) and extra resource requirement (select((('HostA') || ('HostB')))).
2014-02-26 15:05:29.699 87364 ERROR nova.scheduler.ibm.ego.ego_scheduler [req-dde88661-265a-43e1-9c86-c7fdd44974d4 0 5a2c2cae9e93461584d18d3516d483da]
Troubleshooting VM 7924aea9-d18d-4803-b9de-1333683bb3b5 (test1) for attempt 1
+---------------------------+------------------------------------------------------------------------------------------------------+
| Property | Value |
+---------------------------+------------------------------------------------------------------------------------------------------+
| Instance ID | 7924aea9-d18d-4803-b9de-1333683bb3b5 |
| | |
| Scheduled Operation | create |
| | |
| Scheduling Attempt | 1 |
| | |
| Error Message | Placement services cannot create a resource allocation due to the following reason: The EGO resource |
| | manager detected insufficient resources for resource requirement (reserveon(group_all)consume(vcpu= |
| | 160000:memSize=16384)select(( hypervisor_type==powervm)&&( activeCompute==1))) and extra resource re |
| | quirement (select((('HostA') || ('HostB')))).. |
| | |
| Resources Required | (((vcpu-vcpuConsumed)>=160000) && ((memSize-memConsumed)>=16384)) |
| | [] |
| | |
| Resource Groups Required | () |
| | ['HostA', 'HostB'] |
| | |
| PowerVMHMCStorageFilter | (('HostA') || ('HostB')) |
| | ['HostA', 'HostB'] |
| | |
| ImagePropertiesFilter | () |
| | ['HostA', 'HostB'] |
| | |
| PowerVMNetworkFilter | (('HostA') || ('HostB')) |
| | ['HostA', 'HostB'] |
| | |
| RamFilter | () |
| | ['HostA', 'HostB'] |
| | |
| CoreFilter | () |
| | ['HostA', 'HostB'] |
| | |
| ComputeFilter | select(( activeCompute==1)) |
| | ['HostA', 'HostB'] |
| | |
| RetryFilter | () |
| | ['HostA', 'HostB'] |
| | |
| ComputeCapabilitiesFilter | () |
| | ['HostA', 'HostB'] |
| | |
| AvailabilityZoneFilter | () |
| | ['HostA', 'HostB'] |
| | |
| Total VM Requirements | (reserveon(group_all)consume(vcpu=160000:memSize=16384)select(( hypervisor_type==powervm)&&( activeC |
| | ompute==1))) and extra resource requirement (select((('HostA') || ('HostB')))). |
| | |
| | |
| Deployed Host | [] |
| | |
+---------------------------+------------------------------------------------------------------------------------------------------+Understanding the log
Each
filter checks a portion of the requirements string and returns the
set of valid targets based on that filters requirements. If for a
specific filter you see empty brackets ([]) or do not see the host
you expect, you should investigate this area. In the sample log in
the previous section, the problem area is ResourcesRequired.
The deploy was requesting more CPU than was available from either
host. Because of this, no host was found and the log lists empty brackets
for Deployed Host.
If the networking is not
symmetric across the hosts you might see only one of the hosts returned
from
PowerVMNetworkFilter, as follows:| PowerVMNetworkFilter | (('HostA') |
| | ['HostA'] ComputeFilter checks
to see if the host is reporting as "active". If the host is not active, ComputeFilter does
not list the host. In this log, no host passed all filters:| PowerVMNetworkFilter | (('HostA') |
| | ['HostA']
| ComputeFilter | select(( activeCompute==1)) |
| | ['HostB'] |Retry example - hosts exhausted
The scheduler tries to place an instance multiple times if it receives a failure after the placement was scheduled. It retries the operation until it runs out of valid hosts or until it reaches the maximum number of retries, which is 3 by default. In either situation, multiple logs are generated.
In this example, Scheduling
Attempt 3 is the final attempt. The scheduler tried the two
available hosts. On the third retry attempt, no more hosts were available,
so it generated the error log and exited. The ErrorMessage exception
indicates that you should consult the host logs in the same directory
for more information specific to that error.
2014-02-26 15:31:50.356 87364 ERROR nova.scheduler.ibm.ego.ego_scheduler [req-2a709272-efaf-4bcd-a0bf-33b97bc65365 0 5a2c2cae9e93461584d18d3516d483da]
Troubleshooting VM 8673a97a-b515-4f45-8e12-1e9b59ba6f6e (test2) for attempt 3
+---------------------+---------------------------------------------------------------------------------+
| Property | Value |
+---------------------+---------------------------------------------------------------------------------+
| Instance ID | 8673a97a-b515-4f45-8e12-1e9b59ba6f6e |
| | |
| Scheduled Operation | create |
| | |
| Scheduling Attempt | 3 |
| | |
| Error Message | Instance creation failed on all of the 'force_hosts': HostA,HostB |
| | |
+---------------------+---------------------------------------------------------------------------------+
Troubleshooting VM 8673a97a-b515-4f45-8e12-1e9b59ba6f6e (test2) for attempt 2
+---------------------------+------------------------------------------------------------------------------------------------------+
| Property | Value |
+---------------------------+------------------------------------------------------------------------------------------------------+
| Instance ID | 8673a97a-b515-4f45-8e12-1e9b59ba6f6e |
| | |
| Scheduled Operation | create |
| | |
| Scheduling Attempt | 2 |
| | |
| Error Message | Error from last host: HostB (node HostB): [u'Traceback (most recent call last): |
| | ', u' File "/usr/lib/python2.6/site-packages/nova/compute/manager.py", line 1063, in _build_instanc |
| | e |
| | context, instance, bdms) |
| | ', u' File "/usr/lib/python2.6/site-packages/powervc_nova/compute/manager.py", line 1331, in _prep_ |
| | block_device |
| | image_meta) |
| | ', u' File "/usr/lib/python2.6/site-packages/powervc_nova/compute/manager.py", line 1094, in _proce |
| | ss_powervc_image_meta |
| | bootable=True |
| | ', u' File "/usr/lib/python2.6/site-packages/powervc_nova/compute/manager.py", line 894, in _prepar |
| | e_volume_by_image |
| | voltype_flavor = self._get_volume_type_from_flavor(context, instance) |
| | ', u' File "/usr/lib/python2.6/site-packages/powervc_nova/compute/manager.py", line 1161, in _get_v |
| | olume_type_from_flavor |
| | available_volume_types = self.volume_api.list_volume_types(context) |
| | ', u' File "/usr/lib/python2.6/site-packages/powervc_nova/volume/cinder.py", line 450, in list_volu |
| | me_types |
| | cinder.cinderclient(context).volume_types.list() |
| | ', u' File "/usr/lib/python2.6/site-packages/cinderclient/v1/volume_types.py", line 89, in list |
| | return self._list("/types", "volume_types") |
| | ', u' File "/usr/lib/python2.6/site-packages/cinderclient/base.py", line 66, in _list |
| | resp, body = self.api.client.get(url) |
| | ', u' File "/usr/lib/python2.6/site-packages/cinderclient/client.py", line 207, in get |
| | return self._cs_request(url, \'GET\', **kwargs) |
| | ', u' File "/usr/lib/python2.6/site-packages/cinderclient/client.py", line 174, in _cs_request |
| | **kwargs)
|
| | ', u' File "/usr/lib/python2.6/site-packages/cinderclient/client.py", line 157, in request |
| | raise exceptions.from_response(resp, body) |
| | ', u"ClientException: <attribute 'message' of 'exceptions.BaseException' objects> (HTTP 503) |
| | "] |
| | |
| Resources Required | (((vcpu-vcpuConsumed)>=5000) && ((memSize-memConsumed)>=2048)) |
| | |
| Resource Groups Required | () |
| | |
| PowerVMHMCStorageFilter | (('HostA') || ('HostB')) |
| | |
| ImagePropertiesFilter | () |
| | |
| PowerVMNetworkFilter | (('HostA') || ('HostB')) |
| | |
| RamFilter | () |
| | |
| CoreFilter | () |
| | |
| ComputeFilter | select(( activeCompute==1)) |
| | |
| RetryFilter | ((! 'HostA')) |
| | |
| ComputeCapabilitiesFilter | () |
| | |
| | |
| Total VM Requirements | (reserveon(group_all)consume(vcpu=5000:memSize=2048)select(( hypervisor_type==powervm)&&( activeComp |
| | ute==1))) and extra resource requirement (select(((( hypervisor_type==powervm)&&( activeCompute==1)) |
| | ) && ((((! 'HostA')) && (('HostA') || ('HostB')))))). |
| | |
| Deployed Host | ['HostB'] |
| | |
+---------------------------+------------------------------------------------------------------------------------------------------+
Troubleshooting VM 8673a97a-b515-4f45-8e12-1e9b59ba6f6e (test2) for attempt 1
+---------------------------+------------------------------------------------------------------------------------------------------+
| Property | Value |
+---------------------------+------------------------------------------------------------------------------------------------------+
| Instance ID | 8673a97a-b515-4f45-8e12-1e9b59ba6f6e |
| | |
| Scheduled Operation | create |
| | |
| Scheduling Attempt | 1 |
| | |
| Error Message | Error from last host: HostA (node HostA): [u'Traceback (most recent call last): |
| | ', u' File "/usr/lib/python2.6/site-packages/nova/compute/manager.py", line 1063, in _build_instanc |
| | e |
| | context, instance, bdms) |
| | ', u' File "/usr/lib/python2.6/site-packages/powervc_nova/compute/manager.py", line 1331, in _prep_ |
| | block_device |
| | image_meta) |
| | ', u' File "/usr/lib/python2.6/site-packages/powervc_nova/compute/manager.py", line 1094, in _proce |
| | ss_powervc_image_meta |
| | bootable=True |
| | ', u' File "/usr/lib/python2.6/site-packages/powervc_nova/compute/manager.py", line 894, in _prepar |
| | e_volume_by_image |
| | voltype_flavor = self._get_volume_type_from_flavor(context, instance) |
| | ', u' File "/usr/lib/python2.6/site-packages/powervc_nova/compute/manager.py", line 1161, in _get_v |
| | olume_type_from_flavor |
| | available_volume_types = self.volume_api.list_volume_types(context) |
| | ', u' File "/usr/lib/python2.6/site-packages/powervc_nova/volume/cinder.py", line 450, in list_volu |
| | me_types |
| | cinder.cinderclient(context).volume_types.list() |
| | ', u' File "/usr/lib/python2.6/site-packages/cinderclient/v1/volume_types.py", line 89, in list |
| | return self._list("/types", "volume_types") |
| | ', u' File "/usr/lib/python2.6/site-packages/cinderclient/base.py", line 66, in _list |
| | resp, body = self.api.client.get(url) |
| | ', u' File "/usr/lib/python2.6/site-packages/cinderclient/client.py", line 207, in get |
| | return self._cs_request(url, \'GET\', **kwargs) |
| | ', u' File "/usr/lib/python2.6/site-packages/cinderclient/client.py", line 174, in _cs_request |
| | **kwargs) |
| | ', u' File "/usr/lib/python2.6/site-packages/cinderclient/client.py", line 157, in request |
| | raise exceptions.from_response(resp, body) |
| | ', u"ClientException: <attribute 'message' of 'exceptions.BaseException' objects> (HTTP 503) |
| | "] |
| | |
| Resources Required | (((vcpu-vcpuConsumed)>=5000) && ((memSize-memConsumed)>=2048)) |
| | |
| Resource Groups Required | () |
| | |
| PowerVMHMCStorageFilter | (('HostA') || ('HostB')) |
| | |
| ImagePropertiesFilter | () |
| | |
| PowerVMNetworkFilter | (('HostA') || ('HostB')) |
| | |
| RamFilter | () |
| | |
| CoreFilter | () |
| | |
| ComputeFilter | select(( activeCompute==1)) |
| | |
| RetryFilter | () |
| | |
| ComputeCapabilitiesFilter | () |
| | |
| AvailabilityZoneFilter | () |
| | |
| Total VM Requirements | (reserveon(group_all)consume(vcpu=5000:memSize=2048)select(( hypervisor_type==powervm)&&( activeComp |
| | ute==1))) and extra resource requirement (select((('HostA') || ('HostB')))). |
| | |
| Deployed Host | ['HostA'] |
| | |
+---------------------------+------------------------------------------------------------------------------------------------------+Retry example - maximum retries reached
The scheduler tries to place an instance multiple times if it receives a failure after the placement was scheduled. It retries the operation until it runs out of valid hosts or until it reaches the maximum number of retries, which is 3 by default. In either situation, multiple logs are generated.
In this example, the error message indicates that the scheduler retried the operation three times, which was the set maximum number of retries. On the fourth attempt, an error message was generated.
2014-02-27 10:28:46.234 986 ERROR nova.scheduler.ibm.ego.ego_scheduler [req-aa3c520c-ab91-4631-b6b6-f6f041e4b029 0 5a2c2cae9e93461584d18d3516d483da]
Troubleshooting VM 2ea952f3-5c27-4db1-aaeb-39e454e179b3 (retryTestInstance) for attempt 4
+---------------------+------------------------------------------------------------------------------------------------------+
| Property | Value |
+---------------------+------------------------------------------------------------------------------------------------------+
| Instance ID | 2ea952f3-5c27-4db1-aaeb-39e454e179b3 |
| | |
| Scheduled Operation | create |
| | |
| Scheduling Attempt | 4 |
| | |
| Error Message | Placement services has exceeded the maximum scheduling attempts of 3 for instance 2ea952f3-5c |
| | 27-4db1-aaeb-39e454e179b3. |
| | |
+---------------------+------------------------------------------------------------------------------------------------------+