SmartState Analysis fails with error 409 Conflict and Failed to create evm snapshot with EMS. Error: [Timeout::ExitException]: [execution expired]

[chitra705]

While running SmartState Analysis on instances in a ManageIQ environment integrated with OpenStack
ManageIQ version: [Petrosian]

The following errors appears :

Failed to create evm snapshot with EMS. Error: [Excon::Error::Conflict]: [Expected([200, 202]) <=> Actual(409 Conflict) excon.error.response :body => "{\"conflictingRequest\": {\"code\": 409, \"message\": \"Cannot 'createImage' instance afa029f1-a353-40ca-a874-79c4c71db83d while it is in task_state image_uploading\"}}" :cookies => [ ] :headers => { "content-length" => "160" "content-type" => "application/json; charset=UTF-8" "date" => "Mon, 12 May 2025 09:45:28 GMT" "openstack-api-version" => "compute 2.15" "server" => "Apache/2.4.52 (Ubuntu)" "vary" => "OpenStack-API-Version,X-OpenStack-Nova-API-Version" "x-compute-request-id" => "req-6fa3da5e-0525-40ee-8e55-3058d55af6dd" "x-openstack-nova-api-version" => "2.15" "x-openstack-request-id" => "req-6fa3da5e-0525-40ee-8e55-3058d55af6dd" } :host => "192.168.150.33" :local_address => "192.168.210.143" :local_port => 46246 :method => "POST" :omit_default_port => true :path => "/v2.1/servers/afa029f1-a353-40ca-a874-79c4c71db83d/action" :port => 8774 :query => nil :reason_phrase => "Conflict" :remote_ip => "192.168.150.33" :scheme => "http" :status => 409 :status_line => "HTTP/1.1 409 Conflict\r\n" ]

evm[71570]: ERROR -- evm: Q-task_id([job_dispatcher]) MIQ(ManageIQ::Providers::Openstack::CloudManager::Scanning::Job#process_abort) job aborting, Failed to create evm snapshot with EMS. Error: [Timeout::ExitException]: [execution expired]

[agrare]

@chitra705 is that VM always in the image_uploading state or is that temporary?

I just tried to do a SmartState scan on an openstack instance on version 2025.1 and it was successful:

[----] D, [2025-05-20T17:13:37.005159#207852:8afc] DEBUG -- evm: Q-task_id([job_dispatcher]) MIQ(MiqOpenStackInstance#create_evm_snapshot) instance_id=[3ffcd497-0632-40a1-978c-c17d6fc0ef77]: No existing snapshot detected for: demo1
[----] D, [2025-05-20T17:13:37.005471#207852:8afc] DEBUG -- evm: Q-task_id([job_dispatcher]) MIQ(MiqOpenStackInstance#create_evm_snapshot) instance_id=[3ffcd497-0632-40a1-978c-c17d6fc0ef77]: Snapshotting instance: demo1...
[----] D, [2025-05-20T17:13:41.488659#207852:8afc] DEBUG -- evm: Q-task_id([job_dispatcher]) MIQ(MiqOpenStackInstance#create_evm_snapshot) instance_id=[3ffcd497-0632-40a1-978c-c17d6fc0ef77]: queued
[----] D, [2025-05-20T17:13:42.541558#207852:8afc] DEBUG -- evm: Q-task_id([job_dispatcher]) MIQ(MiqOpenStackInstance#create_evm_snapshot) instance_id=[3ffcd497-0632-40a1-978c-c17d6fc0ef77]: queued
[----] D, [2025-05-20T17:13:43.575610#207852:8afc] DEBUG -- evm: Q-task_id([job_dispatcher]) MIQ(MiqOpenStackInstance#create_evm_snapshot) instance_id=[3ffcd497-0632-40a1-978c-c17d6fc0ef77]: queued
[----] I, [2025-05-20T17:13:44.464602#207831:8afc]  INFO -- evm: MIQ(MiqQueue.put) Message id: [14401], Zone: [], Role: [], Server: [], MiqTask id: [], Handler id: [], Ident: [generic], Target id: [], Instance id: [], Task id: [], Command: [MiqQueue.check_for_timeout], Timeout: [600], Priority: [50], State: [ready], Deliver On: [], Data: [], Args: []
[----] D, [2025-05-20T17:13:44.698961#207852:8afc] DEBUG -- evm: Q-task_id([job_dispatcher]) MIQ(MiqOpenStackInstance#create_evm_snapshot) instance_id=[3ffcd497-0632-40a1-978c-c17d6fc0ef77]: queued
[----] D, [2025-05-20T17:13:45.735080#207852:8afc] DEBUG -- evm: Q-task_id([job_dispatcher]) MIQ(MiqOpenStackInstance#create_evm_snapshot) instance_id=[3ffcd497-0632-40a1-978c-c17d6fc0ef77]: queued
[----] D, [2025-05-20T17:13:46.313152#207683:8a20] DEBUG -- evm: MIQ(MiqServer::WorkerManagement::Process#invoke_algorithm) Invoke algorithm started with options: [{:name=>"used_swap_percent_gt_value", :value=>80, :type=>:kill}]
[----] D, [2025-05-20T17:13:46.313348#207683:8a20] DEBUG -- evm: MIQ(MiqServer::WorkerManagement::Process#invoke_algorithm) Executing [kill] algorithm: [used_swap_percent_gt_value]
[----] D, [2025-05-20T17:13:46.313840#207683:8a20] DEBUG -- evm: MIQ(MiqServer::WorkerManagement::Process#invoke_algorithm) Executing [kill] algorithm: [used_swap_percent_gt_value] completed with result: [false]
[----] I, [2025-05-20T17:13:46.319912#207683:8a20]  INFO -- evm: MIQ(MiqServer::WorkerManagement::Process#populate_queue_messages) Fetched 2 miq_queue rows for queue_name=generic, wcount=4, priority=200
[----] I, [2025-05-20T17:13:46.332466#207683:8a20]  INFO -- evm: MIQ(EvmServer#monitor) Server Monitoring Complete - Timings: {:heartbeat=>0.019409656524658203, :server_dequeue=>0.07226896286010742, :role_monitor=>0.001917123794555664, :worker_monitor=>12.362927675247192, :worker_dequeue=>0.018302202224731445, :total_time=>12.47509503364563}
[----] D, [2025-05-20T17:13:46.883479#207852:8afc] DEBUG -- evm: Q-task_id([job_dispatcher]) MIQ(MiqOpenStackInstance#create_evm_snapshot) instance_id=[3ffcd497-0632-40a1-978c-c17d6fc0ef77]: queued
[----] D, [2025-05-20T17:13:47.957235#207852:8afc] DEBUG -- evm: Q-task_id([job_dispatcher]) MIQ(MiqOpenStackInstance#create_evm_snapshot) instance_id=[3ffcd497-0632-40a1-978c-c17d6fc0ef77]: queued
[----] I, [2025-05-20T17:13:48.755679#207849:8afc]  INFO -- evm: MIQ(MiqGenericWorker::Runner#get_message_via_drb) Message id: [14400], Zone: [default], Role: [], Server: [], MiqTask id: [], Handler id: [MiqWorker 1377], Ident: [generic], Target id: [], Instance id: [], Task id: [], Command: [Job.check_jobs_for_timeout], Timeout: [600], Priority: [50], State: [dequeue], Deliver On: [], Data: [], Args: [], Dequeued in: [19.332259923] seconds
[----] I, [2025-05-20T17:13:48.756185#207849:8afc]  INFO -- evm: MIQ(MiqQueue#deliver) Message id: [14400], Delivering...
[----] D, [2025-05-20T17:13:48.756427#207849:8afc] DEBUG -- evm: Checking for timed out jobs
[----] I, [2025-05-20T17:13:48.859831#207849:8afc]  INFO -- evm: MIQ(MiqQueue#delivered) Message id: [14400], State: [ok], Delivered in [0.103647655] seconds
[----] I, [2025-05-20T17:13:48.869901#207849:8afc]  INFO -- evm: MIQ(MiqGenericWorker::Runner#get_message_via_drb) Message id: [14401], Zone: [], Role: [], Server: [], MiqTask id: [], Handler id: [MiqWorker 1377], Ident: [generic], Target id: [], Instance id: [], Task id: [], Command: [MiqQueue.check_for_timeout], Timeout: [600], Priority: [50], State: [dequeue], Deliver On: [], Data: [], Args: [], Dequeued in: [4.421257312] seconds
[----] I, [2025-05-20T17:13:48.870262#207849:8afc]  INFO -- evm: MIQ(MiqQueue#deliver) Message id: [14401], Delivering...
[----] I, [2025-05-20T17:13:48.871266#207849:8afc]  INFO -- evm: MIQ(MiqQueue#delivered) Message id: [14401], State: [ok], Delivered in [0.001004177] seconds
[----] D, [2025-05-20T17:13:48.991057#207852:8afc] DEBUG -- evm: Q-task_id([job_dispatcher]) MIQ(MiqOpenStackInstance#create_evm_snapshot) instance_id=[3ffcd497-0632-40a1-978c-c17d6fc0ef77]: queued
[----] D, [2025-05-20T17:13:50.093788#207852:8afc] DEBUG -- evm: Q-task_id([job_dispatcher]) MIQ(MiqOpenStackInstance#create_evm_snapshot) instance_id=[3ffcd497-0632-40a1-978c-c17d6fc0ef77]: saving
[----] D, [2025-05-20T17:13:51.165693#207852:8afc] DEBUG -- evm: Q-task_id([job_dispatcher]) MIQ(MiqOpenStackInstance#create_evm_snapshot) instance_id=[3ffcd497-0632-40a1-978c-c17d6fc0ef77]: active
[----] D, [2025-05-20T17:13:51.165991#207852:8afc] DEBUG -- evm: Q-task_id([job_dispatcher]) MIQ(MiqOpenStackInstance#create_evm_snapshot) instance_id=[3ffcd497-0632-40a1-978c-c17d6fc0ef77]: EVM snapshot creation complete
[----] I, [2025-05-20T17:13:51.903376#207852:8afc]  INFO -- evm: Q-task_id([job_dispatcher]) MIQ(ManageIQ::Providers::Openstack::CloudManager::Scanning::Job#create_snapshot) Created snapshot, reference: [83c556d7-f7eb-4c23-8b20-f3854b2c812b]
[----] I, [2025-05-20T17:13:51.919541#207852:8afc]  INFO -- evm: Q-task_id([job_dispatcher]) MIQ(ManageIQ::Providers::Openstack::CloudManager::Scanning::Job#call_scan) Enter
[----] I, [2025-05-20T17:13:51.926214#207852:8afc]  INFO -- evm: Q-task_id([job_dispatcher]) MIQ(ManageIQ::Providers::Openstack::CloudManager::Vm#scan_metadata) category=[["vmconfig", "accounts", "software", "services", "system"]] [Array]

I did run into another issue that I'll open an issue for.

[miq-bot]

This issue has been automatically marked as stale because it has not been updated for at least 3 months.

If you can still reproduce this issue on the current release or on master, please reply with all of the information you have about it in order to keep the issue open.

[miq-bot]

This issue has been automatically marked as stale because it has not been updated for at least 3 months.

If you can still reproduce this issue on the current release or on master, please reply with all of the information you have about it in order to keep the issue open.