Multiple problems following installation documentation

[likwid]

I have been interested in ignite (and Firecracker) for awhile, and in trying to walk through the basic installation I ran into several issues. I have been following this guide: https://github.com/weaveworks/ignite/blob/main/docs/installation.md.

I am running on a bare-metal machine I built, so there is no cloud infrastructure involved at all.

Information from my machine:

❯ lscpu | grep Virtualization
Virtualization:                  AMD-V

❯ lsmod | grep kvm
kvm_amd               167936  0
kvm                  1130496  1 kvm_amd
ccp                   110592  1 kvm_amd

❯ kvm-ok
INFO: /dev/kvm exists
KVM acceleration can be used

❯ containerd --version
containerd github.com/containerd/containerd 1.6.12-0ubuntu1~22.04.1

❯ ignite version
Ignite version: version.Info{Major:"0", Minor:"10", GitVersion:"v0.10.0", GitCommit:"4540abeb9ba6daba32a72ef2b799095c71ebacb0", GitTreeState:"clean", BuildDate:"2021-07-19T20:52:59Z", GoVersion:"go1.16.3", Compiler:"gc", Platform:"linux/amd64", SandboxImage:version.Image{Name:"weaveworks/ignite", Tag:"v0.10.0", Delimeter:":"}, KernelImage:version.Image{Name:"weaveworks/ignite-kernel", Tag:"5.10.51", Delimeter:":"}}
Firecracker version: v0.22.4

I created a smoke test per the documentation using this as my source:

apiVersion: ignite.weave.works/v1alpha4
kind: VM
metadata:
  name: smoke-test
  uid: $(uuidgen)
spec:
  image:
    oci: weaveworks/ignite-ubuntu
  cpus: 2
  diskSize: 3GB
  memory: 800MB
status:
  running: true

It does not seem to like uuidgen, even though running the command works on the machine.

✦ ❯ DEBU[0112] FileWatcher: Sending update: MODIFY -> "/etc/firecracker/manifests/smoke-test.yml"
DEBU[0112] FileWatcher: Dispatched events batch and reset the events cache
DEBU[0112] GenericMappedRawStorage: AddMapping: "vm/$(uuidgen)" -> "/etc/firecracker/manifests/smoke-test.yml"
DEBU[0112] SyncStorage: Received update {{CREATE &TypeMeta{Kind:VM,APIVersion:ignite.weave.works/v1alpha4,}} 0xc000532150} true
DEBU[0112] SyncStorage: Sent update: {CREATE &TypeMeta{Kind:VM,APIVersion:ignite.weave.works/v1alpha4,}}
DEBU[0112] FileWatcher: Skipping suspended event MODIFY for path: "/etc/firecracker/manifests/smoke-test.yml"
INFO[0112] Creating VM "$(uuidgen)" with name "smoke-test"...
DEBU[0112] Ensuring image weaveworks/ignite-ubuntu:latest exists, or importing it...
DEBU[0112] Found image with UID f212517b8fa491ea
DEBU[0112] Ensuring kernel weaveworks/ignite-kernel:5.10.51 exists, or importing it...
DEBU[0112] Found kernel with UID 63e77c977a06853f
DEBU[0112] FileWatcher: Skipping suspended event MODIFY for path: "/etc/firecracker/manifests/smoke-test.yml"
ERRO[0112] An error occurred when processing a VM update: command ["dmsetup" "create" "--verifyudev" "ignite-$(uuidgen)-base"] exited with "device-mapper: create ioctl on ignite-\\x24\\x28uuidgen\\x29-base  failed: Device or resource busy\nCommand failed.\n": exit status 1

Now I have a VM named $(uuidgen), which I cannot kill from the command line. I was able to remove the VM by editing the smoke-test.yml file to set running: false, but then I got a go panic.

✦ ❯ DEBU[0201] FileWatcher: Sending update: MODIFY -> "/etc/firecracker/manifests/smoke-test.yml"
DEBU[0201] FileWatcher: Dispatched events batch and reset the events cache
DEBU[0201] SyncStorage: Received update {{MODIFY &TypeMeta{Kind:VM,APIVersion:ignite.weave.works/v1alpha4,}} 0xc000532150} true
DEBU[0201] SyncStorage: Sent update: {MODIFY &TypeMeta{Kind:VM,APIVersion:ignite.weave.works/v1alpha4,}}
DEBU[0201] FileWatcher: Skipping suspended event MODIFY for path: "/etc/firecracker/manifests/smoke-test.yml"
DEBU[0202] FileWatcher: Sending update: DELETE -> "/etc/firecracker/manifests/smoke-test.yml"
DEBU[0202] SyncStorage: Received update {{DELETE &TypeMeta{Kind:VM,APIVersion:ignite.weave.works/__internal,}} 0xc000532150} true
DEBU[0202] GenericMappedRawStorage: RemoveMapping: "vm/$(uuidgen)"
DEBU[0202] FileWatcher: Skipping suspended event DELETE for path: "/etc/firecracker/manifests/smoke-test.yml"
DEBU[0202] SyncStorage: Sent update: {DELETE &TypeMeta{Kind:VM,APIVersion:ignite.weave.works/__internal,}}
INFO[0202] Removing VM "$(uuidgen)" with name "<deleted>"...
panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x0 pc=0xc649f8]

goroutine 97 [running]:
github.com/weaveworks/ignite/pkg/operations.StopVM(0xc00082aa80, 0xc0001b0101, 0x11, 0x0)
	/go/src/github.com/weaveworks/ignite/pkg/operations/remove.go:89 +0x78
github.com/weaveworks/ignite/pkg/operations.CleanupVM(0xc00082aa80, 0x4, 0x155f599)
	/go/src/github.com/weaveworks/ignite/pkg/operations/remove.go:38 +0x2e9
github.com/weaveworks/ignite/pkg/operations/reconcile.remove(0xc00082aa80, 0x1a8, 0x7f428e7e81d8)
	/go/src/github.com/weaveworks/ignite/pkg/operations/reconcile/reconcile.go:166 +0x11e
github.com/weaveworks/ignite/pkg/operations/reconcile.handleDelete(...)
	/go/src/github.com/weaveworks/ignite/pkg/operations/reconcile/reconcile.go:104
github.com/weaveworks/ignite/pkg/operations/reconcile.ReconcileManifests.func2(0xc000622a38, 0xc000680000)
	/go/src/github.com/weaveworks/ignite/pkg/operations/reconcile/reconcile.go:74 +0x2a
github.com/weaveworks/ignite/pkg/operations/reconcile.runHandle(0xc0008d3e78)
	/go/src/github.com/weaveworks/ignite/pkg/operations/reconcile/reconcile.go:85 +0x2b
github.com/weaveworks/ignite/pkg/operations/reconcile.ReconcileManifests(0xc0006160e0)
	/go/src/github.com/weaveworks/ignite/pkg/operations/reconcile/reconcile.go:72 +0x606
github.com/weaveworks/ignite/cmd/ignited/cmd.NewCmdDaemon.func1.1(0xc0006160e0)
	/go/src/github.com/weaveworks/ignite/cmd/ignited/cmd/daemon.go:32 +0x6a
created by github.com/weaveworks/ignite/cmd/ignited/cmd.NewCmdDaemon.func1
	/go/src/github.com/weaveworks/ignite/cmd/ignited/cmd/daemon.go:30 +0xde

[1]  + exit 2     sudo ignited daemon --log-level debug

I am sharing the information in case it is useful for debugging, this is just a play project for me, but I am interested in FireKube ultimately, and getting ignite working seemed like a decent first step.

[woodliu]

@likwid Hi, this issue is cause by the devicemapper, and there is a comment in the source code:

		case update.ObjectEventDelete:
			runHandle(func() error {
				// TODO: Temporary VM Object for removal
				return handleDelete(vm)
			})

I think the author has noticed this, but not finish yet...
And about the crash stack, i think it is caused by the vm.Status.Runtime, it is the only pointer here.

I will try to fix this issue after i finish the research about the ignite source code.

[woodliu]

Hi @likwid This is the root cause, there are some bugs both ignite and libgitops.