breaks local observability stack

[Integralist]

👋🏻

TIL humanlog service is a thing 😅

I was wondering why none of my application's local observability stack was working (specifically, Tempo).

I then noticed

$ lsof -i ":4318"
COMMAND    PID        USER   FD   TYPE             DEVICE SIZE/OFF NODE NAME
humanlog 80634 integralist   11u  IPv4 0x6c84fe089dcb0735      0t0  TCP localhost:4318 (LISTEN)

I tried to kill the process and it spun back up again.

Couldn't find anything via brew services or launchctl and then finally stumbled over:

ps humanlog
 PID   User        │ TTY CPU MEM:▼ CPU Time │ Command
                   │     [%]   [%]          │
 79979 integralist │     0.0   0.6 00:00:00 │ /Users/integralist/.humanlog/bin/humanlog service run

So I ran humanlog service stop and that did the trick.

I'm not sure what causes this service to run, but I have a feeling like it shouldn't be running by default when installed (unless I triggered it by accident when I last installed humanlog?).

EDIT: Looks like this might be a new feature from ~4 days ago:
45748fc

[aybabtme]

Oups, sorry about that, it shouldn't have turned on for you. I tried to make existing users of the CLI be opt-ed out of these new things by default, but I think all the problems you faced with recent updates have muddied the waters and it turned on for you as if you were a new user. 🥺

I'm making humanlog handle more than just logs and making it grow into an OpenTelemetry localhost stack. This is what I've been working on and why all the recent updates.

It's actually very interesting to see that you're running Tempo locally. Since the beginning of humanlog, I always wanted it to be able to search the logs that it parsed and prettified, and recently I started adding support for search and so on. Then I realized I want the same thing for tracing, so I added support for tracing. My idea being that running Grafana/Loki/Tempo locally is a bit of a hassle. So the fact that you actually do this is quite interesting! Would love to chat with you about it, if you're willing? I just wrote some docs about the new stuff in humanlog -> https://humanlog.io/docs/get-started/introduction

[Integralist]

My idea being that running Grafana/Loki/Tempo locally is a bit of a hassle

oh for sure 😄 but in my case I like to use the Tempo UI to validate a bunch of stuff prior to pushing it up to production and discovering things aren't working as I expect.

Essentially I have a ./local/observability/docker-compose.yaml with a bunch of other yaml files that configure the whole stack. Most of which was copied (iirc) from one of the official Grafana repos.

[aybabtme]

Hah you see, that's exactly what I want to make better:

to validate a bunch of stuff prior to pushing it up to production and discovering things aren't working as I expect

I don't like that observability is a second-class citizen when it comes to the local development flow.

In my dreams, it should be possible to craft dashboards, alerts, panels... locally and "push" these changes to production. Like a GitHub PR or a PlanetScale Branch, but for observability. Basically the workflow you've setup for yourself, but as a crafted experience.

Is this something that would be useful to you?

[Integralist]

Is this something that would be useful to you?

I'm interested to understand how these new humanlog features work, but it's unlikely something I'm going to have capacity to dig into any time soon. But I'll be keeping an eye out on the humanlog repo to see how things move.

As far as this issue is concerned I would say the new features should definitely be opt-in. Currently, I'm scripting humanlog service stop into my Make workflow right now. I'll have to see how to opt-out of that new feature.

[aybabtme]

I forgot to mention the essential, you can disable it with:

humanlog config disable query-engine

[Integralist]

@aybabtme thanks for the relevant command.

FYI you should probably do some nil pointer checks as I ran your command but still had humanlog service stop defined in a Makefile target when starting up my application and I notice this runtime panic...

panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x2 addr=0xc8 pc=0x100fc28b4]

goroutine 1 [running, locked to thread]:
main.(*serviceHandler).close(...)
        github.com/humanlogio/humanlog/cmd/humanlog/service.go:452
main.serviceCmd.func2(0x140005b8dd8?)
        github.com/humanlogio/humanlog/cmd/humanlog/service.go:94 +0x64
github.com/urfave/cli.(*App).RunAsSubcommand.func1()
        github.com/urfave/cli@v1.22.14/app.go:379 +0x40
github.com/urfave/cli.(*App).RunAsSubcommand(0x14000485c00, 0x14000344000)
        github.com/urfave/cli@v1.22.14/app.go:396 +0x68c
github.com/urfave/cli.Command.startApp({{0x10298dd1f, 0x7}, {0x10298670d, 0x3}, {0x0, 0x0, 0x0}, {0x1029e0754, 0x3d}, {0x0, ...}, ...}, ...)
        github.com/urfave/cli@v1.22.14/command.go:380 +0x91c
github.com/urfave/cli.Command.Run({{0x10298dd1f, 0x7}, {0x10298670d, 0x3}, {0x0, 0x0, 0x0}, {0x1029e0754, 0x3d}, {0x0, ...}, ...}, ...)
        github.com/urfave/cli@v1.22.14/command.go:103 +0x628
github.com/urfave/cli.(*App).Run(0x14000484a80, {0x14000122180, 0x3, 0x3})
        github.com/urfave/cli@v1.22.14/app.go:277 +0x7e0
main.main()
        github.com/humanlogio/humanlog/cmd/humanlog/main.go:103 +0x10c

You can replicate by just running humanlog config disable query-engine followed by humanlog service stop.

[aybabtme]

@Integralist sorry for the delay and thanks for the report - I've been out for a bit. We'll look at this now :)

[KevRiver]

@Integralist i'm working on it!