Should I be processing low-level/small elements with Pathom?

[kxygk]

I've been playing around with Pathom and it's been great.

Thank you for this amazing library. It really feels like an amazing paradigm shift that's really flown under the radar. I wanna write a blog post once I have this digested a bit better :))

So far I've been rewriting "simple" pipelines in my data processing code. I'm hoping there will be some decoupling and simpler APIs as a result.

But I'm now trying to see if I can abstract more concepts in my application with Pathom

I tried to make a simple example that's hopefully easy to understand.

On a high level I'm curious:

• Can this be made performant?
• Should I just not be doing this in Pathom..?
• How would you architect this problem?

So first in Vanilla Clojure

I have some x-y point. I can view these points as:

• polar coordinates
• normalized (unit length) vectors

You can imagine this is as one little part of some larger application..

(use 'clojure.math)

(def
  xy-points
  "Just a long list of `[x y]` pairs"
  (repeatedly 1000000
              #(vector (rand)
                       (rand))))
#_
(take 5
      xy-points)
;; => ([0.9595777815722415 0.955966600109044]
;;     [0.4582556879334392 0.4586413232063461]
;;     [0.07833332358513934 0.07130513464892763]
;;     [0.7655731396104173 0.37268965701951307]
;;     [0.40480720580295426 0.3954351287740743])

(defn
  to-polar
  "Convert the `[x y]` pairs to `[radius angle]` pairs"
  [[x
    y]]
  [ (sqrt (+ (pow x   ;;radius
                  2)
             (pow y
                  2)))
   (atan2 y
          x) ])
#_
(->> xy-points
     (mapv to-polar)
     (take 5))
;; => ([1.354496828867144 0.7835129670951717]
;;     [0.6483441515706128 0.7858187507103993]
;;     [0.10592701171653926 0.738464855959417]
;;     [0.8514692082173457 0.4530410919554245]
;;     [0.5658955866045997 0.7736871501622257])

(def xy-points-normalized
  "Convert my `[x y]` pairs to a normalized vector
  Also `[x y]`"
  (let [polar-coords (->> xy-points
                          (mapv to-polar))]
    (mapv (fn [[x
                y]
               [radius
                _]]
            (vector (/ x
                       radius)
                    (/ y
                       radius)))
          xy-points
          polar-coords))))
#_
(take 5
      xy-points-normalized)
;; => ([0.9063464150748226 0.4225354137596247]
;;     [0.9724893343242329 0.23294740742410108]
;;     [0.3622086279586388 0.9320970495781651]
;;     [0.6496713506235731 0.7602151907051993]
;;     [0.39979032478244364 0.916606620208663])

This is quite performant.
I could probably get fancier with arrays and transduction etc. ..
It's simple but has lots of downsides:

• Secret index based meanings to values (sometimes x-coord sometimes radius).
• If this is part of a larger application.. you need to remember if your vector has already been normalized or not.
• Maybe you already got the polar form somewhere else and you could reuse it when normalizing your values.. So you need to write another to-polar overload that takes precomputed radius values. Yuck
• Unused values.. I call to-polar during normalization, which calculates an angle which I never use.
  (the example is a bit artificial :)) )

Now I'm trying to rethink this in terms of Pathom and how I can get the engine to do all the work for me!

I just need to define resolvers that take x-y points and return the values I'm interested in

(add-libs {'com.wsscode/pathom3 {:mvn/version "2025.01.16-alpha"}})

(require
'[com.wsscode.pathom3.connect.operation :as pco]
'[com.wsscode.pathom3.connect.indexes :as pci]
'[com.wsscode.pathom3.interface.smart-map :as psm])

(pco/defresolver $radius
"Take `::x` and `::y` values and use pythagoras to return a `::radius`"
[{::keys [x
          y]}]
{::radius (sqrt (+ (pow x   ;;radius
                        2)
                   (pow y
                        2)))})
#_
($radius {::x 3.0
          ::y 5.0})
;; => #:user{:radius 5.830951894845301}

(pco/defresolver $angle
"Take `::x` and `::y` values and return an `::angle`"
[{::keys [x
          y]}]
{::angle (atan2 y
                x)})
#_
($angle {::x 3.0
         ::y 5.0})
;; => #:user{:angle 1.0303768265243125}

(pco/defresolver $normalized-point
"L2 norm.. divide x y points by their length"
[{::keys [x
          y
          radius]}]
{::x-norm (/ x
             radius)
 ::y-norm (/ y
             radius)})

#_
($normalized-point {::x      3.0
                    ::y      5.0
                    ::radius 2.0}) 
;; => #:user{:x-norm 1.5, :y-norm 2.5}
;; here the radius can only be infered with the full `register`
;; so for testing you need to supply the values
#_
(::x-norm  (psm/smart-map (pci/register [$radius
                                         $angle
                                         $normalized-point
                                         $polar-data
                                         $normalized-data])
                          {::x 3.0
                           ::y 4.0}))
;; => 0.6  ;; this is 3.0/5.0 (think 3,4,5 triangle)

So those work great! It seems very composable.
Now I want to work with long lists of points.
Instead of [x y] pairs I should use labeled maps - which I think is a lot more clear

(def xy-point-maps
  "Now I remake the x-y data as labeled maps
  So the x and y are explicitely labeled"
  (repeatedly 100000
              #(hash-map ::x
                         (rand)
                         ::y
                         (rand))))
#_
(take 5
      xy-point-maps)
;; => (#:user{:y 0.36939725673492985, :x 0.9602380622865435}
;;     #:user{:y 0.16960025245899446, :x 0.8414923599000971}
;;     #:user{:y 0.23689244515168162, :x 0.3515261363669846}
;;     #:user{:y 0.41155488019829933, :x 0.27518489053192907}
;;     #:user{:y 0.04144667976964722, :x 0.7061907847196561})

I can then use these using "nested input" to process lists of values

https://pathom3.wsscode.com/docs/resolvers#nested-inputs

As far as I understand the EQL, the system doesn't actually know there will be a list coming in.. so I'm a bit unclear if this is a perf bottleneck.

(pco/defresolver $polar-data
  "Converts a lists of xy point maps to
   a list of polar coordinates"
[{::keys [cartesian-data]}]
{::pco/input [{::cartesian-data [::radius
                                 ::angle ]}]}
  {::polar-data (mapv (fn [{::keys [radius
                                    angle]}]
                        {:radius radius
                         :angle  angle}) ;; maybe superfluous
                      cartesian-data)})
#_
($polar-data {::cartesian-data [{::radius 3.0
                                 ::angle  4.0}]})
;; => #:user{:polar-data [{:radius 3.0, :angle 5.0}]}
#_
($polar-data {::cartesian-data [{::x 3.0
                                 ::y 4.0}]})
;; => #:user{:polar-data [{:radius nil, :angle nil}]}
;; Doesn't work b/c it needs the register to find that `radius`!

(pco/defresolver $normalized-data
  [{::keys [cartesian-data]}]
  {::pco/input [{::cartesian-data [::x-norm
                                   ::y-norm]}]}
  {::normalized-data (mapv (fn [{::keys [x-norm
                                         y-norm]}]
                             {::x-norm x-norm
                              ::y-norm y-norm}) ;; maybe superfluous
                           cartesian-data)})

;;     {:y 0.9528055634606857, :x 0.9674033979599416})

(def env (pci/register [$radius
                        $angle
                        $normalized-point
                        $polar-data
                        $normalized-data]))

(def smap (psm/smart-map env
                         {::cartesian-data xy-point-maps}))

(take 5
      (::normalized-data smap))
;; => (#:user{:x-norm 0.08712330380715341, :y-norm 0.9961975355991032}
;;     #:user{:x-norm 0.47009242740808693, :y-norm 0.8826171931780916}
;;     #:user{:x-norm 0.7068921332233743, :y-norm 0.7073213640113715}
;;     #:user{:x-norm 0.8241905340531331, :y-norm 0.5663126023471587}
;;     #:user{:x-norm 0.9010577036764886, :y-norm 0.4336992214026367})


(take 5
      (::polar-data smap))
;; => ({:radius 0.2674784113124991, :angle 1.4835624270007897}
;;     {:radius 0.9248188165128844, :angle 1.0814008320003392}
;;     {:radius 0.7074320549968458, :angle 0.7857016754029951}
;;     {:radius 0.9759277453181741, :angle 0.602024979576675}
;;     {:radius 0.28699758977324435, :angle 0.4485941613968548})

In the end it works! Pretty cool stuff

I haven't benchmarked it, but I'm guessing if I calculate the polar form, the values are reused when getting the normalizing view

Maybe it'd be interesting to add another resolver that goes from polar to cartesian (I'm not sure how the loop would be handled)

But the result is coming out quite slow and chokes if there are too many points. Maybe I'm doing something wrong here? I don't quite understand the performance characteristics of the nested inputs here. I assume the engine resolver is only run once and reused for the whole sequence. Is that right?

I have a general sense that I'm.. as Steve Jobs says.. "holding it wrong"

Should I avoid making the last two resolvers entirely? They don't really do anything other than repackage the data. Maybe I should just do the nested input at each call site?

I have a specific application in mind where I'm pulling out coordinate pairs from a tech.ml.dataset table (to make all sorts of different plots with thing/geom ) and I'd like to effectively "enhance" each row with derived values using Pathom. I can make low-level row resolvers that take rows and effectively add entries (such as polar coordinates). Or I can have high level resolvers that take columns in the table and derive new columns. But the second options gets messy when the derived entries maybe only make sense for certain rows... So I'm more partial to the first solution if it's relatively fast.

Do you have any other general thoughts or advice architecturally?

Thank you once again for this awesome lib

[kxygk]

Architecturally I'm still not sure what I did is great.

But to just look at the performance angle, I bumped up the number of points to 10,000,000 and then did an extremely basic sampling profile using VisualVM. And I'm seeing 90% of the runtime is spent in the Runner (process_map_subquery, run_graph etc... )

So I'm guessing something isn't getting cached and I'm doing something wrong - b/c the query should be identical and .. I'm guessing.. computed only once

[jacobobryant]

Hey! Offering some thoughts here continuing our conversation from reddit; disclaimer to anyone else reading, I'm not a Pathom maintainer or anything, just a happy user.

The main thing that jumps out at me from your writeup is this:

I can then use these using "nested input" to process lists of values

Everything up to this point (no pun intended) looks pretty reasonable, but here I think you're misunderstanding the use of nested inputs. Those are for when you're doing some computation that requires data from another entity in the graph. From the :game/top-players-avg-score, the important thing from that example is that you're reaching into another entity (player(s)) and getting an attribute (:player/score); the fact that this particular example uses a nested input with cardinality-many isn't essential to the feature. To make a different example that with a cardinality-one nested input, here's a resolver from my own app:

(defresolver feed-sub-subtitle [{:keys [sub.feed/feed]}]
  #::pco{:input [{:sub.feed/feed [:feed/url]}]}
  {:sub/subtitle (:feed/url feed)})

This deals with a "feed sub" entity (which represents a particular user's subscription to an RSS feed and records data like "when did the user subscribe to this feed") which is linked to a "feed" entity (which represents an RSS feed independent of any particular subscriber and records data like the feed URL, when it was last synced, etc). The "subtitle" is some text that should be shown on the app's page for that subscription. So here we're just saying to query for the url of the linked feed entity and use that as the subtitle value. (There are also "email sub" entities which also return :sub/subtitle using a different implementation).

So back to what you're trying to do: process a vector of inputs. I've adapted your example code to do that without nested inputs:

(ns repl.pathom-stuff
  (:require
   [com.wsscode.pathom3.connect.operation :as pco]
   [com.wsscode.pathom3.interface.eql :as pie]
   [com.wsscode.pathom3.connect.indexes :as pci]
   [com.wsscode.pathom3.interface.smart-map :as psm]
   [clojure.math :as math]))

(def xy-point-maps
  "Now I remake the x-y data as labeled maps
   So the x and y are explicitely labeled"
  (repeatedly 100000
              #(hash-map ::x (rand)
                         ::y (rand))))

(pco/defresolver radius
  "Take `::x` and `::y` values and use pythagoras to return a `::radius`"
  [{::keys [x y]}]
  {::radius (math/sqrt (+ (math/pow x 2)
                          (math/pow y 2)))})


(pco/defresolver angle
  "Take `::x` and `::y` values and return an `::angle`"
  [{::keys [x y]}]
  {::angle (math/atan2 y x)})

(pco/defresolver normalized-point
  "L2 norm.. divide x y points by their length"
  [{::keys [x y radius]}]
  {::x-norm (/ x radius)
   ::y-norm (/ y radius)})

(def env (pci/register [radius
                        angle
                        normalized-point]))

(comment

  (def smap (psm/smart-map env {::cartesian-data xy-point-maps}))

  (->> (::cartesian-data smap)
       (take 5)
       (mapv #(select-keys % [::radius ::angle ::x-norm ::y-norm])))
  ; =>
  [#:repl.pathom-stuff{:radius 1.1431687691627064,
                       :angle 0.9163849224556841,
                       :x-norm 0.6086923532670513,
                       :y-norm 0.7934063391946268}
   #:repl.pathom-stuff{:radius 1.008354315551966,
                       :angle 1.1392217997848162,
                       :x-norm 0.41830147622250397,
                       :y-norm 0.9083082488836451}
   #:repl.pathom-stuff{:radius 0.584940665829171,
                       :angle 1.2047573624544963,
                       :x-norm 0.35791962292579027,
                       :y-norm 0.9337523994746466}
   #:repl.pathom-stuff{:radius 0.9961671851935875,
                       :angle 0.4032006320170194,
                       :x-norm 0.9198098936252301,
                       :y-norm 0.39236431997461574}
   #:repl.pathom-stuff{:radius 0.9299379314736511,
                       :angle 0.9374570181781635,
                       :x-norm 0.5918397213803865,
                       :y-norm 0.8060556706558093}]
  )

i.e. if we give pathom a vector of entities, pathom can figure out how to apply the resolvers to each entity.

Note that I personally always use process rather than smart-map, e.g.:

(require '[com.wsscode.pathom3.interface.eql :as pie])

(pie/process env
             {::cartesian-data (into [] (take 5) xy-point-maps)}
             [{::cartesian-data [::radius
                                 ::angle
                                 ::x-norm
                                 ::y-norm]}])
; =>
#:repl.pathom-stuff{:cartesian-data
                    [#:repl.pathom-stuff{:radius 1.1431687691627064,
                                         :angle 0.9163849224556841,
                                         :x-norm 0.6086923532670513,
                                         :y-norm 0.7934063391946268}
                     #:repl.pathom-stuff{:radius 1.008354315551966,
                                         :angle 1.1392217997848162,
                                         :x-norm 0.41830147622250397,
                                         :y-norm 0.9083082488836451}
                     #:repl.pathom-stuff{:radius 0.584940665829171,
                                         :angle 1.2047573624544963,
                                         :x-norm 0.35791962292579027,
                                         :y-norm 0.9337523994746466}
                     #:repl.pathom-stuff{:radius 0.9961671851935875,
                                         :angle 0.4032006320170194,
                                         :x-norm 0.9198098936252301,
                                         :y-norm 0.39236431997461574}
                     #:repl.pathom-stuff{:radius 0.9299379314736511,
                                         :angle 0.9374570181781635,
                                         :x-norm 0.5918397213803865,
                                         :y-norm 0.8060556706558093}]}

Architecturally it's helpful IMO to be explicit up front about what data you're using, same as what we do when defining resolvers. whereas if you have a function like (compute-something my-smart-map), it's impossible to know what data compute-something is actually going to use without tracing through its entire call stack.

Anyway, besides that, using process also gives pathom more opportunity to optimize your query since it knows up front all the data you'll need. You can also then convert your resolvers to batch resolvers which might speed things up. If you're processing 10,000,000 points and it seems slow, you could try a batch resolver to see if your process call gets faster:

(pco/defresolver radius
  "Take `::x` and `::y` values and use pythagoras to return a `::radius`"
  [inputs]
  {::pco/input [::x ::y]
   ::pco/batch? true}
  (mapv (fn [{::keys [x y]}]
          {::radius (math/sqrt (+ (math/pow x 2)
                                  (math/pow y 2)))})
        inputs))

;; similar for the `angle` and `normalized-point` resolvers
;; maybe this allows you to speed things up further by doing vector/matrix computations?

Then you can re-run the same process call above and measure with (time ...) or whatever to see if using the batch resolver actually makes a difference. Since this example is so simple, it wouldn't hurt either to measure the run time of doing it all with plain functions too to get an upper bound on what performance you can expect to get when using pathom.

[kxygk]

Thank you so very much @jacobobryant . You really did a deep dive and I just wanted to start by saying how much I appreciate it.

Maybe I've missed a subtle point,
but have you not essentially moved the my nested calls in to process EQL call?
(my later test seems to indicate no perf difference)

I think my toy example was maybe a bit too simple 😄
Abstractly, what I'm trying to do is test how things compose. I'm building a higher-level resolver graphs that doesn't deal with points and their conversions. Looking at my example.. I get that they're not actually doing anything interesting :)) and so you cut them and move the vector traversal to the process EQL

Maybe I can build a better example of what I mean by high-level resolver..

Say I have a resolver that runs across my data and calculates an angular average

(pco/defresolver $angular-average
  [{::keys [cartesian-data]}]
  {::pco/input [{::cartesian-data [::angle]}]}
  {::angular-average (/ (reduce (fn [sum
                                     next-point]
                                  (::angle next-point))
                                0.0
                                cartesian-data)
                        (count cartesian-data))})

sidenote: I wrote cartesian-data, but this was actually a misnomer. The whole point is that I want to be agnostic wrt the coordinate format. You should be able to feed in polar coordinates and the whole thing "just works"

I could write this as a process EQL query naturally

(/ (reduce (fn [sum
                next-point]
             (::angle next-point))
           0.0
           (-> env
               (pie/process {::cartesian-data xy-point-maps}
                            [{::cartesian-data [::angle]}])
               ::cartesian-data))
   (count xy-point-maps))

(there seems to be no performance difference)

but ostensibly you want to use the angular-average attribute in some subsequent resolver..

(pco/defresolver $some-fancy-plotting
  [{::keys [cartesian-data
            angular-average]}]
  ::pco/input [{::cartesian-data [::x-norm
                                  ::y-norm
                                  :radius
                                  :angle]}]
  {::svg-hiccup (my-fancy-plotting-function cartesian-data
                                            angular-average)})

Running with the toy example, if you did this right, you could feed in data in any format and make your plots.. so you could pinch off separate libraries to say.. handle points from the ones handling plotting.

Without Pathom I guess I would implement this using Protocols. But here ideally you'd also be somehow caching intermediary values for all your points. Like given some x-y data , derived attributes such as radius or angle would be computed once and then fetched at the next call or at a different node in the graph.

Looking at the problem again, I think there is a fundamental problem that well.. clojure vectors aren't arrays. They can have different types at each index. You can have one point as an x-y coordinate and the next point in polar coordinates. So the engine is probably forced to recompute the graph at each element?

It seems this is anticipates and should be resolved by the Caching system. The description seems to match the situation ! :)

For example, a Pathom server handling API requests for serving some UI. This kind of service tends to receive the same queries over and over. Instead of re-planning on every query, you can use a more persistent cache to re-use this computation across requests.

But I haven't been able to see any perf difference in the EQL version when i add a cache.

https://pathom3.wsscode.com/docs/cache#persistent-caching

I have two environments

(def env2 (pci/register [$radius
                        $angle
                        $normalized-point
                        $polar-data
                        $normalized-data
                        $angular-average]))


(defonce plan-cache* (atom {}))

(def env3 (-> (pci/register [$radius
                        $angle
                        $normalized-point
                        $polar-data
                        $normalized-data
                             $angular-average])
              (pcp/with-plan-cache plan-cache*)))

And even in the process notation they seem to have the same runtime

(println "Normal EQL `angular average` fetch with CACHE")
(time
  (-> env3
      (pie/process  {::cartesian-data xy-point-maps}
                    [::angular-average]))
)




(println "Normal EQL `angular average` fetch with NO CACHE")
(time
  (-> env2
      (pie/process  {::cartesian-data xy-point-maps}
                    [::angular-average]))
)

(EDIT: Checking in VisualVM it still seems to be spinning in the runner)

Again, thank you so much for discussing this with me :))

[jacobobryant]

The $angular-average resolver does make more sense, i.e. you're using nested inputs because you're doing a computation that actually does need to use all the inputs at once, rather than simply transforming each input independently.

Re: caching, a couple things. I see you're using with-plan-cache, but there's also with-resolver-cache that you might want to be trying out. That's the one that will memoize your resolvers. I think Pathom will set up an atom cache in each process call if you don't provide a cache via with-resolver-cache, but if you're trying to cache things between resolver calls, then yeah you'll probably want to try that out.

That being said: you'll be most likely to see benefits from caching if your resolvers are doing something slow like a network request/database query. For simple computation like this, I wouldn't necessarily expect to see much benefit from caching. Especially since you're already observing that most of the compute time is coming from Pathom code, not your resolver code.

e.g. if you're doing a process call that ends up calling one of your resolvers 10,000,000 times (e.g. to do the cartesian -> polar coordinates conversion), it'll still have to do 10,000,000 cache lookups. In which case you know, maybe you were actually on the right track in your first post: (ab)using nested inputs so you can do that conversion all in a single resolver call maybe would help you get better performance.

(Also heads up, if you experiment with batch resolvers and caching, I'm actually having some trouble getting those two to work together in my own code at the moment... still investigating)

I might take a step back here though. Pathom is a tool that makes the common trade-off of letting you work at a higher level at the cost of some performance. IMO that makes sense for e.g. modeling a business domain with a bunch of different entities and such. For a situation like this where you have a lot of relatively simple data, it might be worth double checking if that tradeoff is worth it? Using some protocols with your own caching logic built in might not be a bad idea. Of course since these posts are just examples, it's quite possible your actual needs are complex to warrant Pathom. Good to check though!

[kxygk]

Great stuff :)))

Re: caching, a couple things. I see you're using with-plan-cache, but there's also with-resolver-cache that you might want to be trying out. That's the one that will memoize your resolvers.

I think the bottleneck at the moment is the runner. (I'm inferring it from the stack I see in VisualVM... so not 100% sure here) It keeps seeing "Okay, I have an ::x and ::y and he wants a ::angle, lets build the graph and figure out how to do that". And it keeps doing this over and over for each point instead of recycling it's derivation.

Ideally it'd figure this out once and then reuse it :)) So I think I just have some bug here. I'll need to poke around more

As far as I understand, the resolver cache is different! The resolver seeing "Oh x = 0.35435 and y = 0.74324. I've seen this input before! I'll go find the corresponding output attribute values" So this would be useful if I need the ::angle somewhere else again - so I don't go about recomputing them. Haven't gotten around to testing this part yet though :)) (***)

For a situation like this where you have a lot of relatively simple data, it might be worth double checking if that tradeoff is worth it? Using some protocols with your own caching logic built in might not be a bad idea. Of course since these posts are just examples, it's quite possible your actual needs are complex to warrant Pathom. Good to check though!

A completely fair thing to ask!

There might just be a breaking point and some stuff that are impossible to do with Pathom - which is I guess what I'm trying to explore here. In theory they seem more flexible than Protocols

But I'm actually tbh hoping Pathom yields a big net performance boost. Auto-parallelization of resolver "branches" and caching are things that can be very annoying to do "manually". When you change some values and only the necessary stuff is recalculated and independent stuff gets calculated on separate threads.. that's magic :))

Going along with the toy example... For instance I could make a Point Protocol. And I could then have two records - one for "x,y" and one for "radius,angle" points. I could then implement "get-angle" "get-normalized-x-y" functions for each? But if I then call "convert-to-polar" and the angle had been previously calculated.. am I going to reuse that value? Maybe you can memoize the protocol functions and make it work but it seems challenging!

A lot of my code right now just ends up recalculating stuff in all sorts of places. If you are passing around a bunch of points, It's much easier to say re-calculate a mean or variance right where you need it instead of calculating it at some top level and pass it around everywhere. I guess I'm hoping the resolvers save me here haha

(***)
My concern there is it's unclear how to maintain the resolver caches. For instance in this scenario: Your messing around at the REPL generating data points. When the points are stale they get garbage collected. But their corresponding cache entries don't get cleared (could be a single simple resolver like ::x,::y -> ::angle, but could be a graph). So either the cache balloons in size with the app lifetime or you have to set some arbitrary size and hope you don't end up clearing non-stale data :S

Neither solution is very satisfying... Maybe the high level resolvers could tune the cache size and keep separate indices (since it knows the data size) but it starts to get complicated.. haven't wrapped my head about that yet

[wilkerlucio]

Hello folks, great discussion here :)

About Pathom performance for this kind of compute intensive tasks, the overhead really adds up... I've been exactly in a situation like yours @kxygk, trying to push that to the limits. In my case I was trying to process GraphQL schemas to turn them into Pathom dynamic resolvers. But after comparing to a straith Clojure implementation, the different is clear, Pathom can add too much overhead in these scenarios.

Some of the reasons that Pathom adds considerable overhead and are tricky to avoid (in case of many resolver calls with fast computations):

1 - Computation just figure out the plan (even when its cached)
To make a plan, Pathom needs 3 things: your indexes, your normalized AST, the input shape. The plan cache will use these 3 to try to find a cached plan, but just to get those 3 there is already a cost.

The indexes are easy, they are mostly constant and Pathom uses its hash.

The AST is already a thing, first because we need to get your EQL and convert it to the AST format (Pathom tries its best to reduce it, internally everything is AST, queries are only handled in the boundaries, where they get converted to AST and used in that form though out the request). So, this means its better to do a single request with an array input than many in a loop, to reduce the conversion cost. But going to AST isn't all, because to properly cache we also need to normalize the ident inputs and the params, because we only care about their shape (existance and names), but not their values, so we need to strip those values to make the cache effective (so that [{[:user/id 1] [:user/name]}] and [{[:user/id 2] [:user/name]}] do the same cache hit).

After that, we also need to convert the input data in the shape format, again to ignore the values and keep only the shape. This is pretty quick if your input is small, but can become large at times and costly. And there is no way to avoid this, in case of collections Pathom does that to each entry in your collection. The reason for that is that Pathom prioritizes figuring the result, so it can't assume your collection entries are consistent, maybe most items have :a and :b, but one has :a and :c, so the plan for that one might be different.

Of course, if its not cached, you are also paying the plan computation, but I guess you got that, the devil is in the details even when its cached. So when your processing is mostly fast calcutations, this Pathom overhead can take over your processing time.

2 - Entity validation
After resolving an entity, Pathom will check if all the required attributes are resolved, and for that again we need to do an extra traversal into your data, and that adds up when you have a lot of entities.

3 - Entity masking
When Pathom goes to send data as input, it has to mask the current entity to avoid leaking data (if a resolver asks only for :a and :b, if the entity has :c, it should not be available in that input, so we avoid users getting data by accident, which would lead to very confusing situations, hard to debug). But that also means another step in which we have to traverse the input shape to grab only the requested attributes.

4 - Subsequent planning
Pathom only plans for one entity at a time, the main reason I choose this path was considering that I can't know what your resolvers actually return, so I can't make a full ahead-of-time plan. This means the costs I mention in 1 have to be redone for every entity you process. I think it is possible to improve this one, by trying to compute all ahead of time, and only recompute some plans in case of some data mismatch.

5 - Plugin engine
Pathom has many plugin entry points, and altough it try to make it as cheap as possible to handle them (specially when there aren't plugins installed), that still a little check that has to run in every operation that supports plugin extensions.

6 - Tracing status
Another thing that can happen a lot, its Pathom computing trace data. This is something I want to improve before the stable release because I want to change the whole tracing mechanism. Currently even if you are not tracing, Pathom still needs to pull in times and try to merge that in the entity meta, which at minimum is a presence check to see if there run stats being accumulated.

7 - output masking
When Pathom is processing, it only keeps accumulating data, but once its time to finish the processing it needs to mask just the data requested in the top level query. That's another step that needs traversal and adds overhead.

Those are all small things, but if your resolvers are mostly fast computations, these small things can become more costly than your process itself. And those are the ones I can remember from top of mind, there are problem other little things like these spread around.

The other end of the spectrum is when your resolvers are mostly IO that take a while, and coordination is tricky part, that's where Pathom performance will shine, being able to batch, paralellize and cache optimize all the things that are tricky to handle in these situations.

So its up to you to design how much you want to delegate out, you can always reduce the number of resolvers and do more calculations by hand to make it faster.

I hope that helps give you a better understand where the overhead comes from. And if you have any other question dont hesitate to ask :)

[kxygk]

Sorry i disappeared for a bit. Got a nasty cold for a few days there :))

Thank you so much for weighing in @wilkerlucio . My next step was to use the Viz thing to look in to things more - b/c like you describe and i was starting to suspect, there are other things going on in the Runner other than drawing up the plan - and there is no real easy way to avoid all the luggage that comes with the Pathom model

I think dropping back to "do more calculations by hand to make it faster" sensible if you can hide some parts safely. Like if you have some index for generation plots, some parts maybe can be wrapped up in local-only functions that the user doesn't touch.

IDEA 1

I think here that inverts the architecture a bit. I can have some ::cartesian-points-vec on the input and then work with those. Each point would be opaque to Pathom's engine, but the resolver would know how to run across, unpack the points and do the transformation.

When you wanted use this new library/index you'd have some ::my-fancy-dataset that would encapsule say a ::cartesian-points-vec and you prompt for the ::my-fancy-dataset's ::angular-average

But this sort of left me feeling like I'm losing a lot of flexiblity in the API.

IDEA 2

I then thought.. you can actually preserve the current pattern (mostly) by just vectorizing everything (this is actually basically the initial column approach in a sense)

::xs ::ys ::radii ::angles etc.

The resolvers are identical except they basically mapv over all the inputs
(you can add a check that the inputs match in length)

it's a bit goofy that "a point" now gets exploded across several attribute vectors, but it seems optimal this way - and the engine runs once.

The only downside is that you can't sort/filter/split your data vectors while preserving cached values, but that's probably a manageable downside (maybe to revisit later :)) )

I guess in the big picture it'd be a bit weird some things are vectorized while others aren't (and you'll have to manually manage parallelization here). I have to actually use this in a big project and see how it all plays out first :))) too early to speculate

Thank you both for helping me think about this topic deeper !

[jacobobryant]

Glad you're making some progress! Random thought; in python I figure you'd probably use a pandas dataframe for this kind of thing (and probably you'd compute all the derived data up front?). Might be worth thinking about if that approach does everything you need it to? I'm not too familiar with the data science landscape in clojure, so not sure if we have anything like dataframes yet.

Also re: this:

My concern there is it's unclear how to maintain the resolver caches. [...]

If you want to use long-lived resolver caches, you could look into using something fancier than a regular atom. e.g. core.cache has an LRU cache implementation: https://github.com/clojure/core.cache

[kxygk]

Is the atom cache only keeping the results of the last call?

And yeah, in many situations you can precompute values in a table for sure :)) (in Clojure its generally done with tech.ml.dataset) which is what I do now when I need to use numeric data. The example was largely me trying to think of the simplest example with 2 layers (point level resolvers and aggregate level resolvers ie averages).

However tables are a bit of a problematic as an "API format" or thing to pass around. For instance sometimes I may have some linear algebra libraries and plotting libraries. It's a bit awkward to pass in tables. But resolvers/attributes work very nicely here. You can have some plotting that extracts some columns are your x-y values and maybe colors things by some 3rd value and sets the shape based on some aggregate attribute. The aggregate attribute maybe was already computed earlier in some linear algebra step where you were fitting some line for instance

I guess my next experiment will be to try to hook this up with cljfx and see if it works for statemanagment in a GUI. I think Pathom provides a more sophisticated and decoupled solution than the subscription model they use

Then I'll start converting a larger GUI application I have working right now

[kxygk]

Well I took a stab at CLFX + Pathom. Thanks for your help guys. I'm very happy with the result

https://github.com/kxygk/ednless/blob/master/pathomfx.clj

https://old.reddit.com/r/Clojure/comments/1rv8d27/little_demo_of_pathom_and_cljfx_working_together/?