Raw Calls - What is necessary?

[skilesare]

I've recently been frustrated by Motoko's inability to do raw calls as seen in the cycle wallet(https://github.com/dfinity/cycles-wallet/blob/5725370020593bd744ddbc589c117d76d6ca8ae4/wallet/src/lib.rs#L625).

To be able to do this I would imagine motoko would need to:

• expose the raw call
• enable some kind of type reflection
• formalize some kind of serialization/deserialization

I've seen some mention of these in some of the open/closed meta-issue issues here and I thought I'd stand on top of the building yelling to hopefully get these some attention. :)

What I don't want to do is to have to create a function on every motoko pass through canister called __forwardCall(principal: Principal; method: text, args: [Nat8]/WorkSpace/[DataZones]) and have to have every canister handle a __handleForwardCall that deserializs all the parameters manually into known types and organize the community to universally add these to their canisters. perhaps there is a compiler way to automate this kind of thing? Seems overkill when the underlying functions can just be exposed some how.

[crusso]

If all you want is the ability to dynamically invoke a method by name with an already serialized blob and get the raw (undeserialized) blob back asynchronously, then I think we can actually provide that fairly easily.

call_raw : (Principal, Text, Blob) -> async Blob

which should be enough to implement the call function you mention in the Rust code.

I'm not sure what the cycles argument is used for in Rust, but I imagine it is specifying the number of cycles to deduct from the wallet and could probably be handled separately by adding cycles to the call.

I think the issues you mentioned might have been trying to solve a much more general and harder problem: how to support candid functions and shared functions with polymorphic types which is indeed much trickier.

I guess it depends on what you are trying to achieve in your applications. If it's just implementing this particular wallet function, I think that's achievable. But more ambitious stuff might be tricky indeed.

@rossberg?

[skilesare]

Yes,

call_raw : (Principal, Text, Blob) -> async Blob

gets me 80% of the way to where I want to be and should 100% enable building a motoko wallet canister(or similar pattern) actor.

There is a huge need for some kind of general-purpose candid interpreter/serializer/deserializer, but I think that is a much bigger issue to tackle.

I've been thinking about an axiom along the lines of "any canister that could be built with rust should be able to be built with motoko" as a kind of guiding principle that I'd love to get the team's feedback on. I would imagine there exists a finite set of features that keep this from being true at the moment and it would be interesting to catalog them. "call_raw" is on that list. Most of the list likely revolves around low-level system stuff that should be ignored/abstracted by most devs, but should be accessible as the need arises. I'm doing work on certificates and I'm thinking that some of that stuff may be on the list as well. Looks like they are cbor encoded? Another topic perhaps, but I keep running into these things and I'd like to be proactive in cataloging them so the community and team can coordinate on priorities.

I've really been enjoying the language over the last couple of weeks and feel like I can get a lot done with it now. Thanks for engaging!

[skilesare]

I guess I'll also add query_raw to the list as a request as well. I can see the need for a general-purpose caching canister that can be configured to query a bunch of other canisters to have data 'at the ready' in a way that the caching canister shouldn't have to know about the structure of the data. The canister that uses the caching canister can take the responsibility of deserializing into known types.

[BrantBrown]

It is necessary to implement in motoko.

call(Principal,Text,args)
call_raw(Principal, Text,Blob) 

[zire]

@chenyan-dfinity Hey Yan, could you take a look at this issue? Some developers are watching this one closely and waiting for a solution. Thanks!

[chenyan-dfinity]

I think what's needed here is a way of transferring generic data over the wire. See related discussion dfinity/candid#245.

Meanwhile, as a stop gap measure, it's always possible to use both Rust and Motoko in your project, if certain feature is not available in one of the languages.

[skilesare]

I think what's needed here is a way of transferring generic data over the wire. See related discussion dfinity/candid#245.

This would be wonderful, but what we need immediately is just a call_raw(principal, method, blob): Blob that gives the user the ability to handle whatever comes their way or to just pass the call through to another canister. This maximises optionality.

Meanwhile, as a stop gap measure, it's always possible to use both Rust and Motoko in your project, if certain feature is not available in one of the languages.

Is it? I've never seen an example that imports a rust library into motoko! I understood that is was a coming soon feature. We could have a motoko canister talked to a rust canister, but that adds a x-canister call which is slow and can't do a query.

[chenyan-dfinity]

We could have a motoko canister talked to a rust canister, but that adds a x-canister call which is slow and can't do a query.

Yes, they can communicate at the canister level, and it will be an inter-canister call. Not ideal.

Motoko playground is using Rust canister to parse and transform Wasm module: https://github.com/dfinity/motoko-playground/blob/main/service/pool/Main.mo#L109, and it doesn't add much latency in practice.

[BrantBrown]

@chenyan-dfinity
Motoko x-canister calls are only called in a predefined way , we need a general way to get more flexibility
For example, DFT : transferAndCall,we need to support calling any method of any canister with any parameters.We support it through ic-cdk's api::call, motoko does not have an api like api::call.

It is necessary to implement in motoko.

call(Principal,Text,args)
call_raw(Principal, Text,Blob) 

[skilesare]

I'm popping this back up another two months later. @matthewhammer mentioned today that you all @rossberg, @crusso, etc discussed it in a meeting and the it wasn't a priority at the moment. I was kind of bummed so I went back to the drawing board. I spent a few minutes so messing around with the following:

** Edited **

import Debug "mo:base/Debug";

actor Echo1 {

  var message : Text = "";

  // Say the given phase.
  public func say(phrase : Text) : async Text {
    Debug.print(debug_show(phrase));
    message := phrase;
    return phrase;
  };

  public func get() : async Text{
    return message;
  }
};

import Debug "mo:base/Debug";

actor Echo2 {

  var message : Text = "";

  public func say(phrase : Text) : async Text {
    Debug.print(debug_show(phrase));
    message := phrase;
    return phrase;
  };

  public func get() : async Text{
    return message;
  }
};

I thought maybe I'd been a idiot for a few months and it might be as simple as telling the candid type definition in Echo2 that a function took a blob and then the receiving function in Echo 1 would say "hey...this is binary data, I bet it is in my candid syntax...I'll parse it as such"...but no luck. It seems that the something is putting the IDL into the message and then sending it across and then when it gets to Echo 1 if fails with The Replica returned an error: code 4, message: "IC0503: Canister vo5te-2aaaa-aaaaa-aaazq-cai trapped explicitly: IDL error: unexpected IDL type when parsing Text". Sigh.

After that spent about 2 hours in the motoko repo trying to ferret out if there was some lines here or there I could change that would get the call_raw exposed as a Prim functor or something. It is like right there...I can see ICCallPrim places and I can see the Prim.mo file, but OCaml, Haskell, and what ever else is going on here is so foreign to me that I might as well be trying to bang out Shakespeare with a room full of monkeys.

But I need this functionality so I'm going to keep banging my head against a wall. I could go into a fairly long diatribe about why we need it, but I'm starting to think that maybe I'm just missing something. In my mind this single feature is holding Motoko back from actually being a lingua franca for the IC.

What am I missing?

I really don't want to shift all my attention to another sdk, but we need something simple and straight forward that ALL programs can be written in. I've got a lot invested in Motoko right now, but the reality is I can't a build a wallet for the IC right now. And I really...really... need to build a wallet and some other passthrough proxy like services. I need to support my users interacting with canisters and standards that don't exist yet through an identity canister...or wallet...or proxy...call it what you will. I want to hold assets, data, and permissions and potential send those to services that I don't know about at the time the code is installed. This does not seem like an optional thing it is essential and already caused me to go chasing less interesting problems because the really valuable ones can't be solved with motoko right now.

The other option here is to convince the community that every public function needs a my function_variant(params : VariantWrapper) : async VariantWrapper{} that wraps all types in a corresponding variant. That just seems to undo most of what motoko was trying to do. I still want to use proper motoko for everything...there are just a few instances where I really need break through to serve the needs of my users.

So as an exercise...because maybe I have some resources to do so....if I wanted to expose this functionality to motoko...where would I make changes? If I was going to fork the code and make it so that an array of bytes could be sent to a motoko canister without strict IDL definitions and the other canister would assume it was in the right format and take in the message...I'm fin if it throws if the format isn't right....how would I do that? What language would I need to know and what files would I need to look in? Is it even possible to fork and implement, or is there something in the way the IC works that prevents it...Rust can do it so I don't think it is at the IC level.

[chenyan-dfinity]

I'm lost. The code for Echo1 and Echo2 looks exactly the same.

[rossberg]

Same here, some c&p error with your example?

[skilesare]

Sorry...yes...a copy paste error...up too late.

import Debug "mo:base/Debug";

actor Echo1 {

  var message : Text = "";

  // Say the given phase.
  public func say(phrase : Text) : async Text {
    Debug.print(debug_show(phrase));
    message := phrase;
    return phrase;
  };

  public func get() : async Text{
    return message;
  }
};

import Debug "mo:base/Debug";
import Blob "mo:base/Blob";
actor Echo2 {


  type first = actor {
    say : (Blob) -> async Blob;
    get : () -> async Text;
  };

  public func testSay() : async (){
    let thisFirst : first = actor("vo5te-2aaaa-aaaaa-aaazq-cai");
    Debug.print(debug_show(await thisFirst.say(Blob.fromArray([1:Nat8,2:Nat8, 3:Nat8, 4:Nat8]))));
    Debug.print(debug_show(await thisFirst.get()));
  };



};

[crusso]

(summarising from another slack conversation)

This is what we want to implement in Motoko, right?

Candid:

https://github.com/dfinity/cycles-wallet/blob/9ef38bb7cd0fe17cda749bf8e9bbec5723da0e95/wallet/src/lib.did#L150

Rust Code:

https://github.com/dfinity/cycles-wallet/blob/9ef38bb7cd0fe17cda749bf8e9bbec5723da0e95/wallet/src/lib.rs#L666

As I said above, I think it just requires a dynamic call instruction that take a principal, method name and blob and returns the raw binary response as an async blob. Needs compiler support but no candid extension AFAICT.

If you want Motoko to do more than just pass on a given blob and return a given blob, then we would indeed need do quite a lot more work. But for the scenario of the particular method above, I see no real difficulty and no need for a Candid extension and additional serialization prims.

(The reason this would need some compiler support rather than just a new primitive is that the return type is asynchronous, so I would probably need to extend the async/await pipeline to translate the function appropriately.)