Cache more blockchain state so as to offload zcashd (performance)

[LarryRuane]

zcash/zcash#5957 (comment)

This is also partly a lightwalletd problem. It's not great that zcashd's global locking (inherited from upstream) can cause these kinds of slowdowns. However, lightwalletd was originally designed as a caching layer so that it could scale to large numbers of clients, and not be overly affected by this kind of issue. The fact that for the newer transparent gRPC methods it only acts as a direct passthrough to the zcashd RPC, without any caching or timeout handling, is a problem that should be solved there. In particular, if lightwalletd built and maintained the address index itself (from the same stream of blocks that it builds its compact block cache), then it would be completely impervious to this kind of problem.

[str4d]

2b73689 is the commit that changed lightwalletd from storing its own information (per its original design, with SQLite as the example backing DB), to always requesting data from zcashd where possible (e.g. GetTransaction changed to calling getrawtransaction).

- lightwalletd consists of three loosely coupled components: an "ingester", a "frontend", and an arbitrary storage layer (such as a SQL database) that connects the two. The ingester receives raw block data, parses out the transactions and block metadata, then stores them in a format convenient for the frontend to serve to clients. Thus, these components can operate and scale independently of each other and are connected only by a shared storage convention.
+ lightwalletd is a backend service that provides a bandwidth-efficient interface to the Zcash blockchain for mobile and other wallets, such as [Zecwallet](https://github.com/adityapk00/zecwallet-lite-lib).

This change effectively bound lightwalletd's performance to that of the zcashd RPC, giving rise to problems like zcash/zcash#5957. I'd recommend logically reverting this commit (directly reverting it is probably too hard given over two years of further development), and then building the address indexes etc. on top of the re-introduced lightwalletd storage layer.

[str4d]

Note that in the case of zcash/zcash#5957, caching does not solve the performance problem, because users are not going to be querying lightwalletd with the same transparent addresses, so we'd just get both constant cache misses, and unused cache bloat.

To properly decouple lightwalletd's performance from zcashd's RPC, lightwalletd needs to be reverted to the ingestor <-> storage layer <-> frontend model it originally had, with the address index built and maintained proactively by the ingestor so it pays the cost of zcashd RPC slowness once, up-front, instead of on every client request.

[LarryRuane]

Anyone reading this should be sure to also read the last several comments in zcash/zcash PR 5957. To make sure we're on the same page, lightwalletd does still have the ingestor <-> storage layer <-> frontend model when it comes to blocks. PR #149 combined the ingestor and the frontend into a single process, rather than being two separate executables. The other thing it did was remove transaction caching. I still think this was a good change, because caching transactions require almost as much state as the entire blockchain, currently about 30 gb, compared with 500 mb for the lightwalletd compact blocks cache.

But the main reason it was probably okay to drop transaction caching is that transactions are always requested by the wallets individually, on-demand from users, rather than in batches. This differs from compact blocks, which are often requested at a very high rate. A wallet's initial sync would be much slower if lightwalletd didn't cache compact blocks. Whereas, the extra delay in fetching a single transaction (due to having to request it from zcashd) probably isn't noticeable to the human user.

It is true that lightwalletd doesn't cache the t-address to txid-list mapping (the GetTaddressTxids gRPC causes lightwalletd to synchronously call zcashd's getaddresstxids RPC), but the original design didn't cache this mapping either. And this turned out not to be the performance problem after all.

It's also worth noting that the more caching lightwalletd does, the more complex reorg handling becomes. There is currently a fair amount of complexity just for blocks.

Assuming that my thinking here is correct (and it certainly may not be! I was not around during the initial lightwalletd design discussions), I'm willing to leave this ticket open, but it's unclear to me what priority to give it, or if we want to make any changes at all.

[LarryRuane]

I don't think we have sufficient motivation to work on this in the foreseeable future (see my most recent comment), so closing at least for now.