replace SQL storage with in-memory cache

[LarryRuane]

Remove the storage/ directory (sqlite3.go) in favor of storing the most recent N blocks in memory. Remove the transaction table in favor of making queries directly to zcashd (no caching). This is made performant enough by the usage of Insight indices (addressindex, enabled by zcashd --insightexplroer).

If a request for a block from before the cache, the request is sent to zcashd directly. The result is returned to the client but not cached. Possible security issue here, are we exposing interest in blocks to network observes at the lightwalletd and zcashd levels?

Also, remove the cmd/ingest directory, so that there is no ingest executable (leaving only the server executable). Move the block ingestor functionality into the server, and handle reorgs there. This needs careful review.

Criteria:

• Security Review of existing code
• two code reviews

[LarryRuane]

I made some performance measurements of Aditya's change to cache blocks in memory. I found a fairly large block (mainnet height 627635). If the block is in cache (by default, the most recent 40k blocks), the real (elapsed) time to request it from the client is 0.425 ms (everything running locally). A request for a similar-size block that's not in cache (older than 40k blocks ago, mainnet height 203105), which requests the block from the local zcashd, takes 398 ms. On smaller blocks, the difference isn't so great; with empty blocks, the difference is about a factor of 10.

Using our current code, with SQL, the time for any block (since they're all in SQL) is 0.660 ms. But that measurement is probably optimistic, because the block is already in the operating system's cache, or sqlite's cache (if it has one), and isn't being read from disk. If it needs to be read from disk, that will be significantly slower (I didn't have a way to measure that).

I also did some executable space measurements on testnet.

The current version (using sqlite) has two executables. The ingestor uses about 900 mb (memory image size), and the server uses about 1.5 gb. So that's 2.4 gb total.
The new version (Aditya's) is just one executable, and with the default cache size (40k blocks) uses 1.1 gb. (I think maybe the reason the SQL version is larger is that it must link in all the sqlite3 code. But that's just a guess.)

I reduce the cache size to essentially zero, and the executable size is 700 mb. I'm giving you rounded numbers, obviously, but using more precise numbers, the difference is 370 mb. This must be the cache alone. That means that each cached block uses 9.25 kbytes, which seems to be in the right ballpark for testnet since most blocks are empty; the block explorer shows most blocks are 1654 bytes (these are empty), and a few in the 2k - 10k range. There's some memory overhead in the map data structure that implements the cache (that gives the ability to look up blocks quickly).

[LarryRuane]

In case anyone would like to replicate my performance measurements, I installed grpcurl by following the instructions at https://github.com/fullstorydev/grpcurl

go get github.com/fullstorydev/grpcurl
go install github.com/fullstorydev/grpcurl/cmd/grpcurl

I modified its source so that a given RPC will be repeated a given number of times, by applying this patch that I developed: https://gist.github.com/LarryRuane/d3424159333f64bc63237088362b7edd

Then I started lightwalletd, and ran the following (for example):

time go run cmd/grpcurl/grpcurl.go -repeat 800 -plaintext -d '{"height":298695}' localhost:9067 cash.z.wallet.sdk.rpc.CompactTxStreamer/GetBlock | wc -l

(The wc -l is just to consume the output while showing that the expected amount of output is produced.)

[LarryRuane]

@gmale, a little more on performance: I increased the block cache size from the default (40k) to the maximum (sapling activation height to current tip, about 224k blocks), and the time to download that number of blocks (sync time) was 32 minutes on mainnet. This is still much faster than the sync time of the SQL version (which, if I recall, is many hours).

But it's important to note that this sync happens every time lightwalletd is started, since (with Aditya's design) it has no persistent storage.

After downloading these blocks, memory usage is 1.3 gb (1263 mb, to be exact), compared with 1.1 gb when caching the default 40k blocks. So it may actually be acceptable to cache all blocks since sapling. We'll just have to get more experience to decide what the default should be (but note that the block cache size can be set using the -cache-size argument). These compact blocks don't contain much data:

$ grpcurl -plaintext -d '{"height":640508}' localhost:9067 cash.z.wallet.sdk.rpc.CompactTxStreamer/GetBlock
{
  "height": "640508",
  "hash": "OAPH/5sfShwgrz3y+iNqDNZTY6wrKxRFwQuwAAAAAAA=",
  "prevHash": "bqInfb1C+FYiiMzBWuAf/PmXLgLvXfxNFl5HAAAAAAA=",
  "time": 1574128375
}

This sync (block download) time shouldn't even matter, in that it should be (functionally) zero, because during the sync, if a wallet requests a block that hasn't been downloaded yet, the lightwalletd should just fetch it immediately, while the (separate) download thread proceeds in the background. I think this general technique is called cache acceleration. When the background thread reaches that block height, it simply skips that block since it's already present.

This will require a small code change, as currently, if the wallet tries to access a block that hasn't been downloaded yet, this happens:

$ grpcurl -plaintext -d '{"height":643508}' localhost:9067 cash.z.wallet.sdk.rpc.CompactTxStreamer/GetBlock # lightwalletd
ERROR:
  Code: Unknown
  Message: Block requested is newer than latest block. Requested: 643508 Latest: 604181

(The error message is very misleading since that's not really the latest block in the blockchain during sync; it's only the latest block that the lightwalletd knows about.)

[LarryRuane]

Fixed by #149.