Skip to content

add ability to mock zcashd rpc calls for testing, add tests #164

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
LarryRuane merged 1 commit into from
Jan 31, 2020
Merged

add ability to mock zcashd rpc calls for testing, add tests #164

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
10 changes: 5 additions & 5 deletions CONTRIBUTING.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -19,7 +19,7 @@ Note: Please replace `your_username`, with your actual GitHub username
git clone git@github.com:your_username/lightwalletd.git
cd lightwalletd
git remote set-url origin git@github.com:your_username/lightwalletd.git
git remote add upstream git@github.com:zcash-hackworks/lightwalletd.git
git remote add upstream git@github.com:zcash/lightwalletd.git
git remote set-url --push upstream DISABLED
git fetch upstream
git branch -u upstream/master master
Expand All @@ -39,7 +39,7 @@ After issuing the above commands, your `.git/config` file should look similar to
remote = upstream
merge = refs/heads/master
[remote "upstream"]
url = git@github.com:zcash-hackworks/lightalletd.git
url = git@github.com:zcash/lightalletd.git
fetch = +refs/heads/*:refs/remotes/upstream/*
pushurl = DISABLED
```
Expand All @@ -64,7 +64,7 @@ To checkout an existing branch (assuming you are in `lightwalletd` directory):
```bash
git checkout [existing_branch_name]
```
If you are fixing a bug or implementing a new feature, you likely will want to create a new branch. If you are reviewing code or working on existing branches, you likely will checkout an existing branch. To view the list of current Lightwalletd GitHub issues, click [here](https://github.com/zcash-hackworks/lightwalletd/issues).
If you are fixing a bug or implementing a new feature, you likely will want to create a new branch. If you are reviewing code or working on existing branches, you likely will checkout an existing branch. To view the list of current Lightwalletd GitHub issues, click [here](https://github.com/zcash/lightwalletd/issues).

## Make & Commit Changes
If you have created a new branch or checked out an existing one, it is time to make changes to your local source code. Below are some formalities for commits:
Expand All @@ -83,7 +83,7 @@ git remote -v
```bash
origin git@github.com:your_username/lightwalletd.git (fetch)
origin git@github.com:your_username/lightwalletd.git (push)
upstream git@github.com:zcash-hackworks/lightwalletd.git (fetch)
upstream git@github.com:zcash/lightwalletd.git (fetch)
upstream DISABLED (push)
```
This output should be consistent with your `.git/config`:
Expand All @@ -96,7 +96,7 @@ This output should be consistent with your `.git/config`:
url = git@github.com:your_username/lightwalletd.git
fetch = +refs/heads/*:refs/remotes/origin/*
[remote "upstream"]
url = git@github.com:zcash-hackworks/lightwalletd.git
url = git@github.com:zcash/lightwalletd.git
fetch = +refs/heads/*:refs/remotes/upstream/*
pushurl = DISABLED
```
Expand Down
6 changes: 3 additions & 3 deletions Dockerfile
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -43,8 +43,8 @@
# Create layer in case you want to modify local lightwalletd code
FROM golang:1.13 AS lightwalletd_base

ADD . /go/src/github.com/zcash-hackworks/lightwalletd
WORKDIR /go/src/github.com/zcash-hackworks/lightwalletd
ADD . /go/src/github.com/zcash/lightwalletd
WORKDIR /go/src/github.com/zcash/lightwalletd
RUN make \
&& /usr/bin/install -c ./server /usr/bin/

Expand All @@ -60,4 +60,4 @@ USER $LWD_USER
WORKDIR /srv/$LWD_USER

ENTRYPOINT ["server"]
CMD ["--help"]
CMD ["--help"]
20 changes: 11 additions & 9 deletions Makefile
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -20,34 +20,36 @@ all: build

# Lint golang files
lint:
@golint -set_exit_status
golint -set_exit_status

@LarryRuane LarryRuane Jan 29, 2020

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I find it much more helpful to see what's actually happening


show_tests:
@echo ${GO_TEST_FILES}

# Run unittests
test:
@go test -v -coverprofile=coverage.txt -covermode=atomic ./...
go test -v ./...

@LarryRuane LarryRuane Jan 29, 2020

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

there is no need for atomic, and there's a separate make target for coverage


# Run data race detector
race:
GO111MODULE=on CGO_ENABLED=1 go test -v -race -short ./...

# Run memory sanitizer (need to ensure proper build flag is set)
msan:
@go test -v -msan -short ${GO_TEST_FILES}
go test -v -msan -short ${GO_TEST_FILES}

# Generate global code coverage report, ignore generated *.pb.go files

# Generate global code coverage report
coverage:
@go test -coverprofile=coverage.out -covermode=atomic ./...
go test -coverprofile=coverage.out ./...
sed -i '/\.pb\.go/d' coverage.out

@LarryRuane LarryRuane Jan 29, 2020

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

the .pb. files are generated (protobuf), and their code coverage is not relevant


# Generate code coverage report
coverage_report:
@go tool cover -func=coverage.out
coverage_report: coverage
go tool cover -func=coverage.out

# Generate code coverage report in HTML
coverage_html:
@go tool cover -html=coverage.out -o coverage.html
coverage_html: coverage
go tool cover -html=coverage.out

# Generate documents
docs:
Expand Down
4 changes: 2 additions & 2 deletions README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -17,15 +17,15 @@ The Lightwalletd Server is experimental and a work in progress. Use it at your o

# Overview

[lightwalletd](https://github.com/zcash-hackworks/lightwalletd) is a backend service that provides a bandwidth-efficient interface to the Zcash blockchain. Currently, lightwalletd supports the Sapling protocol version as its primary concern. The intended purpose of lightwalletd is to support the development of mobile-friendly shielded light wallets.
[lightwalletd](https://github.com/zcash/lightwalletd) is a backend service that provides a bandwidth-efficient interface to the Zcash blockchain. Currently, lightwalletd supports the Sapling protocol version as its primary concern. The intended purpose of lightwalletd is to support the development of mobile-friendly shielded light wallets.

lightwalletd is a backend service that provides a bandwidth-efficient interface to the Zcash blockchain for mobile and other wallets, such as [Zecwallet](https://github.com/adityapk00/zecwallet-lite-lib).

Lightwalletd has not yet undergone audits or been subject to rigorous testing. It lacks some affordances necessary for production-level reliability. We do not recommend using it to handle customer funds at this time (October 2019).

To view status of [CI pipeline](https://gitlab.com/mdr0id/lightwalletd/pipelines)

To view detailed [Codecov](https://codecov.io/gh/zcash-hackworks/lightwalletd) report
To view detailed [Codecov](https://codecov.io/gh/zcash/lightwalletd) report

# Local/Developer docker-compose Usage

Expand Down
91 changes: 46 additions & 45 deletions cmd/server/main.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -16,12 +16,11 @@ import (
"google.golang.org/grpc/peer"
"google.golang.org/grpc/reflection"

"github.com/zcash-hackworks/lightwalletd/common"
"github.com/zcash-hackworks/lightwalletd/frontend"
"github.com/zcash-hackworks/lightwalletd/walletrpc"
"github.com/zcash/lightwalletd/common"
"github.com/zcash/lightwalletd/frontend"
"github.com/zcash/lightwalletd/walletrpc"
)

var log *logrus.Entry
var logger = logrus.New()

func init() {
Expand All @@ -32,10 +31,10 @@ func init() {
})

onexit := func() {
fmt.Printf("Lightwalletd died with a Fatal error. Check logfile for details.\n")
fmt.Println("Lightwalletd died with a Fatal error. Check logfile for details.")
}

log = logger.WithFields(logrus.Fields{
common.Log = logger.WithFields(logrus.Fields{

@LarryRuane LarryRuane Jan 29, 2020

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

There is no reason the logging object can't be a global variable, and doing that simplifies many of the function signatures, and makes unit tests easier to write.

"app": "frontend-grpc",
})

Expand All @@ -48,12 +47,7 @@ func LoggingInterceptor() grpc.ServerOption {
return grpc.UnaryInterceptor(logInterceptor)
}

func logInterceptor(
ctx context.Context,
req interface{},
info *grpc.UnaryServerInfo,
handler grpc.UnaryHandler,
) (interface{}, error) {
func logInterceptor(ctx context.Context, req interface{}, info *grpc.UnaryServerInfo, handler grpc.UnaryHandler) (interface{}, error) {
reqLog := loggerFromContext(ctx)
start := time.Now()

Expand All @@ -77,9 +71,9 @@ func logInterceptor(
func loggerFromContext(ctx context.Context) *logrus.Entry {
// TODO: anonymize the addresses. cryptopan?
if peerInfo, ok := peer.FromContext(ctx); ok {
return log.WithFields(logrus.Fields{"peer_addr": peerInfo.Addr})
return common.Log.WithFields(logrus.Fields{"peer_addr": peerInfo.Addr})
}
return log.WithFields(logrus.Fields{"peer_addr": "unknown"})
return common.Log.WithFields(logrus.Fields{"peer_addr": "unknown"})
}

type Options struct {
Expand All @@ -89,7 +83,6 @@ type Options struct {
logLevel uint64 `json:"log_level,omitempty"`
logPath string `json:"log_file,omitempty"`
zcashConfPath string `json:"zcash_conf,omitempty"`
veryInsecure bool `json:"very_insecure,omitempty"`
cacheSize int `json:"cache_size,omitempty"`
wantVersion bool
}
Expand Down Expand Up @@ -122,6 +115,23 @@ func main() {
return
}

// production (unlike unit tests) use the real sleep function
common.Sleep = time.Sleep

if opts.logPath != "" {
// instead write parsable logs for logstash/splunk/etc
output, err := os.OpenFile(opts.logPath, os.O_APPEND|os.O_CREATE|os.O_WRONLY, 0644)
if err != nil {
os.Stderr.WriteString(fmt.Sprintf("Cannot open log file %s: %v\n",
opts.logPath, err))
os.Exit(1)
}
defer output.Close()
logger.SetOutput(output)
logger.SetFormatter(&logrus.JSONFormatter{})
}

@LarryRuane LarryRuane Jan 17, 2020 •
edited
Loading

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is moved, makes sense to open the log file as soon as possible. Slight change (improvement) in behavior -- if you specify -log-file '', the process will log to stdout. (Without these changes, it's not possible to log to stdout.)

logger.SetLevel(logrus.Level(opts.logLevel))

filesThatShouldExist := []string{
opts.zcashConfPath,
}
Expand All @@ -134,41 +144,27 @@ func main() {

for _, filename := range filesThatShouldExist {
if !fileExists(filename) {
os.Stderr.WriteString(fmt.Sprintf("\n ** File does not exist: %s\n\n", filename))
flag.Usage()
common.Log.WithFields(logrus.Fields{
"filename": filename,
}).Error("cannot open required file")
os.Stderr.WriteString("Cannot open required file: " + filename + "\n")
os.Exit(1)
}
}

if opts.logPath != "" {
// instead write parsable logs for logstash/splunk/etc
output, err := os.OpenFile(opts.logPath, os.O_APPEND|os.O_CREATE|os.O_WRONLY, 0644)
if err != nil {
log.WithFields(logrus.Fields{
"error": err,
"path": opts.logPath,
}).Fatal("couldn't open log file")
}
defer output.Close()
logger.SetOutput(output)
logger.SetFormatter(&logrus.JSONFormatter{})
}

logger.SetLevel(logrus.Level(opts.logLevel))

// gRPC initialization
var server *grpc.Server
var transportCreds credentials.TransportCredentials
var err error

if (opts.tlsCertPath == "") && (opts.tlsKeyPath == "") {
log.Warning("Certificate and key not provided, generating self signed values")
common.Log.Warning("Certificate and key not provided, generating self signed values")
tlsCert := common.GenerateCerts()
transportCreds = credentials.NewServerTLSFromCert(tlsCert)
} else {
transportCreds, err = credentials.NewServerTLSFromFile(opts.tlsCertPath, opts.tlsKeyPath)
if err != nil {
log.WithFields(logrus.Fields{
common.Log.WithFields(logrus.Fields{
"cert_file": opts.tlsCertPath,
"key_path": opts.tlsKeyPath,
"error": err,
Expand All @@ -188,15 +184,18 @@ func main() {

rpcClient, err := frontend.NewZRPCFromConf(opts.zcashConfPath)
if err != nil {
log.WithFields(logrus.Fields{
common.Log.WithFields(logrus.Fields{
"error": err,
}).Fatal("setting up RPC connection to zcashd")
}

// indirect function for test mocking (so unit tests can talk to stub functions)
common.RawRequest = rpcClient.RawRequest

// Get the sapling activation height from the RPC
// (this first RPC also verifies that we can communicate with zcashd)
saplingHeight, blockHeight, chainName, branchID := common.GetSaplingInfo(rpcClient, log)
log.Info("Got sapling height ", saplingHeight, " chain ", chainName, " branchID ", branchID)
saplingHeight, blockHeight, chainName, branchID := common.GetSaplingInfo()
common.Log.Info("Got sapling height ", saplingHeight, " chain ", chainName, " branchID ", branchID)

// Initialize the cache
cache := common.NewBlockCache(opts.cacheSize)
Expand All @@ -207,12 +206,13 @@ func main() {
cacheStart = saplingHeight
}

go common.BlockIngestor(rpcClient, cache, log, cacheStart)
// The last argument, repetition count, is only nonzero for testing
go common.BlockIngestor(cache, cacheStart, 0)

// Compact transaction service initialization
service, err := frontend.NewLwdStreamer(rpcClient, cache, log)
service, err := frontend.NewLwdStreamer(cache)
if err != nil {
log.WithFields(logrus.Fields{
common.Log.WithFields(logrus.Fields{
"error": err,
}).Fatal("couldn't create backend")
}
Expand All @@ -223,7 +223,7 @@ func main() {
// Start listening
listener, err := net.Listen("tcp", opts.bindAddr)
if err != nil {
log.WithFields(logrus.Fields{
common.Log.WithFields(logrus.Fields{
"bind_addr": opts.bindAddr,
"error": err,
}).Fatal("couldn't create listener")
Expand All @@ -234,17 +234,18 @@ func main() {
signal.Notify(signals, syscall.SIGINT, syscall.SIGTERM)
go func() {
s := <-signals
log.WithFields(logrus.Fields{
common.Log.WithFields(logrus.Fields{
"signal": s.String(),
}).Info("caught signal, stopping gRPC server")
os.Stderr.WriteString("Caught signal: " + s.String() + "\n")
os.Exit(1)
}()

log.Infof("Starting gRPC server on %s", opts.bindAddr)
common.Log.Infof("Starting gRPC server on %s", opts.bindAddr)

err = server.Serve(listener)
if err != nil {
log.WithFields(logrus.Fields{
common.Log.WithFields(logrus.Fields{
"error": err,
}).Fatal("gRPC server exited")
}
Expand Down
Loading