The fragile way of using Tezos

Last year, I wrote about oracles and eventually about an automated insurance system using a weather oracle. Oracle is just a fancy name for a server that publishes information to the blockchain. The system also had a server that offers people insurance policies. I originally built these servers to interact with the blockchain by executing the command line tezos-client included with Tezos. At the time, it was a reasonable choice because the RPC API wasn’t finished yet and was not stable. Over time however, it became clear that a more robust way of working with Tezos is needed for our servers. This article will provide some background details and show how our Go package, called tzsuite, can be used to build robust systems using Tezos.

If you’ve read the previous articles, you might remember that setting up the insurance smart contracts required several steps, including publishing 4 contracts and signing a message with the owner’s keys. Afterwards, the end-users just need to call the main contract (provider), but to test changes to the contracts, we still needed to go through the setup procedure. To make this less annoying, I wrote a script that executes tezos-client and does what’s needed. This proved to be too fragile, as changes to tezos-client made it impossible to send more than 1 operation per block.

The simple workaround was to wait for time_between_blocks seconds between each operation, but this breaks on a number of occasions. The most common reason is that a baker misses a block and a baker with lower priority bakes it later. Another plausible scenario is that there are a lot of operations and our operation doesn’t get included in a block right away. We can try waiting a bit longer, but we are always just hoping that our operations were included properly.

In contrast, using the RPC of a Tezos node, we can provide a set of operations (a batch) to be evaluated together. This lets us make more transactions in a single block, or even publish multiple contracts. I wanted a very simple API for sending operations, which required keeping an internal state, dropping invalid operations so that they don’t make the whole batch fail, waiting for the next block when the maximum operation size is reached, and so on. Once the code got complex enough, I moved it into a separate package. Using the RPC also lets us do things we couldn’t do before – we can notify the caller when an operation is published, when it gets included in a block or even provide the addresses of newly published contracts.

Since the RPC interaction package is written in Go, we can use channels to send information around. This lets us easily make our programs event driven and avoid the timing issues mentioned above. Another benefit is that we can easily use a remote node. We could have configured tezos-client to use a remote node as well, but this way our servers don’t depend on anything besides what’s described in their source code.

Let’s look at the new script that sets up the insurance smart contracts now, as it shows nicely what can be accomplished with this approach. Note that most error handling is not shown for simplicity.

If you haven’t seen Go channels before, there are two main actions you can do with them: send and receive. You can also close them, which is helpful if you are trying to send a simple signal to multiple receivers. In the example below, we spin up a new goroutine and send a simple message to the main program.

// channels 101
numbers := make(chan int)
// define an anonymous function, and run it in a goroutine immediately
go function(ch chan int, i int) {
    ch <- i
}(numbers, 1)
result := <-numbers
fmt.Println("received a number:", result)

The package for using the Tezos RPC is called rpc, so we can call its functions with the rpc. prefix. First, we call New to get a new RPC client. It will hold the state that’s needed to organize operations into batches and more. We can use one of the preset configurations (in this example the preset for Tezos alphanet). The *rpcURL and *seed are supplied at runtime as command line arguments.

ch := make(chan error)
shutdown := make(chan struct{})

r := rpc.New(rpc.ConfAlphanet(*rpcURL), nil, shutdown, ch)

go func(ch chan error, shutdown chan struct{}) {
    err := <-ch
    fmt.Println("error from goroutine:", err)
    close(shutdown)
}(ch, shutdown)

The channels are used to receive errors from the package and to signal that we want to shut down the client. We spin up a goroutine to monitor the error channel and shut down everything if there are any errors.

To pay for all the operations required during the setup, we will use an existing address we control, that has enough funds. We regenerate the keys, using the tzutil package that provides functions for key manipulation. Then we create two new identities to act as the owners of the oracle and the insurance system, respectively.

moneyKeys, err := tzutil.KeysFromStringSeed(*seed)
oracleKeys, err := tzutil.KeysWithoutSeed()
providerKeys, err := tzutil.KeysWithoutSeed()

Most of the operations will be paid for by the moneyKeys, but let’s transfer some funds to the brand new identities in case we need them. The arguments to BatchTransfer mirror the order used by tezos-client which most developers are familiar with: transfer amount from source to destination. The last argument is used to specify parameters when calling smart contracts.

done, err := r.BatchTransfer(moneyKeys, 10000000, moneyKeys.PKH, oracleKeys.PKH, nil)
go func(done <-chan string) {
    opHash := <-done
    fmt.Println("transfer to oracle injected in operation:", opHash)
    blockHash := <-done
    fmt.Println("transfer to oracle included in block:", blockHash)
}(done)

done, err = r.BatchTransfer(moneyKeys, 10000000, moneyKeys.PKH, providerKeys.PKH, nil)
go func(done <-chan string) {
    opHash := <-done
    fmt.Println("transfer to provider injected in operation:", opHash)
    blockHash := <-done
    fmt.Println("transfer to provider included in block:", blockHash)
}(done)

The first returned value is a channel over which we receive notifications about the operation being published, and eventually included in a block. In this example we just log the event and move on.

Next, we publish (originate) two smart contracts – the oracle contract, and a placeholder for the handler of oracle responses. The arguments are given in the same order as when using tezos-client originate contract: the desired manager (owner) of the new contract, initial balance, the address that pays for the origination, and the contract code and initial storage (both contained in the script). The last few arguments are flags that specify whether the contract is spendable, delegatable and the address of the delegate.

// oracle contract
done, oracleDone, err := r.BatchOriginate(moneyKeys, oracleKeys.PKH, 0, moneyKeys.PKH, prepareOracleScript(oracleKeys.PKH), false, false, "")
included := make(chan string)
go func(done <-chan string, included chan<- string) {
    opHash := <-done
    fmt.Println("origination of oracle contract injected in operation:", opHash)
    blockHash := <-done
    fmt.Println("origination of oracle contract included in block:", blockHash)
    included <- blockHash
}(done, included)

// placeholder proxy contract
done, placeholderDone, err := r.BatchOriginate(moneyKeys, providerKeys.PKH, 0, moneyKeys.PKH, json.RawMessage(placeholderScript), false, false, "")
go func(done <-chan string, included chan<- string) {
    opHash := <-done
    fmt.Println("origination of placeholder contract injected in operation:", opHash)
    blockHash := <-done
    fmt.Println("origination of placeholder contract included in block:", blockHash)
    included <- blockHash
}(done, included)

The returned values are similar to BatchTransfer with one important addition – a second channel which lets us know the address or addresses of newly originated contracts.

Since the operations get batched automatically, the four operations shown above are all likely to be batched together. This is how it looks on tzscan:

We are going to use the addresses we get from oracleDone and placeholderDone as the initial storage of the main contract (the provider), so we need to make sure the origination operations were included. We create a new channel (called included), and we send to it from the logging goroutines. Below in the main program, we don’t continue until we have received both messages (the hashes are not needed in this case, so we throw them away).

oracle := <-oracleDone
placeholder := <-placeholderDone
// make sure both originations made it into a block before continuing
<-included
<-included

fmt.Println("oracle address is:", oracle)
fmt.Println("placeholder address is:", placeholder)

// provider contract
// we need the previous two contracts for this one
done, providerDone, err := r.BatchOriginate(moneyKeys, providerKeys.PKH, 20000000, moneyKeys.PKH, prepareProviderContract(providerKeys.PublicStr,    oracle, placeholder), false, false, "")
go func(done <-chan string, included chan<- string) {
    opHash := <-done
    fmt.Println("origination of provider contract injected in operation:", opHash)
    blockHash := <-done
    fmt.Println("origination of provider contract included in block:", blockHash)
    included <- blockHash
}(done, included)

We use the same technique once more, to wait for the origination of the provider contract to be included. Then we create a contract that will handle oracle responses (proxy), wrap them in an appropriate type and send them to the provider contract. We needed to use a placeholder when originating the provider contract to let the type system check the proxy contract type.

provider := <-providerDone
<-included

// real proxy contract
// we need the oracle PKH and provider contract for this one
done, proxyDone, err := r.BatchOriginate(moneyKeys, providerKeys.PKH, 0, moneyKeys.PKH, prepareProxyContract(oracleKeys.PKH, provider), false, false, "")
go func(done <-chan string, included chan<- string) {
    opHash := <-done
    fmt.Println("origination of real proxy contract injected in operation:", opHash)
    blockHash := <-done
    fmt.Println("origination of real proxy contract included in block:", blockHash)
    included <- blockHash
}(done, included)

proxy := <-proxyDone
<-included

Finally, we take the address of the proxy contract, sign it with the provider’s private key and initialize the provider. Now the provider is operational, and won’t accept any oracle responses that aren’t coming from its proxy contract.

_, sig, err := r.HashAndSignData(prepareHDI(proxy), providerKeys)
// initialize the provider contract
// we need to sign the proxy contract address and send it along with the signature
done, err = r.BatchTransfer(moneyKeys, 10000000, moneyKeys.PKH, provider, initParams(proxy, sig))
opHash := <-done
fmt.Println("initialization of provider injected in operation:", opHash)
blockHash := <-done
fmt.Println("initialization of provider included in block:", blockHash)

After the initialization gets included, we are done and can log the information necessary to run the oracle and insurer servers.

fmt.Println("It works.")
fmt.Printf("Oracle Identity: %#v\n", oracleKeys)
fmt.Printf("Oracle Contract: %v\n", oracle)
fmt.Printf("Provider Identity: %#v\n", providerKeys)
fmt.Printf("Provider Contract: %v\n", provider)
fmt.Printf("Proxy Contract: %v\n", proxy)

If you’d like to compare the new event driven script to the old one, the old script can be found in an older commit.

With the new script, we can easily set up a complicated system of smart contracts even on a relatively unstable network like the alphanet, where it can take up to 3 minutes for new blocks to show up (if no bakers were missing blocks it would be 30 seconds).

I’ve run a few benchmarks and it seems possible to have around 290 transactions to regular addresses in one batch. Transactions to smart contracts take up a bit more space.

I think this is a powerful approach to building on Tezos, and I am currently talking to the Tezos Foundation about making our Go packages open source. If you’d like to use the packages, you can let TF know on Twitter. If you liked the article, do share it with those who might find it useful.