Refiner Onboarding

Refiner Nodes

The Refiner is a concurrent Block Specimen data processing framework designed for scalable and verified data transformations. Block results are only one of the many possible outputs which are complete representations of the actual blocks being produced on the Ethereum blockchain (that can be queried using an RPC call to an Ethereum node client like erigon and go-ethereum) and some additional useful fields.

The Refiner Whitepaper goes into detail on its purpose and mission.

Running the Refiner stack involves running three main pieces of Covalent Network OS infrastructure, rudder, ipfs-pinner and evm-server that coordinate and execute a transformation pipeline per Block Specimen. Running these nodes are not as disk I/O (or cpu/memory) intense as running bsp-geth and bsp-agent on local machines, however they do require sufficient bandwidth for access to distributed store resources and sending transaction proofs using ethereum clients - connecting to our EVM public blockchain partner - moonbeam. We shall setup all of these step-by-step in this guide in two main ways:

Run with Docker Compose (Recommended method)
Build & Run from Source (Optional method)

Hardware Requirements

Minimum

2 vCPUs (cores)
3GB RAM
200GB HDD free storage (mostly storing artifacts on local ipfs; can be pruned periodically)
8 MBit/sec download Internet service

Recommended

4 vCPUs
8 GB+ RAM
SSD with >= 500GB storage space
25+ MBit/sec download Internet service

Software Requirements (Docker Setup)

64-bit Linux, Mac OS 13+
SSL certificates
docker, docker-compose, direnv

Validator & Operator Prerequisites

Provide Covalent with two public addresses
- Staking Address: This is the address of the wallet holding your CQT on Moonbeam
- Operator Address: This is a public address tied to the private-public key pair an operator will use to sign proof transactions to the proof-chain contract. It is not tied to any network. It can be generated using the BIP-44 mnemonic multi-account deterministic algorithm. It can be generated using https://iancoleman.io/bip39/ (select 24 words, Coin: ETH)
Please provide the public address to this pair to Covalent. Keep the private key secret and safe. Please make sure that the Staking address and Operator Address are separate.
35,000 CQT on Moonbeam. If you have not bridged your CQT to Moonbeam, here is a guide on how.
GLMR is needed to pay for gas on Moonbeam
- in order to send proofs of block specimens. This costs approximately 5 GLMR per day. This should be held at the same address as the Operator Address.
- in order to stake your CQT
Access the Covalent Network Operator Dashboard. Import your Validator private key into Metamask, select the Moonbeam network (chain_id: 1284) and connect your Staking wallet to the operator dashboard. Your operator should be currently disabled.
Self-Stake the Minimum CQT Staking requirement on Moonbeam (35,000 CQT) for refiner. This is done using the Change Stake button in your operator dashboard. Proceed to first set Stake Amount to 35000 and click Approve. After the on chain approval transaction completes, click on Stake. Once the staking transaction completes, your Staking Status should display Sufficiently Staked.
Then proceed to turn on your Operator Status as using the Operator Status toggle:
Get access to a reliable Moonbeam https RPC. Providers and RPC URLs can be found here. Ping the Covalent team on discord for recommendations.
Validators should create a web3.storage account and a valid API token, update theWEB3_JWT (for ipfs-pinner) with the API token provided. We don't advice sharing of API tokens.

Run with Docker Compose

You can (preferably) use docker to run the system. Instructions for building from source is provided in the next section.

Install Docker

Follow the docker install instructions for your platform/architecture.

Onboarding instructions will be based on Ubuntu 22.04 LTS x86_64 / amd64.

Install direnv

sudo apt update
sudo apt get direnv

# bash users - add the following line to your ~/.bashrc
eval "$(direnv hook bash)"
source ~/.bashrc

# zsh users - add the following line to your ~/.zshrc
eval "$(direnv hook zsh)"
source ~/.zshrc

Clone and Set Env Vars

Clone the covalenthq/rudder repo.

git clone https://github.com/covalenthq/rudder
cd rudder
cat docker-compose-mbeam.yml

Check global environment variables

$ cat .envrc

export IPFS_PINNER_URL="http://127.0.0.1:3001"
export EVM_SERVER_URL="http://127.0.0.1:3002"

[[ -f .envrc.local ]] && source_env .envrc.local

Note: .envrc.local overrides any env vars set with .envrc on calling direnv allow .

Set local environment variables

$ touch .envrc.local

export BLOCK_RESULT_OPERATOR_PRIVATE_KEY="<<BRP-OPERATOR-PK-WITHOUT-0x-PREFIX>>"
export NODE_ETHEREUM_MAINNET="<<HTTPS-MOONBEAM-RPC-URL>>"
export IPFS_PINNER_URL="http://ipfs-pinner:3001"
export EVM_SERVER_URL="http://evm-server:3002"
export WEB3_JWT="<<WEB3.STORAGE-API-TOKEN>>"

Note: When passing the private key into the env vars as above please remove the 0x prefix so the private key env var has exactly 64 characters.

BLOCK_RESULT_OPERATOR_PRIVATE_KEY: Your personal Block Result Producer (BRP) operator private key
WEB3_JWT: Your personal web3.storage API token use by the ipfs-pinner service
IPFS_PINNER_URL: Service (ipfs-pinner) used by rudder to access IPFS assets like Block Specimens (service is automatically invoked and run with the in repo docker compose files eg: docker-compose-mbeam.yml)
EVM_SERVER_URL: Service (evm-server) used by rudder for stateless execution of Block Specimens into indexable (queryable) Block Results

Start Services

Load env vars into the shell.

$ direnv allow .

# make sure you see these being loaded
direnv: loading ~/rudder/.envrc
direnv: loading ~/rudder/.envrc.local
direnv: export +BLOCK_RESULT_OPERATOR_PRIVATE_KEY +ERIGON_NODE +EVM_SERVER_URL +IPFS_PINNER_URL +NODE_ETHEREUM_MAINNET +WEB3_JWT

Start all 3 services in the background for Moonbeam

$ docker compose -f "docker-compose-mbeam.yml" up -d --remove-orphans


[+] Running 3/3
 ⠿ Container rudder       Started                            3.2s
 ⠿ Container ipfs-pinner  Started                            1.9s
 ⠿ Container evm-server   Started                            1.8s

NOTE: On a system where an ipfs-pinner instance is already running, check the instruction in the Appendix to run rudder docker alongside.

Monitor the logs for Block Result submissions.

$ docker compose -f "docker-compose-mbeam.yml" logs -f –tail 2

rudder       | [info] curr_block: 4591264 and latest_block_num:4591263
ipfs-pinner  | 2023/06/22 13:45:49 Received /health request: source= 127.0.0.1:54420 status= OK
rudder       | [info] curr_block: 4591264 and latest_block_num:4591263
ipfs-pinner  | 2023/06/22 13:46:00 Received /health request: source= 127.0.0.1:54430 status= OK
rudder       | [info] curr_block: 4591264 and latest_block_num:4591264
rudder       | [info] listening for events at 4591264
rudder       | [info] found 0 bsps to process

Check step 6 in Run Rudder from source section for a successful Refiner stack run log output with performance metrics.

Deployment As Service Unit

Here you can find an example of a systemd service unit file that can be used to auto-start/restart of the docker-compose service for Refiner.

[Unit]
Description=Refiner docker compose
PartOf=docker.service
After=docker.service

[Service]
User=blockchain
Group=blockchain
Environment=HOME=/home/blockchain/tmp
Environment="BLOCK_RESULT_OPERATOR_PRIVATE_KEY=<<BRP-OPERATOR-PK-WITHOUT-0x-PREFIX>>"
Environment="NODE_ETHEREUM_MAINNET=<<HTTPS-MOONBEAM-RPC-URL>>"
Environment="IPFS_PINNER_URL=http://ipfs-pinner:3001" #service in docker
Environment="EVM_SERVER_URL=http://evm-server:3002" #service in docker
Environment="WEB3_JWT=<<WEB3.STORAGE-API-TOKEN>>"
Type=simple
ExecStart=docker compose -f "/home/blockchain/tmp/docker-compose-mbeam.yml" up --remove-orphans
Restart=always
TimeoutStopSec=infinity

[Install]
WantedBy=multi-user.target

After adding the env vars in their respective fields in the service unit file, enable the service and start it.

sudo systemctl enable rudder-compose.service
sudo systemctl start rudder-compose.service

Note 1: When passing the private key into the env vars as above please remove the 0x prefix so the private key env var has exactly 64 characters.

Note 2: In order to run docker compose as a non-root user for the above shown service unit you need to create a docker group (if it doesn't exist) and add the user (say) "blockchain" to the docker group

sudo groupadd docker
sudo usermod -aG docker blockchain
sudo su - blockchain
docker run hello-world

Build & Run From Source

Prerequisites

MacOS 12.x (M1/Intel) Installation requirements

Install XCode (skip if you already have xcode and have been developing on your mac)

xcode-select --install
sudo xcodebuild -license
agree

You must also Install brew (for mac m1/intel) with this script. This installs all the minimum necessary terminal/command-line tools to get started with easy software development on a mac.

Installation Time: 35-40 mins depending on your machine and network.

Install git, go, asdf, erlang, elixir, direnv, go-ipfs

Git is used as the source code version control manager across all our repositories.
Go is the programming language that is used to develop on go-ethereum, bsp-agent, erigon (EVM plugin) all which are entirely written in go.
Asdf is a CLI tool that can manage multiple language runtime versions on a per-project basis.
Erlang is a programming language used to build massively scalable soft real-time systems with requirements on high availability.
Elixir is a programming language that runs on the Erlang VM, known for creating low-latency, distributed, high concurrency fault-tolerant systems.
IPFS as the InterPlanetary File System (IPFS) is a protocol, hypermedia and file sharing peer-to-peer network for storing and sharing data in a distributed file system.
Direnv is used for secret management and control. Since all the necessary parameters to the agent that are sensitive cannot be passed into a command line flag. Direnv allows for safe and easy management of secrets like ethereum private keys for the operator accounts on the Covalent Network and redis instance access passwords etc. As these applications are exposed to the internet on http ports it’s essential to not have the information be logged anywhere. To enable “direnv” on your machine add these to your ~./bash_profile or ~./zshrc depending on which you use as your default shell after installing it using brew.

MacOS 12.x (M1/Intel) Install dependencies

/bin/bash -c "$(curl -fsSL https://raw.githubusercontent.com/Homebrew/install/HEAD/install.sh)"

brew install coreutils curl git wget direnv asdf

For setting up asdf on other shells please refer to this guide.

Linux x86_64 (Ubuntu 22.04 LTS) Install dependencies

sudo apt update
sudo apt install build-essential coreutils libssl-dev automake autoconf libncurses5-dev curl git wget direnv unzip

git clone https://github.com/asdf-vm/asdf.git ~/.asdf --branch v0.11.2
echo ". $HOME/.asdf/asdf.sh" >> ~/.bashrc
echo ". $HOME/.asdf/completions/asdf.bash" >> ~/.bashrc
source ~/.bashrc

Install required asdf version manager plugins for erlang and elixir

asdf plugin add erlang https://github.com/asdf-vm/asdf-erlang.git
asdf plugin add elixir https://github.com/asdf-vm/asdf-elixir.git
asdf plugin add golang https://github.com/kennyp/asdf-golang.git

Add to your bash shell for asdf (for any other shell, check the asdf guide.)

`echo -e "\n. \"$(brew --prefix asdf)/etc/bash_completion.d/asdf.bash\"" >> ~/.bash_profile`

Install Erlang and Elixir using the plugins

asdf install erlang 25.3
asdf install elixir 1.14.3-otp-24
asdf install golang 1.18.1

Set the versions
```
asdf global erlang 25.3
asdf global elixir 1.14.3-otp-24
asdf global golang 1.18.1
```
This will create a .tool-versions file in your home directory. ASDF will use these versions whenever a project doesn't specify versions of its own.

Enable direnv for shell/zsh

#bash users - add the following line to your ~/.bashrc
eval "$(direnv hook bash)"

# zsh users - add the following line to your ~/.zshrc
eval "$(direnv hook zsh)"

After adding this line do not forget to source your bash / powershell config with the following, by running it in your terminal

source ~/.zshrc
source ~/.bashrc

Install latest go-ipfs (kubo) and initialize go-ipfs

MacOS 12.x (M1/Intel)

wget https://dist.ipfs.tech/go-ipfs/v0.18.0/go-ipfs_v0.18.0_darwin-arm64.tar.gz
tar -xzvf go-ipfs_v0.18.0_darwin-arm64.tar.gz
cd go-ipfs
bash install.sh
ipfs init

Linux x86_64 (Ubuntu 22.04 LTS)

wget https://dist.ipfs.tech/go-ipfs/v0.18.0/go-ipfs_v0.18.0_linux-amd64.tar.gz
tar -xzvf go-ipfs_v0.18.0_darwin-arm64.tar.gz
cd go-ipfs
bash install.sh
ipfs init

Note: To avoid permissions and netscan issues execute the following against ipfs binary home directory application

sudo chmod -R 700 ~/.ipfs
ipfs config profile apply server

Run EVM-Server

The EVM-Server is a stateless EVM block execution tool. It's stateless because in Ethereum nodes like geth, it doesn't need to maintain database of blocks to do a block execution or re-execution. The evm-server service transforms Block Specimens to Block Results entirely on the input of underlying capture of Block Specimen data.

Clone covalenthq/erigon that contains the fork for this particular stateless execution function, build the evm-server binary
```
$ make evm
```

Run the generated binary

$ ./build/bin/evm t8n --server.mode

  [INFO] [06-22|13:53:54.148] Listening port=3002

Note: evm-server occupies the port 3002 on your local at http://evm-server:3002", so make sure this port in not occupied by any other service prior to start.

Run IPFS-Pinner

The IPFS-Pinner is an interface to the storage layer of the Covalent Network using a decentralized storage network underneath. Primarily it's a custom IPFS (Inter Planetary File System) node with pinning service components for Web3.storage and Pinata; content archive manipulation etc. Additionally there's support for fetching using dweb.link. It is meant for uploading/fetching artifacts (like Block Specimens and uploading Block Results or any other processed/transformed data asset) of the Covalent Decentralized Network.

Clone covalenthq/ipfs-pinner and build the pinner server binary

git clone https://github.com/covalenthq/ipfs-pinner.git --depth 1
cd ipfs-pinner
make server-dbg

Set the environment variable required by ipfs-pinner by getting WEB3_JWT from web3.storage and adding it to an .envrc file.
```
$ cat .envrc

export WEB3_JWT="<<WEB3.STORAGE-API-TOKEN>>"
```

Start the ipfs-pinner server

$ make run

generating 2048-bit RSA keypair...done
peer identity: Qmd9dT1hRTvaTZn9DAiaCatk3azz86Lwny8FhBd2jV8Kw5
2023/02/02 21:52:21 failed to sufficiently increase receive buffer size (was: 208 kiB, wanted: 2048 kiB, got: 416 kiB). See https://github.com/lucas-clemente/quic-go/wiki/UDP-Receive-Buffer-Size for details...
2023/02/02 21:52:22 Listening...

Note: ipfs-pinner occupies the port 3001 on your local at http://ipfs-pinner:3001", so make sure this port in not occupied by any other service prior to start.

Repo Migrations

In case of the following error

2023-03-10T08:46:18.621-0800	FATAL	ipfs-pinner	ipfs-pinner/pinner.go:29	error initializing ipfs node: ipfs repo needs migration, please run migration tool.
See https://github.com/ipfs/fs-repo-migrations/blob/master/run.md
Sorry for the inconvenience. In the future, these will run automatically.

Run the following migrations

wget https://dist.ipfs.tech/fs-repo-migrations/v2.0.2/fs-repo-migrations_v2.0.2_darwin-arm64.tar.gz
tar -ztvf fs-repo-migrations_v2.0.2_darwin-arm64.tar.gz
cd fs-repo-migrations/
./fs-repo-migrations

Run Rudder

Rudder is the primary orchestrator and supervisor for all transformation pipeline processes that locates a source Covalent Network data object to apply a tracing/execution/transformational rule to and outputs a new object generated from using such a rule.

Both the source and output are available through a decentralized storage service such as a wrapped IPFS node. A transformation-proof transaction is emitted, confirming that it has done this work along with the output content ids (ipfs) access URL. To define what these three components are:

Source: The Block Specimen that serves as an input to the Refiner. Proof transactions made earlier to a smart contract with the respective cids (content ids) are where the source is checked.

Rule: A transformation plugin (or server) that can act on the Block Specimen (source). These can be compared to blueprints that have been shown to produce the data objects needed. Furthermore, anyone can create these rules to get a desired data object. Rule (or server) versions thus need to exist, tied to the decoded block specimen versions they are applied on.

Target: The output generated from running the rule over the object that came from the source that is the block result.

Clone the rudder repo

git clone https://github.com/covalenthq/rudder.git
cd rudder
git checkout main

Get your BLOCK_RESULT_OPERATOR_PRIVATE_KEY that has GLMR tokens for gas on Moonbeam and is already whitelisted as Block Result Producer operator. Set the following environment variables for the local rudder by creating an .envrc.local file
```
touch .envrc.local
```

Copy paste the environment variables into this file

export BLOCK_RESULT_OPERATOR_PRIVATE_KEY="<<BRP-OPERATOR-PK-WITHOUT-0x-PREFIX>>"
export NODE_ETHEREUM_MAINNET="<<HTTPS-MOONBEAM-RPC-URL>>"
export IPFS_PINNER_URL="http://127.0.0.1:3001"
export EVM_SERVER_URL="http://127.0.0.1:3002"
export WEB3_JWT="<<WEB3.STORAGE-API-TOKEN>>"

Call to load .envrc.local + .envrc files with the command below and observe the following output, make sure the environment variables are loaded into the shell.

$ direnv allow .

direnv: loading ~/Covalent/rudder/.envrc
direnv: loading ~/Covalent/rudder/.envrc.local
direnv: export +BLOCK_RESULT_OPERATOR_PRIVATE_KEY +ERIGON_NODE +IPFS_PINNER_URL +NODE_ETHEREUM_MAINNET

Get all the required dependencies and build the rudder app for the prod environment (this points to Moonbeam BRP contracts). Note: Windows is currently not supported.

$ mix local.hex --force && mix local.rebar --force && mix deps.get
$ MIX_ENV=dev mix release --overwrite; #`MIX_ENV=prod mix release` will build executable binaries pointing to the Moonbeam mainnet smart contracts.
..
....
 evm-server: http://evm-server:3002
 ipfs-node: http://ipfs-pinner:3001
 * assembling rudder-0.2.12 on MIX_ENV=dev
 * skipping runtime configuration (config/runtime.exs not found)
 * skipping elixir.bat for windows (bin/elixir.bat not found in the Elixir installation)
 * skipping iex.bat for windows (bin/iex.bat not found in the Elixir installation)

 Release created at _build/dev/rel/rudder

     # To start your system
     _build/dev/rel/rudder/bin/rudder start

 Once the release is running:

     # To connect to it remotely
     _build/dev/rel/rudder/bin/rudder remote

     # To stop it gracefully (you may also send SIGINT/SIGTERM)
     _build/dev/rel/rudder/bin/rudder stop

 To list all commands:

     _build/dev/rel/rudder/bin/rudder

For any issues associated with building and re-compiling execute the following commands, that cleans, downloads and re-compiles the dependencies for rudder.
```
rm -rf _build deps && mix clean && mix deps.get && mix deps.compile
```

Start the rudder application and execute the proof-chain block specimen listener call which should run the Refiner pipeline pulling Block Specimens from IPFS using the cids read from recent proof-chain finalized transactions, decoding them, and uploading and proofing Block Results while keeping a track of failed ones and continuing (soft real-time) in case of failure. The erlang concurrent fault tolerance allows each pipeline to be an independent worker that can fail (for any given Block Specimen) without crashing the entire pipeline application. Multiple pipeline worker children threads continue their work in the synchronous queue of Block Specimen AVRO binary files running the stateless EVM binary (evm-server) re-execution tool.

$ MIX_ENV=dev mix run --no-halt --eval 'Rudder.ProofChain.BlockSpecimenEventListener.start()';

  ..
  ...
  rudder       | [info] found 1 bsps to process
  ipfs-pinner  | 2023/06/29 20:28:30 unixfsApi.Get: getting the cid: bafybeiaxl44nbafdmydaojz7krve6lcggvtysk6r3jaotrdhib3wpdb3di
  ipfs-pinner  | 2023/06/29 20:28:30 trying out https://w3s.link/ipfs/bafybeiaxl44nbafdmydaojz7krve6lcggvtysk6r3jaotrdhib3wpdb3di
  ipfs-pinner  | 2023/06/29 20:28:31 got the content!
  rudder       | [info] Counter for ipfs_metrics - [fetch: 1]
  rudder       | [info] LastValue for ipfs_metrics - [fetch_last_exec_time: 0.001604]
  rudder       | [info] Sum for ipfs_metrics - [fetch_total_exec_time: 0.001604]
  rudder       | [info] Summary for ipfs_metrics  - {0.001604, 0.001604}
  rudder       | [debug] reading schema `block-ethereum` from the file /app/priv/schemas/block-ethereum.avsc
  rudder       | [info] Counter for bsp_metrics - [decode: 1]
  rudder       | [info] LastValue for bsp_metrics - [decode_last_exec_time: 0.0]
  rudder       | [info] Sum for bsp_metrics - [decode_total_exec_time: 0.0]
  rudder       | [info] Summary for bsp_metrics  - {0.0, 0.0}
  rudder       | [info] submitting 17586995 to evm http server...
  evm-server   | [INFO] [06-29|20:28:31.859] input file at                            loc=/tmp/3082854681
  evm-server   | [INFO] [06-29|20:28:31.862] output file at:                          loc=/tmp/1454174090
  evm-server   | [INFO] [06-29|20:28:32.112] Wrote file                               file=/tmp/1454174090
  rudder       | [info] writing block result into "/tmp/briefly-1688/briefly-576460747542186916-YRw0mRjfExGMk4M672"
  rudder       | [info] Counter for bsp_metrics - [execute: 1]
  rudder       | [info] LastValue for bsp_metrics - [execute_last_exec_time: 3.14e-4]
  rudder       | [info] Sum for bsp_metrics - [execute_total_exec_time: 3.14e-4]
  rudder       | [info] Summary for bsp_metrics  - {3.14e-4, 3.14e-4}
  ipfs-pinner  | 2023/06/29 20:28:32 generated dag has root cid: bafybeic6ernzbb6x4qslwfgklveisyz4vkuqhaafqzwlvto6c2njonxi3e
  ipfs-pinner  | 2023/06/29 20:28:32 car file location: /tmp/249116437.car
  [119B blob data]
  ipfs-pinner  | 2023/06/29 20:28:34 Received /health request: source= 127.0.0.1:34980 status= OK
  ipfs-pinner  | 2023/06/29 20:28:34 uploaded file has root cid: bafybeic6ernzbb6x4qslwfgklveisyz4vkuqhaafqzwlvto6c2njonxi3e
  rudder       | [info] Counter for ipfs_metrics - [pin: 1]
  rudder       | [info] LastValue for ipfs_metrics - [pin_last_exec_time: 0.002728]
  rudder       | [info] Sum for ipfs_metrics - [pin_total_exec_time: 0.002728]
  rudder       | [info] Summary for ipfs_metrics  - {0.002728, 0.002728}
  rudder       | [info] 17586995:48f1e992d1ac800baed282e12ef4f2200820061b5b8f01ca0a9ed9a7d6b5ddb3 has been successfully uploaded at ipfs://bafybeic6ernzbb6x4qslwfgklveisyz4vkuqhaafqzwlvto6c2njonxi3e
  rudder       | [info] 17586995:48f1e992d1ac800baed282e12ef4f2200820061b5b8f01ca0a9ed9a7d6b5ddb3 proof submitting
  rudder       | [info] Counter for brp_metrics - [proof: 1]
  rudder       | [info] LastValue for brp_metrics - [proof_last_exec_time: 3.6399999999999996e-4]
  rudder       | [info] Sum for brp_metrics - [proof_total_exec_time: 3.6399999999999996e-4]
  rudder       | [info] Summary for brp_metrics  - {3.6399999999999996e-4, 3.6399999999999996e-4}
  rudder       | [info] 17586995 txid is 0xd8a8ea410240bb0324433bc26fdc79d496ad0c8bfd18b60314a05e3a0de4fb06
  rudder       | [info] Counter for brp_metrics - [upload_success: 1]
  rudder       | [info] LastValue for brp_metrics - [upload_success_last_exec_time: 0.0031149999999999997]
  rudder       | [info] Sum for brp_metrics - [upload_success_total_exec_time: 0.0031149999999999997]
  rudder       | [info] Summary for brp_metrics  - {0.0031149999999999997, 0.0031149999999999997}
  rudder       | [info] Counter for rudder_metrics - [pipeline_success: 1]
  rudder       | [info] LastValue for rudder_metrics - [pipeline_success_last_exec_time: 0.0052]
  rudder       | [info] Sum for rudder_metrics - [pipeline_success_total_exec_time: 0.0052]
  rudder       | [info] Summary for rudder_metrics  - {0.0052, 0.0052}

Check logs for any errors in the pipeline process and note the performance metrics in line with execution. Checkout the documentation on what is being measured and why here.

  $ tail -f logs/log.log
  ..
  ...
  rudder       | [info] Counter for rudder_metrics - [pipeline_success: 1]
  rudder       | [info] LastValue for rudder_metrics - [pipeline_success_last_exec_time: 0.0052]
  rudder       | [info] Sum for rudder_metrics - [pipeline_success_total_exec_time: 0.0052]
  rudder       | [info] Summary for rudder_metrics  - {0.0052, 0.0052}

For any issues associated with building and re-compiling execute the following commands, that cleans, downloads and re-compiles the dependencies for rudder.
```
rm -rf _build deps && mix clean && mix deps.get && mix deps.compile
```
Now you can check proof-chain for block result proof transactions made by your Block Result Producer (BRP): https://moonbase.moonscan.io/address/0xeFD84B6F1C04Ad9A7D666E21C9608a2ea2FE4B83
If you got everything working so far. Congratulations! You're now a Refiner operator on the Covalent Network. Set up Grafana monitoring and alerting from links in the additional resources section.

Sample Systemd Service Units

If the Refiner stack is running successfully and producing Block Results, congrats! As a way to manage the services, you might want to use systemd. We next provide sample systemd service files, so that any crash in one of the components would restart that component, rather than the system halting. Don't forget to replace the placeholders with actual values in these sample files.

Refiner - Service Unit File

[Unit]
Description=Refiner docker compose
PartOf=docker.service
After=docker.service

[Service]
User=blockchain
Group=blockchain
Environment=HOME=/home/blockchain/tmp
Environment="BLOCK_RESULT_OPERATOR_PRIVATE_KEY=<<BRP-OPERATOR-PK-WITHOUT-0x-PREFIX>>"
Environment="NODE_ETHEREUM_MAINNET=<<HTTPS-MOONBEAM-RPC-URL>>"
Environment="IPFS_PINNER_URL=http://ipfs-pinner:3001"
Environment="EVM_SERVER_URL=http://evm-server:3002"
Environment="WEB3_JWT=<<WEB3.STORAGE-API-TOKEN>>"
Type=simple
ExecStart=docker compose -f "/home/blockchain/tmp/docker-compose-mbeam.yml" up --remove-orphans
Restart=always
TimeoutStopSec=infinity

[Install]
WantedBy=multi-user.target

Additional Resources

Public reference to understand more about the functions of the various components can be found in the Covalent Network Whitepaper.
Public reference to understand more about the block specimen producer and its role in the Covalent Network can be found in the BSP whitepaper.
Setting up monitoring and alerting with Grafana and Prometheus for Refiner
Refiner contributions and bug bounty guidelines
Detailed Refiner Architecture

Appendix

Run With Existing IPFS-Pinner Service

On a system where an ipfs-pinner instance is already running, use this modified .envrc.local and docker-compose-mbeam.yml

export BLOCK_RESULT_OPERATOR_PRIVATE_KEY="BRP-OPERATOR-PK-WITHOUT-0x-PREFIX"
export NODE_ETHEREUM_MAINNET="<<HTTPS-MOONBEAM-RPC-URL>>"
export IPFS_PINNER_URL="http://host.docker.internal:3001"
export EVM_SERVER_URL="http://evm-server:3002"
export WEB3_JWT="<<WEB3.STORAGE-API-TOKEN>>"