snarkOS

## Table of Contents * [1. Overview](#1-overview) * [2. Build Guide](#2-build-guide) * [2.1 Requirements](#21-requirements) * [2.2 Installation](#22-installation) * [3. Run an Aleo Node](#3-run-an-aleo-node) * [3.1 Run an Aleo Client](#31-run-an-aleo-client) * [3.2 Run an Aleo Validator](#32-run-an-aleo-validator) * [3.3 Run an Aleo Prover](#33-run-an-aleo-prover) * [4. FAQs](#4-faqs) * [5. Command Line Interface](#5-command-line-interface) * [6. Development Guide](#6-development-guide) * [6.1 Quick Start](#61-quick-start) * [6.2 Operations](#62-operations) * [6.3 Local Devnet](#63-local-devnet) * [6.4 Feature Flags](#64-feature-flags) * [7. Contributors](#7-contributors) * [8. License](#8-license) [comment]: <> (* [4. JSON-RPC Interface](#4-json-rpc-interface)) [comment]: <> (* [5. Additional Information](#5-additional-information)) ## 1. Overview __snarkOS__ is a decentralized operating system for zero-knowledge applications. This code forms the backbone of [Aleo](https://aleo.org/) network, which verifies transactions and stores the encrypted state applications in a publicly-verifiable manner. ## 2. Build Guide ### 2.1 Definitions The following snarkOS node types exist in the Aleo network: - **Validator**: Validator nodes participate in consensus and must be started with an account that is bonded into the committee. - **Client**: Clients do not participate in consensus but maintain a ledger. They are capable of providing information about the network as well as accepting solutions and transactions and communicating them to their peers. All clients run the same software, however, for the purposes of configuration management, this document defines two types of clients: - Core Client: Client node connected directly to a validator node. - Outer Client: Client node connected only to other clients or prover nodes. - **Prover**: Prover nodes are dedicated to solving the Aleo puzzle. They do not participate in consensus or maintain a copy of the ledger. ### 2.2 Requirements The following are the requirements to run an Aleo node: - **OS**: 64-bit architectures only, latest up-to-date for security - Clients: Ubuntu 22.04 (LTS), macOS Ventura or later, Windows 11 or later - Validators: Ubuntu 22.04 (LTS) - **CPU**: 64-bit architectures only, Latest Intel Xeon or Better - Clients: 24-cores (32-cores or larger preferred) - Validators: 64-cores (128-cores or larger preferred) - **RAM**: DDR4 or better - Clients: 128GiB of memory (192GiB or larger preferred) - Validators: 256GiB of memory (384GiB or larger preferred) - **Storage**: PCIe Gen 3 x4, PCIe Gen 4 x2 NVME SSD, or better - Clients: 2TB of disk space (4TB or larger preferred) - Validators: 4TB of disk space (6TB or larger preferred) - **Network**: Symmetric, commercial, always-on - Clients: 250Mbps of upload **and** download bandwidth - Validators: 500Mbps of upload **and** download bandwidth No explicit recommendations are made for proving nodes as proving hardware may be highly variable. If interested in running Aleo Provers nodes, please refer to resources published by the Aleo community. ### 2.3 Installation Before beginning, please ensure your machine has Rust installed, with at least [this version](rust-toolchain). Instructions to [install Rust can be found here.](https://www.rust-lang.org/tools/install) Start by cloning this GitHub repository: ``` git clone --branch mainnet --single-branch https://github.com/ProvableHQ/snarkOS.git ``` Next, move into the `snarkOS` directory: ``` cd snarkOS ``` **[For Ubuntu users]** A helper script to install dependencies is available. From the `snarkOS` directory, run: ``` ./build_ubuntu.sh ``` Lastly, install `snarkOS`: ``` cargo install --locked --path . ``` #### Optional: CUDA Acceleration for Provers > This CUDA build is optional. The current snarkOS puzzle does not leverage CUDA acceleration—it is a leftover from a previous event, although CUDA may become relevant again with ARC-43. > > The CUDA feature is considered unstable and experimental; expect breaking changes. For prover operators who want to experiment with GPU support: ``` cargo install --locked --path . --features cuda ``` Please ensure ports `4130/tcp` and `3030/tcp` are open on your router and OS firewall. ### 2.4 Port Configuration #### 2.4.1 For Core Clients | Port | Protocol | Allow/Deny | Source | Explanation | |----------|----------|------------|------------------------------|------------------------------------------------------------| | 4130/tcp | TCP | Allow | All IPv4/IPv6 | TCP traffic to peers | #### 2.4.2 For Outer Clients | Port | Protocol | Allow/Deny | Source | Explanation | |----------|----------|------------|------------------------------|------------------------------------------------------------| | 3030/tcp | TCP | Allow | All IPv4/IPv6 | REST server | | 4130/tcp | TCP | Allow | All IPv4/IPv6 | TCP traffic to peers | #### 2.4.3 For Validators | Port | Protocol | Allow/Deny | Source | Explanation | |----------|----------|------------|------------------------------|------------------------------------------------------------| | 4130/tcp | TCP | Allow | All IPv4/IPv6 | TCP traffic to peers | | 5000/tcp | TCP | Allow | Trusted Validator IPs | TCP traffic between validators for BFT communication | | 3000/tcp | TCP | Allow | Internal VPC or VPN | Metrics dashboard, should only be open within an internal VPC or VPN | | 3030/tcp | TCP | Deny | All IPv4/IPv6 | REST server. This should **always** be disabled for validators | | 9000/tcp | TCP | Allow | Internal VPC or VPN | Metrics export, should only be open within an internal VPC or VPN | | 9090/tcp | TCP | Allow | Internal VPC or VPN | Prometheus metrics, should only be open within an internal VPC or VPN | **Note:** Ensure that your open file limit is set to 16,384 or above. For the recommended setting run: ``` # Increase the open file limit for the current user (replace with your username) echo " - nofile 65536" | sudo tee -a /etc/security/limits.conf # Increase the default system open file limit sudo bash -c 'echo "DefaultLimitNOFILE=65536" >> /etc/systemd/system.conf' ``` ## 3. Run an Aleo Node ## 3.1 Run an Aleo Client Start by following the instructions in the [Build Guide](#2-build-guide). The guide below provides information on running `core` and `outer` clients (as defined in Section 2.2.) Aleo community members running validators are recommended to run 1-3 `core` clients as their exclusive client peers. This will ensure network traffic from the public internet is verified prior to reaching the validator. Any client **not** connected directly to a validator can be considered an `outer` client. ### 3.1.1 Run an Aleo Core Client The following command is recommended when starting a client node that is connected to a validator: `snarkos start --client --nodisplay --node 0.0.0.0:4130 --peers "validator_ip:4130,core_client_ip_1:4130,core_client_ip_2:4130,core_client_ip3:4130,outer_client_ip_1:4130,..." --verbosity 1 --norest` To start a core client node, you can also run the following command from the `snarkOS` directory: ``` ./scripts/run-core-client.sh ``` ### 3.1.2 Run an Aleo Outer Client The following command is recommended when starting a client node that is NOT connected to a validator: `snarkos start --client --nodisplay --node 0.0.0.0:4130 --peers "core_client_ip_1:4130,core_client_ip_2:4130,core_client_ip3:4130,outer_client_ip_1:4130,..." --verbosity 1 --rest 0.0.0.0:3030` To start an outer client node, you can also run the following command from the `snarkOS` directory: ``` ./scripts/run-outer-client.sh ``` Outer clients can be bootstrap clients that serve as accessible entry points for new nodes joining the network with publicly known or static IPs. For bootstrap clients, we also recommend the use of `--rotate-external-peers` to avoid the bootstrap peerlist from filling up. ## 3.2 Run an Aleo Validator Start by following the instructions in the [Build Guide](#2-build-guide). The following command is recommended when starting a validator node: `snarkos start --validator --nodisplay --bft 0.0.0.0:5000 --node 0.0.0.0:4130 --peers "validator_ip_1:4130,validator_ip_2:4130,...,core_client_ip_1:4130,core_client_ip_2:4130,..." --validators "validator_ip_1:5000,validator_ip_2:5000,..." --verbosity 1 --norest --private-key-file ~/snarkOS/privatekey` Instead of specifying a private key file (`--private-key-file` flag), the private key can also be defined explicitly (`--private-key` flag). To start a validator, you can also run the following command from the `snarkOS` directory: ``` ./scripts/run-validator.sh ``` ### 3.2.1 Enable Validator Telemetry Metrics (Optional) Validator telemetry allows you to track participation in consensus. This is optional and can be enabled using the `telemetry` feature flag. Once enabled, telemetry metrics are available through: 1. Node logs 2. REST API endpoints ``` // GET /{network}/validators/participation // GET /{network}/validators/participation?metadata={true} ``` You can enable telemetry in one of the following ways: #### 1. Enable via [installation](#2.3-installation) Add the `telemetry` feature flag to the installation command. ``` cargo install --locked --path . --features telemetry ``` #### 2. Enable via `./run-validator.sh` Run the `./scripts/run-validator.sh` script and enable telemetry when prompted: ``` Do you want to enable validator telemetry? (y/n, default: y): ``` ## 3.3 Run an Aleo Prover Start by following the instructions in the [Build Guide](#2-build-guide). Next, generate an Aleo account address: ``` snarkos account new ``` This will output a new Aleo account in the terminal. **Please remember to save the account private key and view key.** The following is an example output: ``` Attention - Remember to store this account private key and view key. Private Key APrivateKey1xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx <-- Save Me And Use In The Next Step View Key AViewKey1xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx <-- Save Me Address aleo1xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx <-- Save Me ``` Next, to start a proving node, from the `snarkOS` directory, run: ``` ./scripts/run-prover.sh ``` When prompted, enter your Aleo private key: ``` Enter the Aleo Prover account private key: APrivateKey1xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx ``` ## 4. FAQs ### 1. My node is unable to compile. - Ensure your machine has Rust installed, with at least [this version](rust-toolchain). Instructions to [install Rust can be found here.](https://www.rust-lang.org/tools/install) - If large errors appear during compilation, try running `cargo clean`. - Ensure `snarkOS` is started using `./scripts/run-client.sh` or `./scripts/run-prover.sh`. ### 2. My node is unable to connect to peers on the network. - Ensure ports `4130/tcp` and `3030/tcp` are open on your router and OS firewall. - Ensure `snarkOS` is started using `./scripts/run-client.sh` or `./scripts/run-prover.sh`. ### 3. I can't generate a new address ### - Before running the command above (`snarkos account new`) try `source ~/.bashrc` - Also double-check the spelling of `snarkos`. Note the directory is `/snarkOS`, and the command is `snarkos` ### 4. How do I use the CLI to sign and verify a message? 1. Generate an account with `snarkos account new` if you haven't already 2. Sign a message with your private key using `snarkos account sign --raw -m "Message" --private-key-file=` 3. Verify your signature with `snarkos account verify --raw -m "Message" -s sign1SignatureHere -a aleo1YourAccountAddress` Note, using the `--raw` flag with the command will sign plaintext messages as bytes rather than [Aleo values](https://developer.aleo.org/guides/aleo/language#data-types-and-values) such as `1u8` or `100field`. ### 5. What is the different between `node-data` and `ledger`? Ledger contains only public ledger information, while `node-data` contains information specific to the node that created it. The latter should not be shared with untrusted sources, and, for validators, contains data required to participate in consensus. ### 6. At startup I get an error telling me the nodes still uses the old storage format. What should I do? The node should have created a new folder for the node data. For example, for mainnet a folder should exist at `~/.aleo/storage/node-data-0`. The error message will tell you what data to migrate. Alternatively, you can start the node with `--auto-migrate-node-data` and it will attempt to do this automatically. The following is an overview of all files that may be needed to be migrated. - The JSON Web token secret, located at `~/.aleo/storage/jwt_secrect_{address}.txt`. Move it to `~/.aleo/storage/node-data-0/jwt_secret_{address}.txt`. Note, if you are running different nodes with different addresses there may be multiple of these. The error message will tell you which one to migrate. - The router peer cache located at `~/.aleo/storage/ledger-0/cache_router_peers`. Migrate it to `~/.aleo/storage/node-data-0/router-peer-cache`. - The gateway peer cache located at `~/.aleo/storage/ledger-0/cache_gateway_peers` (only for validators). Migrate it to `~/.aleo/storage/node-data-0/gateway-peer-cache`. - The latest proposal cache located at `~/.aleo/storage/current-proposal-cache-0` (only for validators). Migrate it to `~/.aleo/storage/node-data-0/current-proposal-cache`. ## 5. Command Line Interface To run a node with custom settings, refer to the options and flags available in the `snarkOS` CLI. The full list of CLI flags and options can be viewed with `snarkos --help`: ``` snarkOS The Aleo Team USAGE: snarkos [OPTIONS] OPTIONS: -h, --help Print help information -v, --verbosity Specify the verbosity [options: 0, 1, 2, 3] [default: 2] SUBCOMMANDS: account Commands to manage Aleo accounts clean Cleans the snarkOS node storage help Print this message or the help of the given subcommand(s) start Starts the snarkOS node update Update snarkOS ``` The following are the options for the `snarkos start` command: ``` --network Specify the network ID of this node [options: 0 = mainnet, 1 = testnet, 2 = canary] [default: 0] --prover Start the node as a prover --client Start the node as a client (default). Client are "full nodes", i.e, validate and execute all blocks they receive, but they do not participate in AleoBFT consensus. --bootstrap-client Start the node as a bootstrap client. --validator Start the node as a validator. Validators are "full nodes", like clients, but also participate in AleoBFT. --noupdater Disable checking for new versions at startup --private-key Specify the account private key of the node --private-key-file Specify the path to a file containing the account private key of the node --node Set the IP address and port used for P2P communication --bft Set the IP address and port used for BFT communication. This argument is only allowed for validator nodes --peers Specify the IP address and port of the peer(s) to connect to (as a comma-separated list). These peers will be set as "trusted", which means the node will not disconnect from them when performing peer rotation. Setting peers to "" has the same effect as not setting the flag at all, except when using `--dev`. --validators Specify the IP address and port of the validator(s) to connect to --rest Specify the IP address and port for the REST server --rest-rps Specify the requests per second (RPS) rate limit per IP for the REST server [default: 10] --jwt-secret Specify the JWT secret for the REST server (16B, base64-encoded) --jwt-timestamp Specify the JWT creation timestamp; can be any time in the last 10 years --norest If the flag is set, the node will not initialize the REST server --nojwt If the flag is set, the node will not require JWT authentication for the REST server --trusted-peers-only If the flag is set, the node will only connect to trusted peers and validators --nodisplay Write log message to stdout instead of showing a terminal UI. This is useful, for example, for running a node as a service instead of in the foreground or to pipe its output into a file. --verbosity Specify the log verbosity of the node. [options: 0 (lowest log level) to 6 (highest level)] [default: 1] --log-filter Set a custom log filtering scheme, e.g., "off,snarkos_bft=trace", to show all log messages of snarkos_bft but nothing else --logfile Specify the path to the file where logs will be stored [default: /var/folders/6v/1bwnpyjd1r5f9wr_9hq25qsm0000gn/T/snarkos.log] --metrics Enable the metrics exporter --metrics-ip Specify the IP address and port for the metrics exporter --ledger-storage Specify the directory that holds all ledger data, e.g., blocks and transactions. This flag overrides the default path, even when `--dev` is set. The old name for this flag (`--storage`) is DEPRECATED and will eventually be removed. --node-data-storage Specify the directory that holds node-specific data, that is not part of the global ledger. This flag overrides the default path, even when `--dev` is set. That folder may contain sensitive data, such as the JWT secret, and should not be shared with untrusted parties. For validators, it also contains the latest proposal cache, which is required to participate in consensus. --cdn Enables the node to prefetch initial blocks from a CDN --nocdn If the flag is set, the node will not prefetch from a CDN --dev Enables development mode used to set up test networks. The purpose of this flag is to run multiple nodes on the same machine and in the same working directory. To do this, set the value to a unique ID within the test work. For example if there are four nodes in the network, pass `--dev 0` for the first node, `--dev 1` for the second, and so forth. If you do not explicitly set the `--peers` flag, this will also populate the set of trusted peers, so that the network is fully connected. Additionally, if you do not set the `--rest` or the `--norest` flags, it will also set the REST port to `3030` for the first node, `3031` for the second, and so forth. --dev-num-validators If development mode is enabled, specify the number of genesis validator [default: 4] --dev-num-clients If development mode is enabled, specify the number of clients. This is only used by validators to automatically populate their set of trusted peers. This option cannot be used while also passing the `--peers` flag. --no-dev-txs If development mode is enabled, specify whether node 0 should generate traffic to drive the network --dev-bonded-balances If development mode is enabled, specify the custom bonded balances as a JSON object -h, --help Print help (see a summary with '-h') ``` ## 6. Development Guide ### 6.1 Quick Start In the first terminal, start the first validator by running: ``` cargo run --release -- start --nodisplay --dev 0 --validator ``` In the second terminal, start the second validator by running: ``` cargo run --release -- start --nodisplay --dev 1 --validator ``` In the third terminal, start the third validator by running: ``` cargo run --release -- start --nodisplay --dev 2 --validator ``` In the fourth terminal, start the fourth validator by running: ``` cargo run --release -- start --nodisplay --dev 3 --validator ``` From here, this procedure can be used to further start-up provers and clients. ### 6.2 Operations It is important to initialize the nodes starting from `0` and incrementing by `1` for each new node. The following is a list of options to initialize a node (replace `` with a number starting from `0`): ``` cargo run --release -- start --nodisplay --dev --validator cargo run --release -- start --nodisplay --dev --prover cargo run --release -- start --nodisplay --dev --client cargo run --release -- start --nodisplay --dev ``` When no node type is specified, the node will default to `--client`. ### 6.3 Local Devnet #### 6.3.1 Install `tmux` To run a local devnet with the script, start by installing `tmux`.

macOS

To install `tmux` on macOS, you can use the `Homebrew` package manager. If you haven't installed `Homebrew` yet, you can find instructions at [their website](https://brew.sh/). ```bash # Once Homebrew is installed, run: brew install tmux ```

Ubuntu

On Ubuntu and other Debian-based systems, you can use the `apt` package manager: ```bash sudo apt update sudo apt install tmux ```

Windows

There are a couple of ways to use `tmux` on Windows: ### Using Windows Subsystem for Linux (WSL) 1. First, install [Windows Subsystem for Linux](https://docs.microsoft.com/en-us/windows/wsl/install). 2. Once WSL is set up and you have a Linux distribution installed (e.g., Ubuntu), open your WSL terminal and install `tmux` as you would on a native Linux system: ```bash sudo apt update sudo apt install tmux ```

#### 6.3.2 Start a Local Devnet To start a local devnet, run: ``` ./scripts/devnet.sh ``` Follow the instructions in the terminal to start the devnet. #### 6.3.3 View a Local Devnet #### Switch Nodes (forward) To toggle to the next node in a local devnet, run: ``` Ctrl+b n ``` #### Switch Nodes (backwards) To toggle to the previous node in a local devnet, run: ``` Ctrl+b p ``` #### Select a Node (choose-tree) To select a node in a local devnet, run: ``` Ctrl+b w ``` #### Select a Node (manually) To select a node manually in a local devnet, run: ``` Ctrl+b :select-window -t {NODE_ID} ``` #### 6.3.4 Stop a Local Devnet To stop a local devnet, run: ``` Ctrl+b :kill-session ``` Then, press `Enter`. ### Clean Up To clean up the node storage, run: ``` cargo run --release -- clean --dev ``` ## 6.4 Feature Flags By default, the metrics feature is turned on for some internal crates. * **history** - Enables a /history REST endpoint. * **telemetry** - Allows the node to upload telemetry data. * **cuda** - Allows some operations to run on the (NVidia) GPU, instead of on the CPU. See [CUDA acceleration for provers](#optional-cuda-acceleration-for-provers) for install tips and current puzzle status. * **locktick** - This feature turns on code for detecting deadlocks. * **test_targets** - This feature allows the lowering of coinbase and proof targets for testing. ## 6.5 Local backups The snarkOS node implementation uses rocksdb under the hood. By using its native checkpointing mechanism, you can create backups locally and efficiently. The backups leverage hard links on your filesystem, thereby incurring only a marginal amount of extra space. The aim of these local backups is for you to be able to recover quickly in case your node were to halt. You can find a basic sample script in `scripts/backup.sh` which you can run as a cron-job e.g. every minute. Each run of the script creates a new backup folder with a timestamp postfix. It will ensure a backup is kept which is 1 minute old, 5 minutes old, 1 hour old and 1 day old. In more detail, on each run it will: - always overwrite the latest backup - only overwrite the 5 minute backup if it is older than 5 minutes - only overwrite the 1 hour backup if it is older than 1 hour - only overwrite the 1 day backup if it is older than 1 day You may want to change the `NETWORK`, `BASE_DIR`, `ENDPOINT` and `JWT` variables. ## 7. Contributors Thank you for helping make snarkOS better! [🧐 What do the emojis mean?](https://allcontributors.org/docs/en/emoji-key)

_{Howard Wu} 💻 🚧 🤔 👀	_{Raymond Chu} 💻 🚧 🤔 👀	_ljedrz 💻 🚧 🤔 👀	_{Niklas Long} 💻 🚧 🤔 👀	_{Collin Chin} 💻 📖 👀	_{Mike Turner} 💻 📖 👀	_{Georgios Konstantopoulos} 💻
_{Kobi Gurkan} 💻	_Vesa-Ville 💻	_jules 💻	_Daniil 💻	_akattis 💻	_{William Cannon} 💻	_wcannon-aleo 💻
_{Sam De Roeck} 💻	_soft2dev 💻	_{Ali Mousa} 💻	_pyk 💻	_Belsy 💻	_apruden2008 💻	_{Fabiano Prestes} 💻
_Haruka 💻	_e4m7he6g 💻	_{Gregório Granado Magalhães} 💻	_{Evgeny Garanin} 💻	_{Macro Hoober} 💻	_{code-pangolin} 💻	_kaola526 💻
_clarenous 💻	_Kostyan 💻	_{Austin Abell} 💻	_{Youssef El Housni} 💻	_{ghostant-1017} 💻	_{Miguel Gargallo} 💻	_{Chines Wang} 💻
_{Ayush Goswami} 💻	_{Tim - o2Stake} 💻	_liu-sen 💻	_Palamar 💻	_swift-mx 💻	_{Caesar Wang} 💻	_{Paul IP} 💻
_{Philip Glazman} 💻	_{Ruslan Nigmatulin} 💻	_{François Garillot} 💻	_aolcr 💻	_{Maciej Zwoliński} 💻	_{Nacho Avecilla} 💻	_{Max Bruce} 💻
_Belsy 💻	_Santala 💻	_deadline 💻	_{CedricYanYuhui} 💻	_{Craig Johnson} 💻	_{Vaclav Barta} 💻	_Dependabot 💻
Add your contributions

This project follows the [all-contributors](https://github.com/all-contributors/all-contributors) specification. Contributions of any kind are welcome! ## 8. License We welcome all contributions to `snarkOS`. Please refer to the [license](#7-license) for the terms of contributions. [![License: GPL v3](https://img.shields.io/badge/License-Apache%202.0-blue.svg)](./LICENSE.md)

Statistics

Topics

Details

Quality Indicators