Validator

How to run a Validator

OS Requirements

We highly recommend installing MyIonCtrl using the supported operating systems:

• Ubuntu 20.04

• Ubuntu 22.04

• Debian 11

Hardware Requirements

Node Usage on Personal Local Machine

You shouldn't run any type of node on your personal local machine for long, even if it satisfies the requirements. Nodes actively use disks and can damage them quickly.

Validator Node

Minimal Hardware Requirements

• 16 cores CPU

• 128 GB RAM

• 1TB NVME SSD OR Provisioned 64+k IOPS storage

• 1 Gbit/s network connectivity

• Public IP address (fixed IP address)

• 100 TB/month traffic on peak load

Typically, you'll need at least a 1 Gbit/s connection to reliably accommodate peak loads (the average load is expected to be approximately 100 Mbit/s).

We draw special attention to IOPS disk requirements, as it is crucially important for smooth network operation.

Port Forwarding

All types of nodes require a static external IP address, one UDP port to be forwarded for incoming connections, and all outgoing connections to be open. The node uses random ports for new outgoing connections and must be visible to the outside world over NAT.

This can be done with your network provider or by renting a server to run a node.

You can find out which UDP port is open using the netstat -tulpn command.

Run a Node

Switch to Non-Root User

This step is required to successfully install and use MyIonCtrl — don't ignore non-root user creation. Without this step, there will be no errors during installation, but MyIonCtrl will not work properly.

If you don't have a non-root user, you can create one with the following steps (otherwise skip the first two steps and go to the third):

1. Login as root and create a new user:

   sudo adduser <username>

2. Add your user to the sudo group:

   sudo usermod -aG sudo <username>

3. Log into the new user (if you are using SSH, you will need to stop the current session and reconnect with the correct user):

   ssh <username>@<server-ip-address>

Install MyIonCtrl

Download and run the installation script from the non-root user account with sudo privileges:

wget https://raw.githubusercontent.com/ice-blockchain/myionctrl/refs/heads/ion-mainnet/scripts/install.sh
sudo bash install.sh -m validator

Install Options

• -d - Download a dump of the latest blockchain state. This will reduce synchronization time by several times. Dump is currently not activated. Please avoid using it.

• -i - Ignore minimum requirements, use it only if you want to check the compilation process without real node usage.

• -m - Mode: can be validator or liteserver.

Run MyIonCtrl

Run MyIonCtrl console from the local user account used for installation:

myionctrl

Check the MyIonCtrl status using the status command:

status

The following statuses should be displayed:

• myioncore status: Should be in green.

• local validator status: Should also be in green.

• local validator out of sync: Initially, a n/a string is displayed. As soon as the newly created validator connects with other validators, the number will be bigger. As synchronization progresses, this number decreases. When it falls below 20, the validator is synchronized.

Example of the status command output

Make sure you have the same output for status. For all nodes, the "Local Validator status" section should appear. Otherwise, check the troubleshooting section and check node logs.

Wait until "Local validator out of sync" becomes less than 20 seconds.

When a new node has been started, even from dump, it's needed to wait up to 3 hours before the "out of sync" number starts to decrease. This is because the node still needs to establish its place in the network, propagate its addresses via DHT tables, etc.

Uninstall MyIonCtrl

Download the script and run it:

sudo bash /usr/src/myionctrl/uninstall.sh

Archive Node

Configuration Steps

1. Use this configuration only if you intend to run the node as an archive node. If you are unsure about the purpose or requirements of an archive node, it is likely not suitable for your needs.

2. Update the ExecStart parameter in the /etc/systemd/system/validator.service file to include the following:

   ExecStart=/usr/bin/ion/validator-engine/validator-engine --threads 11 --daemonize --global-config /usr/bin/ion/global.config.json --db /var/ion-work/db/ --logname /var/ion-work/log --state-ttl 315360000 --archive-ttl 315360000 --block-ttl 315360000 --verbosity 1

   This configuration ensures 10 years of data retention.

3. After modifying the file, reload the systemd configuration with the following command:

   systemctl daemon-reload

An archive node can function as either a validator or a lite-server, provided it has sufficient storage. Archive nodes require significantly more storage than standard validators, with 4 TB being the recommended minimum.

Tips & Tricks

List of Available Commands

You can use the help command to get a list of available commands:

help

Validator Wallet

View the List of Wallets

Check out the list of available wallets in the MyIonCtrl console using the wl command:

wl

During the installation of MyIonCtrl, the validator_wallet_001 wallet is created:

Activate the Wallets

Send the necessary number of coins to the wallet and activate it.

Activate the wallet using the aw command (wallet name is optional; if no arguments provided, it will activate all available wallets):

aw [wallet name]

You should set the stake size:

set stake 1500000

Your Validator is Now Ready

myioncore will automatically join the elections. If the bet is accepted and your node becomes a validator, the bet can only be withdrawn in the second election (according to the rules of the electorate).

Maintain Guidelines

Slashing of Poor Validators

If the validator processes less than 90% of the expected number of blocks during a validation round, it will be fined by 404 ICE.

As an ION validator, follow these crucial steps to ensure network stability and avoid slashing penalties in the future.

Essential Actions

• Ensure your hardware meets or exceeds minimal system requirements.

• Use MyIonCtrl.

• Keep MyIonCtrl updated as per notifications.

• Set up monitoring dashboards for RAM, Disk, Network, and CPU usage.

• Monitor the efficiency of your validator with dashboards.

• Check with MyIonCtrl via check_ef.

• Build dashboards with APIs.

Validation and Effectiveness APIs

MyIonCtrl allows checking the effectiveness of validators via the check_ef command, which outputs your validator efficiency data for the last round and the current round. This command retrieves data by calling the checkloadall utility. Ensure that your efficiency is greater than 90% for the full round period.

In case of low efficiency, take action to fix the problem.

Database Grooming

ION Node/Validator keeps its database within the path specified by the --db flag of validator-engine, usually /var/ion-work/db. This directory is created and managed by the node, but it is recommended to perform a database grooming/cleanup task once a month to remove some artifacts.

Do not forget to stop the validator process before performing these steps, as failure to do so will likely cause database corruption.

Steps for Database Cleanup

1. Switch to root user:

   sudo -s

2. Stop the validator service:

   systemctl stop validator

3. Verify that the validator is not running:

   systemctl status validator

4. Perform database cleanup:

   find /var/ion-work/db -name 'LOG.old*' -exec rm {} +

5. Start the validator service:

   systemctl start validator

6. Verify that the validator process is running by analyzing the processes and log. The validator should re-sync with the network within a few minutes.

Backups

The easiest and most efficient way to backup the validator is to copy crucial node configuration files, keys, and MyIonCtrl settings:

• Node configuration file: /var/ion-work/db/config.json

• Node private keyring: /var/ion-work/db/keyring

• Node public keys: /var/ion-work/keys

• MyIonCtrl configuration and wallets: $HOME/.local/share/myion* (where $HOME is the home directory of the user who started the installation of MyIonCtrl) or /usr/local/bin/myioncore if installed as root.

This set is everything you need to perform a recovery of your node from scratch.

Snapshots

Modern file systems, such as ZFS, offer snapshot functionality, and most cloud providers also allow customers to make snapshots of their machines, preserving the entire disk for future use.

You must stop the node before performing a snapshot. Failure to do so will most likely result in a corrupt database with unexpected consequences.

Snapshots should not be performed often. If you snapshot your node once a week, then in the worst-case scenario, after recovery, you will have a week-old database, and it will take your node more time to catch up with the network than to perform a new installation using MyIonCtrl's "install from dump" feature (-d flag).

Disaster Recovery

To perform recovery of your node on a new machine:

1. Install MyIonCtrl/node (add -d switch for faster node initialization).

2. Switch to root user:

   sudo -s

3. Stop MyIonCore and validator processes:

   systemctl stop validator
   systemctl stop myioncore

4. Apply backed-up node configuration files:

   • Node configuration file: /var/ion-work/db/config.json

   • Node private keyring: /var/ion-work/db/keyring

   • Node public keys: /var/ion-work/keys

5. Set node IP address (if different):

   • Edit /var/ion-work/db/config.json and set the leaf .addrs[0].ip to the decimal representation of the new IP address. You can use a Python script to convert your IP to decimal.

6. Ensure proper database permissions:

   chown -R validator:validator /var/ion-work/db

7. Apply backed-up MyIonCtrl configuration files:

   • Replace $HOME/.local/share/myion* (ensure that the user is the owner of all files copied).

8. Start MyIonCore and validator processes:

   systemctl start validator
   systemctl start myioncore

Security

Host Level Security

Never install MyIonCtrl under the root user. Use a service account to ensure privilege separation.

Network Level Security

ION Validators are high-value assets that should be protected against external threats. Make your node as invisible as possible by locking down all network connections. On a validator node, only the UDP port used for node operations should be exposed to the internet.

Tools

• ufw firewall interface

• jq JSON command line processor

Setup

1. Install ufw and jq:

   sudo apt install -y ufw jq

2. Basic lockdown of ufw ruleset:

   sudo ufw default deny incoming; sudo ufw default allow outgoing

3. Disable automated ICMP echo request accept:

   sudo sed -i 's/-A ufw-before-input -p icmp --icmp-type echo-request -j ACCEPT/#-A ufw-before-input -p icmp --icmp-type echo-request -j ACCEPT/g' /etc/ufw/before.rules

4. Enable access from management networks:

   sudo ufw insert 1 allow from <MANAGEMENT_NETWORK>

   Repeat for each management network/address.

5. Expose node/validator UDP port to public:

   sudo ufw allow proto udp from any to any port `sudo jq -r '.addrs[0].port' /var/ion-work/db/config.json`

6. Double-check your management networks.

7. Enable ufw firewall:

   sudo ufw enable

8. To check the firewall status:

   sudo ufw status numbered

   Example output of a locked-down node with two management networks:

   Status: active
   
        To                         Action      From
        --                         ------      ----
   [ 1] Anywhere                   ALLOW IN    <MANAGEMENT_NETWORK_A>/28
   [ 2] Anywhere                   ALLOW IN    <MANAGEMENT_NETWORK_B>/32
   [ 3] <NODE_PORT>/udp            ALLOW IN    Anywhere
   [ 4] <NODE_PORT>/udp (v6)       ALLOW IN    Anywhere (v6)

Expose LiteServer Port

sudo ufw allow proto tcp from any to any port `sudo jq -r '.liteservers[0].port' /var/ion-work/db/config.json`

The LiteServer port should not be exposed publicly on a validator.

IP Switch

If you believe that your node is under attack, consider switching your IP address. The approach depends on your hosting provider, such as pre-ordering a second address, cloning your stopped VM, or setting up a new instance via the disaster recovery process.

Ensure you update your new IP address in the node configuration file.