# Validator

## OS Requirements

We highly recommend installing MyIonCtrl using the supported operating systems:

* Ubuntu 20.04
* Ubuntu 22.04
* Debian 11

## Hardware Requirements

### Node Usage on Personal Local Machine

{% hint style="danger" %}
You shouldn't run any type of node on your personal local machine for long, even if it satisfies the requirements. Nodes actively use disks and can damage them quickly.
{% endhint %}

### Validator Node

#### **Minimal Hardware Requirements**

* 16 cores CPU
* 128 GB RAM
* 1TB NVME SSD OR Provisioned 64+k IOPS storage
* 1 Gbit/s network connectivity
* Public IP address (fixed IP address)
* 100 TB/month traffic on peak load

Typically, you'll need at least a 1 Gbit/s connection to reliably accommodate peak loads (the average load is expected to be approximately 100 Mbit/s).

{% hint style="info" %}
We draw special attention to IOPS disk requirements, as it is crucially important for smooth network operation.
{% endhint %}

### Port Forwarding

All types of nodes require a static external IP address, one UDP port to be forwarded for incoming connections, and all outgoing connections to be open. The node uses random ports for new outgoing connections and must be visible to the outside world over NAT.

This can be done with your network provider or by renting a server to run a node.

{% hint style="info" %}
You can find out which UDP port is open using the `netstat -tulpn` command.
{% endhint %}

## Run a Node

### Switch to Non-Root User

{% hint style="warning" %}
This step is required to successfully install and use MyIonCtrl — don't ignore non-root user creation. Without this step, there will be no errors during installation, but MyIonCtrl will not work properly.
{% endhint %}

If you don't have a non-root user, you can create one with the following steps (otherwise skip the first two steps and go to the third):

1. Login as root and create a new user:

   ```bash
   sudo adduser <username>
   ```
2. Add your user to the sudo group:

   ```bash
   sudo usermod -aG sudo <username>
   ```
3. Log into the new user (if you are using SSH, you will need to stop the current session and reconnect with the correct user):

   ```bash
   ssh <username>@<server-ip-address>
   ```

### Install MyIonCtrl

Download and run the installation script from the non-root user account with sudo privileges:

{% code overflow="wrap" %}

```bash
wget https://raw.githubusercontent.com/ice-blockchain/myionctrl/refs/heads/ion-mainnet/scripts/install.sh
sudo bash install.sh -m validator
```

{% endcode %}

### **Install Options**

* `-d` - Download a dump of the latest blockchain state. This will reduce synchronization time by several times. **Dump is currently not activated. Please avoid using it.**
* `-i` - Ignore minimum requirements, use it only if you want to check the compilation process without real node usage.
* `-m` - Mode: can be `validator` or `liteserver`.

### Run MyIonCtrl

Run MyIonCtrl console from the local user account used for installation:

```bash
myionctrl
```

<figure><img src="/files/NF0VOJSwJvLGqdupUOs3" alt=""><figcaption></figcaption></figure>

Check the MyIonCtrl status using the `status` command:

```bash
status
```

The following statuses should be displayed:

* **myioncore status**: Should be in green.
* **local validator status**: Should also be in green.
* **local validator out of sync**: Initially, a `n/a` string is displayed. As soon as the newly created validator connects with other validators, the number will be bigger. As synchronization progresses, this number decreases. When it falls below 20, the validator is synchronized.

<figure><img src="/files/k98UFTTfHvJKSmPrNFEu" alt=""><figcaption><p><strong>Example of the status command output</strong></p></figcaption></figure>

Make sure you have the same output for `status`. For all nodes, the "Local Validator status" section should appear. Otherwise, check the troubleshooting section and check node logs.

Wait until "Local validator out of sync" becomes less than 20 seconds.

When a new node has been started, even from dump, it's needed to wait up to 3 hours before the "out of sync" number starts to decrease. This is because the node still needs to establish its place in the network, propagate its addresses via DHT tables, etc.

### Uninstall MyIonCtrl

Download the script and run it:

```bash
sudo bash /usr/src/myionctrl/uninstall.sh
```

### Archive Node

#### **Configuration Steps**

1. Use this configuration only if you intend to run the node as an archive node. If you are unsure about the purpose or requirements of an archive node, it is likely not suitable for your needs.
2. Update the `ExecStart` parameter in the `/etc/systemd/system/validator.service` file to include the following:

   <pre class="language-bash" data-overflow="wrap"><code class="lang-bash">ExecStart=/usr/bin/ion/validator-engine/validator-engine --threads 11 --daemonize --global-config /usr/bin/ion/global.config.json --db /var/ion-work/db/ --logname /var/ion-work/log --state-ttl 315360000 --archive-ttl 315360000 --block-ttl 315360000 --verbosity 1
   </code></pre>

   This configuration ensures 10 years of data retention.
3. After modifying the file, reload the systemd configuration with the following command:

   ```bash
   systemctl daemon-reload
   ```

{% hint style="info" %}
An archive node can function as either a validator or a lite-server, provided it has sufficient storage. Archive nodes require significantly more storage than standard validators, with 4 TB being the recommended minimum.
{% endhint %}

## Tips & Tricks

### List of Available Commands

You can use the `help` command to get a list of available commands:

```bash
help
```

## Validator Wallet

### View the List of Wallets

Check out the list of available wallets in the MyIonCtrl console using the `wl` command:

```bash
wl
```

During the installation of MyIonCtrl, the `validator_wallet_001` wallet is created:

### Activate the Wallets

Send the necessary number of coins to the wallet and activate it.

Activate the wallet using the `aw` command (wallet name is optional; if no arguments provided, it will activate all available wallets):

```bash
aw [wallet name]
```

You should set the stake size:

```bash
set stake 1500001
```

### Your Validator is Now Ready

`myioncore` will automatically join the elections. If the bet is accepted and your node becomes a validator, the bet can only be withdrawn in the second election (according to the rules of the electorate).

## Maintain Guidelines

### Slashing of Poor Validators

If the validator processes less than 90% of the expected number of blocks during a validation round, it will be fined by 404 ICE.

As an ION validator, follow these crucial steps to ensure network stability and avoid slashing penalties in the future.

### Essential Actions

* Ensure your hardware meets or exceeds minimal system requirements.
* Use MyIonCtrl.
* Keep MyIonCtrl updated as per notifications.
* Set up monitoring dashboards for RAM, Disk, Network, and CPU usage.
* Monitor the efficiency of your validator with dashboards.
* Check with MyIonCtrl via `check_ef`.
* Build dashboards with APIs.

### Validation and Effectiveness APIs

MyIonCtrl allows checking the effectiveness of validators via the `check_ef` command, which outputs your validator efficiency data for the last round and the current round. This command retrieves data by calling the `checkloadall` utility. Ensure that your efficiency is greater than 90% for the full round period.

{% hint style="info" %}
In case of low efficiency, take action to fix the problem.
{% endhint %}

### Database Grooming

ION Node/Validator keeps its database within the path specified by the `--db` flag of validator-engine, usually `/var/ion-work/db`. This directory is created and managed by the node, but it is recommended to perform a database grooming/cleanup task once a month to remove some artifacts.

{% hint style="danger" %}
Do not forget to stop the validator process before performing these steps, as failure to do so will likely cause database corruption.
{% endhint %}

#### **Steps for Database Cleanup**

1. Switch to root user:

   ```bash
   sudo -s
   ```
2. Stop the validator service:

   ```bash
   systemctl stop validator
   ```
3. Verify that the validator is not running:

   ```bash
   systemctl status validator
   ```
4. Perform database cleanup:

   ```bash
   find /var/ion-work/db -name 'LOG.old*' -exec rm {} +
   ```
5. Start the validator service:

   ```bash
   systemctl start validator
   ```
6. Verify that the validator process is running by analyzing the processes and log. The validator should re-sync with the network within a few minutes.

### Backups

The easiest and most efficient way to backup the validator is to copy crucial node configuration files, keys, and MyIonCtrl settings:

* Node configuration file: `/var/ion-work/db/config.json`
* Node private keyring: `/var/ion-work/db/keyring`
* Node public keys: `/var/ion-work/keys`
* MyIonCtrl configuration and wallets: `$HOME/.local/share/myion*` (where `$HOME` is the home directory of the user who started the installation of MyIonCtrl) or `/usr/local/bin/myioncore` if installed as root.

This set is everything you need to perform a recovery of your node from scratch.

### Snapshots

Modern file systems, such as ZFS, offer snapshot functionality, and most cloud providers also allow customers to make snapshots of their machines, preserving the entire disk for future use.

{% hint style="warning" %}
You must stop the node before performing a snapshot. Failure to do so will most likely result in a corrupt database with unexpected consequences.
{% endhint %}

Snapshots should not be performed often. If you snapshot your node once a week, then in the worst-case scenario, after recovery, you will have a week-old database, and it will take your node more time to catch up with the network than to perform a new installation using MyIonCtrl's "install from dump" feature (`-d` flag).

### Disaster Recovery

To perform recovery of your node on a new machine:

1. Install MyIonCtrl/node (add `-d` switch for faster node initialization).
2. Switch to root user:

   ```bash
   sudo -s
   ```
3. Stop MyIonCore and validator processes:

   ```bash
   systemctl stop validator
   systemctl stop myioncore
   ```
4. Apply backed-up node configuration files:
   * Node configuration file: `/var/ion-work/db/config.json`
   * Node private keyring: `/var/ion-work/db/keyring`
   * Node public keys: `/var/ion-work/keys`
5. Set node IP address (if different):
   * Edit `/var/ion-work/db/config.json` and set the leaf `.addrs[0].ip` to the decimal representation of the new IP address. You can use a Python script to convert your IP to decimal.
6. Ensure proper database permissions:

   ```bash
   chown -R validator:validator /var/ion-work/db
   ```
7. Apply backed-up MyIonCtrl configuration files:
   * Replace `$HOME/.local/share/myion*` (ensure that the user is the owner of all files copied).
8. Start MyIonCore and validator processes:

   ```bash
   systemctl start validator
   systemctl start myioncore
   ```

## Security

### **Host Level Security**

Never install MyIonCtrl under the root user. Use a service account to ensure privilege separation.

### **Network Level Security**

ION Validators are high-value assets that should be protected against external threats. Make your node as invisible as possible by locking down all network connections. On a validator node, only the UDP port used for node operations should be exposed to the internet.

### **Tools**

* `ufw` firewall interface
* `jq` JSON command line processor

### **Setup**

1. Install `ufw` and `jq`:

   ```bash
   sudo apt install -y ufw jq
   ```
2. Basic lockdown of `ufw` ruleset:

   ```bash
   sudo ufw default deny incoming; sudo ufw default allow outgoing
   ```
3. Disable automated ICMP echo request accept:

   <pre class="language-bash" data-overflow="wrap"><code class="lang-bash">sudo sed -i 's/-A ufw-before-input -p icmp --icmp-type echo-request -j ACCEPT/#-A ufw-before-input -p icmp --icmp-type echo-request -j ACCEPT/g' /etc/ufw/before.rules
   </code></pre>
4. Enable access from management networks:

   ```bash
   sudo ufw insert 1 allow from <MANAGEMENT_NETWORK>
   ```

   Repeat for each management network/address.
5. Expose node/validator UDP port to public:

   <pre class="language-bash" data-overflow="wrap"><code class="lang-bash">sudo ufw allow proto udp from any to any port `sudo jq -r '.addrs[0].port' /var/ion-work/db/config.json`
   </code></pre>
6. Double-check your management networks.
7. Enable `ufw` firewall:

   ```bash
   sudo ufw enable
   ```
8. To check the firewall status:

   ```bash
   sudo ufw status numbered
   ```

   Example output of a locked-down node with two management networks:

   ```
   Status: active

        To                         Action      From
        --                         ------      ----
   [ 1] Anywhere                   ALLOW IN    <MANAGEMENT_NETWORK_A>/28
   [ 2] Anywhere                   ALLOW IN    <MANAGEMENT_NETWORK_B>/32
   [ 3] <NODE_PORT>/udp            ALLOW IN    Anywhere
   [ 4] <NODE_PORT>/udp (v6)       ALLOW IN    Anywhere (v6)
   ```

### **Expose LiteServer Port**

{% code overflow="wrap" %}

```bash
sudo ufw allow proto tcp from any to any port `sudo jq -r '.liteservers[0].port' /var/ion-work/db/config.json`
```

{% endcode %}

{% hint style="warning" %}
The LiteServer port should not be exposed publicly on a validator.
{% endhint %}

### **IP Switch**

If you believe that your node is under attack, consider switching your IP address. The approach depends on your hosting provider, such as pre-ordering a second address, cloning your stopped VM, or setting up a new instance via the disaster recovery process.

{% hint style="info" %}
Ensure you update your new IP address in the node configuration file.
{% endhint %}


---

# Agent Instructions: Querying This Documentation

If you need additional information that is not directly available in this page, you can query the documentation dynamically by asking a question.

Perform an HTTP GET request on the current page URL with the `ask` query parameter:

```
GET https://docs.ice.io/ion-blockchain/validator.md?ask=<question>
```

The question should be specific, self-contained, and written in natural language.
The response will contain a direct answer to the question and relevant excerpts and sources from the documentation.

Use this mechanism when the answer is not explicitly present in the current page, you need clarification or additional context, or you want to retrieve related documentation sections.
