How to publish a static site over NNCP

Published by Beto Dealmeida on 2021-11-16 02:54:52+00:00

A tutorial on how to use NNCP to publish a static site to a server over an unreliable connection

Let's say you're running a static site generator on your laptop, and you want to push the built website to a machine running a web server. A good tool to do that would be rsync, a command-line utility that synchronizes files and directories efficiently, sending only files that were modified. You can use rsync to synchronize a directory between two computers, sending the updates through an encrypted channel via ssh.

But what if your connection to the web server is unreliable? Maybe the server is down from time to time, maybe the network link is unstable. You could just run rsync, and keep retrying until you succeed, but there are better ways. One better solution for this problem is Node to Node copy, a small collection of utilities written in Go as an evolution of UUCP.

Last week I set up NNCP as a way of publishing my blog (https://blog.taoetc.org/) to a Raspberry Pi that will run under solar power and a 4G connection. Since the documentation for NNCP is very sparse I decided to write down the process, hoping it can be useful for other people.

Set up

Let's assume you have two machines, client and server. The client machine has a directory that is updated frequently, and you want to send those files to the server using NNCP. Let's also assume that your static site generator is smart enough to know which files were modified since the last build.

Installation

The first step is to install NNCP in both machines. I couldn't find packages for Debian stable, so I installed Go and compiled NNCP. If you're going to compile NNCP make sure you have Go version 1.13 or higher; you can verify which version you have by running go version. The compilation requires a command called redo, which I didn't have installed. In that case, you can use contrib/do from the package:

contrib/do all

On the client I installed NNCP in the ~/.local directory, since I wanted to run the commands as my user:

PREFIX=~/.local contrib/do install

On the server I created an nncp system account, and installed NNCP to /usr/local:

PREFIX=/usr/local contrib/do install

Configuration

The next step is to create configuration file, using the command nncp-cfgnew. For the client:

nncp-cfgnew > ~/.local/etc/nncp.hjson

And for the server:

nncp-cfgnew > /usr/local/etc/nncp.hjson

These configuration files contain public and private keys, so make sure that no one else can read them:

chmod 600 ~/.local/etc/nncp.hjson

chown nncp:nncp /usr/local/etc/nncp.hjson
chmod 600 /usr/local/etc/nncp.hjson

By default the configuration file will use a spool directory and store a log file in /var/spool/nncp. On the server I made sure to create the directory and give ownership to the system account:

mkdir -p /var/spool/nncp
chown nncp:nncp /var/spool/nncp

On the client I changed the configuration to have the spool and the log file pointing to ~/.local:

{
  # Path to encrypted packets spool directory
  spool: /home/user/.local/var/spool/nncp
  # Path to log file
  log: /home/user/.local/var/spool/nncp/log

And then created the directory:

mkdir -p ~/.local/var/spool/nncp

The next step is modifying the configuration files so that the nodes (computers) are aware of each other.

Creating a friend-to-friend network

NNCP allows us to build a small network of trusted nodes; unlike UUCP there are no anonymous peers in this network, which is why it's called friend-to-friend.

To create the network you need to add the public keys of all neighboring nodes to the configuration file you created in the previous step, under the neigh section. Initially this section could contain only an entry for a node called self, representing the node itself. All you need to do is copy these keys to the other nodes. For example, for the client:

neigh: {
  # client
  self: {
    id: WWW
    exchpub: XXX
    signpub: YYY
    noisepub: ZZZ
  }

  # "self" from server
  server: {
    id: AAA
    exchpub: BBB
    signpub: CCC
    noisepub: DDD
  }
}

And for the server:

neigh: {
  # server
  self: {
    id: AAA
    exchpub: BBB
    signpub: CCC
    noisepub: DDD
  }

  # "self" from client
  client: {
    id: WWW
    exchpub: XXX
    signpub: YYY
    noisepub: ZZZ
  }
}

With this configuration the nodes can identify each other. But how do they communicate? In addition to the public keys you can also put the address where the node can be found. In this case the client will push files to the server, so you only need to configure the address of the server on the client configuration:

neigh: {
  # client
  self: {
    id: WWW
    exchpub: XXX
    signpub: YYY
    noisepub: ZZZ
  }

  # "self" from server
  server: {
    id: AAA
    exchpub: BBB
    signpub: CCC
    noisepub: DDD

    addrs: {
      internet: server.example.com:5400
    }
  }
}

Now, client knows who server is, and how to talk to it. And server knows who client is when receiving a message from it.

Sending a file

OK, now that you have a network, how do you send a file? From the client machine you simply run:

nncp-file -cfg ~/.local/etc/nncp.hjson file.txt server:

If you run the command above you'll notice that it completes very quickly! The file was not actually sent. Instead, it was queued in the spool at ~/.local/var/spool/nncp. Inside the spool there should be a directory with the ID of server ("AAA", in this example), and a symlink called "server" pointing to it. Inside this directory there should be a directory called "tx", and inside an encrypted and compressed version of our file.

To actually send the file you need to call the server:

nncp-call -cfg ~/.local/etc/nncp.hjson server

This command will try to connect to server and send all the packets that are in the spool. It will fail, because the server is not listening to any incoming connections yet.

Configuring the server

On the server you need to run a daemon listening for incoming connections:

nncp-daemon -autotoss -bind server.example.com:5400

You also need to change the server configuration, so that the it will allow incoming file requests from the client:

neigh: {
  # server
  self: {
    id: AAA
    exchpub: BBB
    signpub: CCC
    noisepub: DDD
  }

  # "self" from client
  client: {
    id: WWW
    exchpub: XXX
    signpub: YYY
    noisepub: ZZZ

    incoming: "/path/to/files/from/client"
  }
}

Now the client can send file packets to the server. The autotoss option in the daemon means that these packets will be automatically processed, and moved to the incoming directory.

If you go back to the client and run:

nncp-file -cfg ~/.local/etc/nncp.hjson file.txt server:
nncp-call -cfg ~/.local/etc/nncp.hjson server

The file file.txt will show up in /path/to/files/from/client/file.txt... success!

Synchronizing files

With this setup, how do you periodically sync the files from a static site generator to the server? You could simply have the generator call the command on files that were modified, pushing them to a directory that the server would then serve. For example:

nncp-file -cfg ~/.local/etc/nncp.hjson index.html server:my_blog/index.html
nncp-file -cfg ~/.local/etc/nncp.hjson posts/hello_world/index.html server:my_blog/hello_world/index.html
nncp-file -cfg ~/.local/etc/nncp.hjson css/theme.css server:my_blog/css/theme.css
nncp-call -cfg ~/.local/etc/nncp.hjson server

This works, but it has a problem: whenever a file changes and you run the command again, the file will not be overwritten. For example, if index.html changes, the next time you send it to the server it will be called index.html.0, to prevent the original one from being overwritten. And there's no configuration flag to allow overwriting files.

Fortunately NNCP also allows you to execute remote commands! On the server we can define a command called "sync", that will use rsync to copy the pushed files to the website directory:

neigh: {
  # server
  self: {
    id: AAA
    exchpub: BBB
    signpub: CCC
    noisepub: DDD
  }

  # "self" from client
  client: {
    id: WWW
    exchpub: XXX
    signpub: YYY
    noisepub: ZZZ

    incoming: "/path/to/files/from/client"
    exec: {
      sync: [
        "/usr/bin/rsync",
        "-av",
        "--remove-source-files",
        "/path/to/files/from/client/",
        "/var/www/example.com"
     ]
   }
  }
}

The rsync command will move any new files that show up in the incoming directory to the root of the website. To execute the command the client needs to run:

echo | nncp-exec -cfg ~/.local/etc/nncp.hjson server sync

The nncp-exec command reads from stdin, which is why you need to have the echo command piping into it. With this, your static site generator can now run:

nncp-file -cfg ~/.local/etc/nncp.hjson index.html server:my_blog/index.html
nncp-file -cfg ~/.local/etc/nncp.hjson posts/hello_world/index.html server:my_blog/hello_world/index.html
nncp-file -cfg ~/.local/etc/nncp.hjson css/theme.css server:my_blog/css/theme.css
echo | nncp-exec -cfg ~/.local/etc/nncp.hjson server sync
nncp-call -cfg ~/.local/etc/nncp.hjson server

The commands above will push the files to the server, and invoke rsync remotely to move the files to the website directory.

Order of execution

There's just one problem with the solution above. NNCP does not guarantee that packets are processed in order. When I tested it on my blog the rsync call was running before all the files were transferred. There's a clever way to solve the problem, though: NNCP packets can have different priorities. You can send the file packets with a higher priority than the execution (sync) packet, and call for the higher packets to be processed first:

nncp-file -nice PRIORITY -cfg ~/.local/etc/nncp.hjson index.html server:my_blog/index.html
nncp-file -nice PRIORITY -cfg ~/.local/etc/nncp.hjson posts/hello_world/index.html server:my_blog/hello_world/index.html
nncp-file -nice PRIORITY -cfg ~/.local/etc/nncp.hjson css/theme.css server:my_blog/css/theme.css
echo | nncp-exec -nice NORMAL -cfg ~/.local/etc/nncp.hjson server sync
nncp-call -nice PRIORITY -cfg ~/.local/etc/nncp.hjson server
nncp-call -nice NORMAL -cfg ~/.local/etc/nncp.hjson server

The first nncp-call will process only the file packets, while the second one will process the sync call.

Calling in the background

The nice thing about nncp-file and nncp-exec is that they queue commands and terminate quickly. Ideally the static site generator will call these commands to queue the operations, and the client will run nncp-call in the background periodically, to transfer the files and run the sync whenever the server is up. To do this, you can specify in the client configuration that the server should be called periodically:

neigh: {
  # client
  self: {
    id: WWW
    exchpub: XXX
    signpub: YYY
    noisepub: ZZZ
  }

  # "self" from server
  server: {
    id: AAA
    exchpub: BBB
    signpub: CCC
    noisepub: DDD

    addrs: {
      internet: server.example.com:5400
    }

    calls: [
      {
        cron: "0 * * * *"
        nice: PRIORITY
      }
      {
        cron: "10 * * * *"
        nice: NORMAL
      }
    ]
  }
}

The configuration above will make the client call the server every hour to process priority packets, and 10 minutes later call it again to process normal packets. This way there's no need to manually call nncp-call.