BTCPay Server and LND gRPC: Connecting the Dots

If you’re a developer or a fan of BTCPay Server and you’re keen to get more out of the Lightning Network, this guide is for you.

We’re going to talk about how to control your dockerized BTCPay Server’s internal LND lightning node using gRPC.

With gRPC, you can automate many tasks, making it easier to manage liquidity and ensure your node is always ready to handle payments. It also allows much finer grained control of your LND node than the BTCPay Server Greenfield API.

For example, for maintaining a balanced node, it’s critical to be able to specify your outgoing channel id when sending rebalancing payments.

The last part of this guide links to some example scripts to get you started.

Let’s dive in and see how it all connects.

Assumptions

You have a dockerized, Linux based BTCPay Server
You are using the BTCPay Server internal Lightning node
The internal node is running LND
You have root access to the server
BTCPay Server is installed as the root user
You are comfortable with the Linux command line
Some familiarity with Python or JavaScript

Overview of BTCPay Server LND Connection

We’re going to break this connection down into 5 main parts:

Modifying the docker container using custom fragments
Configuring LND to allow remote gRPC connections
Generating new TLS certificates and securing the macaroon
Restricting gRPC connections per IP address
Make a simple connection script with Python (I’ll link to JavaScript examples too)

Sound good? Let’s get started ⚡

Step-by-Step Setup Guide

1. Modifying the docker container using custom fragments

We’re going to modify our LND docker container to expose the standard gRPC port 10009 for connection. We use a custom fragment so that updates won’t overwrite our modifications.
We are generally following this guide:

https://github.com/btcpayserver/btcpayserver-docker#how-can-i-customize-the-generated-docker-compose-file

1.1 If you want to connect from your local computer to a remote server for testing, you’ll need to start by finding your IP address with something like https://whatismyipaddress.com/.

1.2 Cd into the directory where BTCPay Server keeps it’s custom fragments
/root/BTCPayServer/btcpayserver-docker/docker-compose-generator/docker-fragments

1.3 Make a custom yml file. I’m going to assume nano here but use whatever you like. If you like pain, use VIM 😛

$ nano expose-lnd-grpc.custom.yml

Here’s the contents:

version: '3'
services:
  lnd_bitcoin:
    ports:
      - "10009:10009"

1.4 Add the custom fragment to our environment variables:

$ export BTCPAYGEN_ADDITIONAL_FRAGMENTS="$BTCPAYGEN_ADDITIONAL_FRAGMENTS;expose-lnd-grpc.custom"

1.5 Apply the changes. Run the setup script from here:

/root/BTCPayServer/btcpayserver-docker

$ . ./btcpay-setup.sh -i

1.6 Test that this was successful

$ docker ps

You should see the port listed now in something like this. Note that the docker container for LND Port 10009 is now open.

5a85a7052343 btcpayserver/lnd:v0.16.4-beta “/sbin/tini -g — /d…” 39 seconds ago Up 38 seconds 0.0.0.0:9735->9735/tcp, :::9735->9735/tcp, 8080-8081/tcp, 0.0.0.0:10009->10009/tcp, :::10009->1

1.7 Test connection from remote server (or computer). This is windows powershell.

$ Test-NetConnection -ComputerName 45.33.82.75 -Port 10009

You should get something like:

ComputerName : [IP ADDRESS OR NAME]

RemoteAddress : [IP ADDRESS]

RemotePort : 10009

InterfaceAlias : Wi-Fi

SourceAddress : [IP ADDRESS]

TcpTestSucceeded : True

2. Configuring LND to allow remote gRPC connections with BTCPay Server

Now that we have port 10009 opened up and tested, we need to configure LND so it will allow remote gRPC connections.
We are generally following this guide:

https://docs.btcpayserver.org/FAQ/LightningNetwork/#how-to-edit-lndconf

2.1 Create a custom fragment in

/root/BTCPayServer/docker-compose-generator/docker-fragments/

$ nano opt-lnd-config.custom.yml

version: '3'
services:
  lnd_bitcoin:
    environment:
      LND_EXTRA_ARGS: |
        tlsextraip=[ADD IP ADDRESS HERE]
        tlsextraip=[ADD IP ADDRESS HERE]

2.2 Note that, if we are connecting from a local computer to this remote server, we’ll need to add both the IP Address of the remote server AND the IP address of our local computer.

We can also add additional servers if we’re making multiple connections just by adding additional tlsextraip=[ADD IP ADDRESS HERE] lines.

2.3 We need to run the setup script or restart the lnd docker container for these configuration changes to take effect.

$ . ./btcpay-setup.sh -i

2.4 Test that the lnd.conf file has been successfully updated with the extra IP addresses

Access the docker container’s bash shell:

$ docker exec -it btcpayserver_lnd_bitcoin /bin/bash

Now run:

# cat /data/lnd.conf

You should see your additional tlsextraip=[ADD IP ADDRESS HERE] lines listed here.

3. Generating new TLS certificates and securing the macaroon

Now that we have the gRPC port opened in the docker container, and configured LND to accept the incoming connection, we need to get authentication. We’ll be doing this by creating copies of the TLS certificate and admin macaroon.

Very very very important note: these are sensitive files! They cannot go in git repos, insecure storage, telegram, instagram etc 🙂

3.1 Get a copy out of the tls.cert file out of the docker container

$ docker cp btcpayserver_lnd_bitcoin:/data/tls.cert /root/tls.cert

3.2 Check out the tls.cert file and make sure our IP address are listed there:

$ openssl x509 -in tls.cert -text -noout

3.3 Next you will switch to the computer that is running the script. If you aren’t setting this up for a remote connection, this step isn’t necessary.

Here’s a powershell example. For linux, I recommend using rsync.

$ scp root@XX.XX.XX.XX:/root/tls.cert "C:\Users\username\path\to\script\tls.cert"

3.4 Get a copy of the admin.macaroon file out of the docker container

$ docker cp btcpayserver_lnd_bitcoin:/data/admin.macaroon /root/admin.macaroon

3.5 Get it onto the local machine (if needed)

$ scp root@XXX.XX.XXX.XX:/root/admin.macaroon "C:\Users\username\path\to\script\admin.macaroon"

3.6 Important step! Are you going to use a git repo? Add the tls.cert file and the admin.macaroon file to the .gitignore file now!

4. Restrict IP addresses that can connect to BTCPay Server

Great, we have everything we need to connect. But let’s make sure no one else can.

We’re going to do this by restricting gRPC connections using UFW (uncomplicated firewall), which comes standard on Ubuntu servers.

Setting up UFW for a server running BTPay Server is pretty easy! In fact, I might extract this into a dedicated post if I’m feeling saucy.

So let’s get anti-social and reject some connections 😛

If you already have UFW (or another firewall) setup, skip to Step 4.4

4.1 Setup default policies to allow all outgoing and disable all incoming traffic

$ sudo ufw default deny incoming
$ sudo ufw default allow outgoing

4.2 Allow SSH connections

$ sudo ufw allow ssh

4.3 Allow https and peer-to-peer connections

$ sudo ufw allow 443 
$ sudo ufw allow 8333
$ sudo ufw allow 9735

4.4 If you are connecting via gRPC from a local computer or other remote server, we need to add a rule for that:

$ sudo ufw allow from XX.XX.XXX.XXX to any port 10009

4.5 Enable UFW

$ sudo ufw enable

4.6 Verify that you can still SSH connect to your server before doing anything else. If not, disable the UFW and troubleshoot before doing anything else!

4.7 Verify the configuration of UFW

$ sudo ufw status verbose
$ sudo ufw status numbered

4.8 Important: verify connections. If this isn’t behaving as expected, disable the UFW and troubleshoot before doing anything else!

Check the server to make sure that it is still accessible from the browser interface
Make sure there are no communication error warnings
Check the logs to make sure there are no errors
Check you can still SSH in

4.9 Everything working as expected? Let’s move on to the fun part!

5. Make a Python connection script

Ok now for the fun part! We should be all set up and ready to make a connection script. I am using Python but there are also lots of examples in JavaScript too.

Note that we have loads more info and script examples listed in the references.

We are generally following the instructions here: https://github.com/lightningnetwork/lnd/blob/master/docs/grpc/python.md

5.1 Make yourself a project directory

$ mkdir btcpay-lnd-connection

5.2 Create a virtual environment in that directory and activate

$ python -m venv venv
$ .\venv\Scripts\Activate

5.3 Install dependencies

venv $  pip install grpcio grpcio-tools googleapis-common-protos

5.4 Clone the google api’s repository (required due to the use of google/api/annotations.proto)

venv $  git clone https://github.com/googleapis/googleapis.git

5.5 Copy the lnd lightning.proto file

venv $ Invoke-WebRequest -Uri "https://raw.githubusercontent.com/lightningnetwork/lnd/master/lnrpc/lightning.proto" -OutFile "lightning.proto"

5.6 Compile the lightning.proto file

venv $  python -m grpc_tools.protoc --proto_path=googleapis:. --python_out=. --grpc_python_out=. Lightning.proto

5.7 We’re going to do the same with the router proto file, assuming that you are going to want to make payments for liquidity management. If not, skip these steps.

venv $ Invoke-WebRequest -Uri "https://raw.githubusercontent.com/lightningnetwork/lnd/master/lnrpc/routerrpc/router.proto" -OutFile "router.proto"

venv $  python -m grpc_tools.protoc --proto_path=googleapis:. --python_out=. --grpc_python_out=. router.proto

5.8 Ok you should be ready! Next is to make a btcpay-lnd-connection.py script using our example. Note you need to change the IP address to match your remote server.

import lightning_pb2 as ln
import lightning_pb2_grpc as lnrpc
import grpc
import os

# Due to updated ECDSA generated tls.cert we need to let gprc know that
# we need to use that cipher suite otherwise there will be a handshake
# error when we communicate with the lnd rpc server.
os.environ["GRPC_SSL_CIPHER_SUITES"] = 'HIGH+ECDSA'

# Replace this path with the path to the tls.cert file on your local machine
# if you have copied it from your remote machine.
cert_path = "tls.cert"

# Read the cert file
cert = open(cert_path, 'rb').read()
creds = grpc.ssl_channel_credentials(cert)

# Specify the path to your macaroon file on the local machine
macaroon_path = "admin.macaroon"

# Read the macaroon file and convert it to hex
with open(macaroon_path, 'rb') as f:
    macaroon_bytes = f.read()
    macaroon = macaroon_bytes.hex()

# Add the macaroon to the gRPC request headers
metadata = [('macaroon', macaroon)]
auth_creds = grpc.metadata_call_credentials(lambda _, cb: cb(metadata, None))
combined_creds = grpc.composite_channel_credentials(creds, auth_creds)

# Replace this with the IP address or domain name of your remote machine
remote_server = "XXX.XX.XX.XXX:10009"

# Note the change here: we use `combined_creds` instead of just `creds`
channel = grpc.secure_channel(remote_server, combined_creds)
stub = lnrpc.LightningStub(channel)

# Retrieve and display the wallet balance
response = stub.WalletBalance(ln.WalletBalanceRequest())
print(response.total_balance)

5.9 Run your script and it should return your LND node on-chain wallet balance!