If you’re a developer or a fan of BTCPay Server and you’re keen to get more out of the Lightning Network, this guide is for you.
We’re going to talk about how to control your dockerized BTCPay Server’s internal LND lightning node using gRPC.
With gRPC, you can automate many tasks, making it easier to manage liquidity and ensure your node is always ready to handle payments. It also allows much finer grained control of your LND node than the BTCPay Server Greenfield API.
For example, for maintaining a balanced node, it’s critical to be able to specify your outgoing channel id when sending rebalancing payments.
The last part of this guide links to some example scripts to get you started.
Let’s dive in and see how it all connects.
- You have a dockerized, Linux based BTCPay Server
- You are using the BTCPay Server internal Lightning node
- The internal node is running LND
- You have root access to the server
- BTCPay Server is installed as the root user
- You are comfortable with the Linux command line
Overview of BTCPay Server LND Connection
We’re going to break this connection down into 5 main parts:
- Modifying the docker container using custom fragments
- Configuring LND to allow remote gRPC connections
- Generating new TLS certificates and securing the macaroon
- Restricting gRPC connections per IP address
Sound good? Let’s get started ⚡
Step-by-Step Setup Guide
1. Modifying the docker container using custom fragments
We’re going to modify our LND docker container to expose the standard gRPC port 10009 for connection. We use a custom fragment so that updates won’t overwrite our modifications.
We are generally following this guide:
1.1 If you want to connect from your local computer to a remote server for testing, you’ll need to start by finding your IP address with something like https://whatismyipaddress.com/.
1.2 Cd into the directory where BTCPay Server keeps it’s custom fragments
1.3 Make a custom yml file. I’m going to assume nano here but use whatever you like. If you like pain, use VIM 😛
$ nano expose-lnd-grpc.custom.yml
Here’s the contents:
version: '3' services: lnd_bitcoin: ports: - "10009:10009"
1.4 Add the custom fragment to our environment variables:
$ export BTCPAYGEN_ADDITIONAL_FRAGMENTS="$BTCPAYGEN_ADDITIONAL_FRAGMENTS;expose-lnd-grpc.custom"
1.5 Apply the changes. Run the setup script from here:
$ . ./btcpay-setup.sh -i
1.6 Test that this was successful
$ docker ps
You should see the port listed now in something like this. Note that the docker container for LND Port 10009 is now open.
5a85a7052343 btcpayserver/lnd:v0.16.4-beta “/sbin/tini -g — /d…” 39 seconds ago Up 38 seconds 0.0.0.0:9735->9735/tcp, :::9735->9735/tcp, 8080-8081/tcp, 0.0.0.0:10009->10009/tcp, :::10009->1
1.7 Test connection from remote server (or computer). This is windows powershell.
$ Test-NetConnection -ComputerName 18.104.22.168 -Port 10009
You should get something like:
ComputerName : [IP ADDRESS OR NAME]
RemoteAddress : [IP ADDRESS]
RemotePort : 10009
InterfaceAlias : Wi-Fi
SourceAddress : [IP ADDRESS]
TcpTestSucceeded : True
2. Configuring LND to allow remote gRPC connections with BTCPay Server
Now that we have port 10009 opened up and tested, we need to configure LND so it will allow remote gRPC connections.
We are generally following this guide:
2.1 Create a custom fragment in
$ nano opt-lnd-config.custom.yml
version: '3' services: lnd_bitcoin: environment: LND_EXTRA_ARGS: | tlsextraip=[ADD IP ADDRESS HERE] tlsextraip=[ADD IP ADDRESS HERE]
2.2 Note that, if we are connecting from a local computer to this remote server, we’ll need to add both the IP Address of the remote server AND the IP address of our local computer.
We can also add additional servers if we’re making multiple connections just by adding additional tlsextraip=[ADD IP ADDRESS HERE] lines.
2.3 We need to run the setup script or restart the lnd docker container for these configuration changes to take effect.
$ . ./btcpay-setup.sh -i
2.4 Test that the lnd.conf file has been successfully updated with the extra IP addresses
Access the docker container’s bash shell:
$ docker exec -it btcpayserver_lnd_bitcoin /bin/bash
# cat /data/lnd.conf
You should see your additional tlsextraip=[ADD IP ADDRESS HERE] lines listed here.
3. Generating new TLS certificates and securing the macaroon
Now that we have the gRPC port opened in the docker container, and configured LND to accept the incoming connection, we need to get authentication. We’ll be doing this by creating copies of the TLS certificate and admin macaroon.
Very very very important note: these are sensitive files! They cannot go in git repos, insecure storage, telegram, instagram etc 🙂
3.1 Get a copy out of the tls.cert file out of the docker container
$ docker cp btcpayserver_lnd_bitcoin:/data/tls.cert /root/tls.cert
3.2 Check out the tls.cert file and make sure our IP address are listed there:
$ openssl x509 -in tls.cert -text -noout
3.3 Next you will switch to the computer that is running the script. If you aren’t setting this up for a remote connection, this step isn’t necessary.
Here’s a powershell example. For linux, I recommend using rsync.
$ scp root@XX.XX.XX.XX:/root/tls.cert "C:\Users\username\path\to\script\tls.cert"
3.4 Get a copy of the admin.macaroon file out of the docker container
$ docker cp btcpayserver_lnd_bitcoin:/data/admin.macaroon /root/admin.macaroon
3.5 Get it onto the local machine (if needed)
$ scp root@XXX.XX.XXX.XX:/root/admin.macaroon "C:\Users\username\path\to\script\admin.macaroon"
3.6 Important step! Are you going to use a git repo? Add the tls.cert file and the admin.macaroon file to the .gitignore file now!
4. Restrict IP addresses that can connect to BTCPay Server
Great, we have everything we need to connect. But let’s make sure no one else can.
We’re going to do this by restricting gRPC connections using UFW (uncomplicated firewall), which comes standard on Ubuntu servers.
Setting up UFW for a server running BTPay Server is pretty easy! In fact, I might extract this into a dedicated post if I’m feeling saucy.
So let’s get anti-social and reject some connections 😛
If you already have UFW (or another firewall) setup, skip to Step 4.4
4.1 Setup default policies to allow all outgoing and disable all incoming traffic
$ sudo ufw default deny incoming $ sudo ufw default allow outgoing
4.2 Allow SSH connections
$ sudo ufw allow ssh
4.3 Allow https and peer-to-peer connections
$ sudo ufw allow 443 $ sudo ufw allow 8333 $ sudo ufw allow 9735
4.4 If you are connecting via gRPC from a local computer or other remote server, we need to add a rule for that:
$ sudo ufw allow from XX.XX.XXX.XXX to any port 10009
4.5 Enable UFW
$ sudo ufw enable
4.6 Verify that you can still SSH connect to your server before doing anything else. If not, disable the UFW and troubleshoot before doing anything else!
4.7 Verify the configuration of UFW
$ sudo ufw status verbose $ sudo ufw status numbered
4.8 Important: verify connections. If this isn’t behaving as expected, disable the UFW and troubleshoot before doing anything else!
- Check the server to make sure that it is still accessible from the browser interface
- Make sure there are no communication error warnings
- Check the logs to make sure there are no errors
- Check you can still SSH in
4.9 Everything working as expected? Let’s move on to the fun part!
5. Make a Python connection script
Note that we have loads more info and script examples listed in the references.
We are generally following the instructions here: https://github.com/lightningnetwork/lnd/blob/master/docs/grpc/python.md
5.1 Make yourself a project directory
$ mkdir btcpay-lnd-connection
5.2 Create a virtual environment in that directory and activate
$ python -m venv venv $ .\venv\Scripts\Activate
5.3 Install dependencies
venv $ pip install grpcio grpcio-tools googleapis-common-protos
5.4 Clone the google api’s repository (required due to the use of google/api/annotations.proto)
venv $ git clone https://github.com/googleapis/googleapis.git
5.5 Copy the lnd lightning.proto file
venv $ Invoke-WebRequest -Uri "https://raw.githubusercontent.com/lightningnetwork/lnd/master/lnrpc/lightning.proto" -OutFile "lightning.proto"
5.6 Compile the lightning.proto file
venv $ python -m grpc_tools.protoc --proto_path=googleapis:. --python_out=. --grpc_python_out=. Lightning.proto
5.7 We’re going to do the same with the router proto file, assuming that you are going to want to make payments for liquidity management. If not, skip these steps.
venv $ Invoke-WebRequest -Uri "https://raw.githubusercontent.com/lightningnetwork/lnd/master/lnrpc/routerrpc/router.proto" -OutFile "router.proto" venv $ python -m grpc_tools.protoc --proto_path=googleapis:. --python_out=. --grpc_python_out=. router.proto
5.8 Ok you should be ready! Next is to make a btcpay-lnd-connection.py script using our example. Note you need to change the IP address to match your remote server.
import lightning_pb2 as ln import lightning_pb2_grpc as lnrpc import grpc import os # Due to updated ECDSA generated tls.cert we need to let gprc know that # we need to use that cipher suite otherwise there will be a handshake # error when we communicate with the lnd rpc server. os.environ["GRPC_SSL_CIPHER_SUITES"] = 'HIGH+ECDSA' # Replace this path with the path to the tls.cert file on your local machine # if you have copied it from your remote machine. cert_path = "tls.cert" # Read the cert file cert = open(cert_path, 'rb').read() creds = grpc.ssl_channel_credentials(cert) # Specify the path to your macaroon file on the local machine macaroon_path = "admin.macaroon" # Read the macaroon file and convert it to hex with open(macaroon_path, 'rb') as f: macaroon_bytes = f.read() macaroon = macaroon_bytes.hex() # Add the macaroon to the gRPC request headers metadata = [('macaroon', macaroon)] auth_creds = grpc.metadata_call_credentials(lambda _, cb: cb(metadata, None)) combined_creds = grpc.composite_channel_credentials(creds, auth_creds) # Replace this with the IP address or domain name of your remote machine remote_server = "XXX.XX.XX.XXX:10009" # Note the change here: we use `combined_creds` instead of just `creds` channel = grpc.secure_channel(remote_server, combined_creds) stub = lnrpc.LightningStub(channel) # Retrieve and display the wallet balance response = stub.WalletBalance(ln.WalletBalanceRequest()) print(response.total_balance)
5.9 Run your script and it should return your LND node on-chain wallet balance!
venv $ python btcpay-lnd-connection.py
5.10 There’s so much more cool stuff you can do once this connection is established. See the conclusion and references for lots of examples.
Did you manage to fetch your on-chain wallet balance? Achieving this connection alone paves the way for numerous exciting possibilities.
Using gRPC, you can do much more than what the BTCPay Server Greenfield API offers, especially with the enhanced control over your LND node.
Just having this successful connection opens up some really cool possibilities.
- Node liquidity management through balancing applications
- Leveraging bi-directional streaming with invoice subscription services
- Sending payments
- Configuring channel acceptors
I hope this serves as a handy reference and inspiration.
If you’re looking for help with BTCPay Server, Bitcoin apps and Lightning automations, make sure to reach out at https://velascommerce.com/#contact