4.6 KiB
Installing External IP Controller
This document describes how to expose Kubernetes services using External IP Controller.
Introduction
One of the possible ways to expose k8s services on a bare metal deployment is using External IPs. Each node runs a kube-proxy process which programs iptables rules to trap access to External IPs and redirect them to the correct backends.
So in order to accessa k8s service from the outside we just need to route public traffic to one of k8s worker nodes which has kube-proxy running and thus has needed iptables rules for External IPs.
Deployment scheme
Description
External IP controller is a k8s application which is deployed on top of a k8s cluster and which configures External IPs on k8s worker node(s) to provide IP connectivity.
For further details please read External IP controller documentation
Ansible Playbook
The playbook is utils/kargo/externalip.yaml
and the
ansible role is utils/kargo/roles/externalip
.
The nodes that have externalip
role assigned to them
will run External IP controller application which will manage Kubernetes
services' external IPs. Playbook labels such nodes and then creates
DaemonSet with appropriate nodeSelector
.
External IP scheduler will be running as a standard Deployment (ReplicaSet) with specified number of replicas.
ECMP deployment
Deployment model
In this sample deployment we're going to deploy External IP controller on a set of nodes and provide load balancing and high availability for External IPs based on Equal-cost multi-path routing (ECMP).
For further details please read Documentation about ECMP deployment
Inventory
You can take inventory generated previously for Kargo deployment.
Using utils/kargo/externalip.yaml
playbook with such
inventory will deploy External IP controller on all
kube-node
worker nodes.
Custom yaml
Custom Ansible yaml for ECMP deployment is stored here:
utils/jenkins/extip_ecmp.yaml
. Here is the content:
# Type of deployment
extip_ctrl_app_kind: "DaemonSet"
# IP distribution model
extip_distribution: "all"
# Netmask for external IPs
extip_mask: 32
# Interface to bring IPs on, should be "lo" for ECMP
extip_iface: "lo"
Deployment
Just run the following ansible-playbook command:
export ws=/home/workspace
ansible-playbook -e ansible_ssh_pass=vagrant -u vagrant -b \
-i ${ws}/inventory/inventory.cfg \
--become-user=root ${ws}/utils/jenkins/extip_ecmp.yaml \
-e @${ws}/utils/kargo/externalip.yaml
This will deploy the application according to your inventory.
Routing
This application only brings IPs up or down on a specified interface.
We also need to provide routing to those nodes with external IPs. So for
Kubernetes cluster with Calico networking plugin we already have
calico-node
container running on every k8s worker node.
This container also includes BGP speaker which monitors local routing
tables and announces changes via BGP protocol. So in order to include
external IPs to BGP speaker export we need to add the following custom
export filter for Calico:
cat << EOF | etcdctl set /calico/bgp/v1/global/custom_filters/v4/lo_iface
if ( ifname = "lo" ) then {
if net != 127.0.0.0/8 then accept;
}
EOF
Please note that this will only configure BGP for
calico-node
. In order to announce routing to your network
infrastructure you may want to peer Calico with routers. Please check
this URL for details:
Kargo docs: Calico BGP Peering with border routers
Uninstalling and undoing customizations
Uninstall k8s applications by running the following commands on the first kube-master node in your ansible inventory:
kubectl delete -f /etc/kubernetes/extip_scheduler.yml
kubectl delete -f /etc/kubernetes/extip_controller.yml
Remove custom Calico export filter:
etcdctl rm /calico/bgp/v1/global/custom_filters/v4/lo_iface
Also remove external IPs from lo interface on the nodes with the command like this:
ip ad del 10.0.0.7/32 dev lo
Where 10.0.0.7/32
is external IP.