plus dedicated physical network ports ❏ Security - you are the only tenant on your hardware ❏ Hardware customizations - Packet accommodates custom hardware needs very well ❏ Global coverage with regional data centers around the world Other Advantages of Packet
❏ No native virtual networking (VPCs) ❏ Missing services ❏ Network isolation (cloud firewall) ❏ Load balancing ❏ Host protection ❏ Basic storage ❏ And many more Challenges with Bare Metal Cloud
Network policies, PSP, updates ❏ Based on Typhoon - derived from CoreOS’ Tectonic ❏ Includes a collection of components for k8s ❏ Networking, load balancers, storage, authentication, etc. ❏ Runs atop Flatcar Linux ❏ Fork of CoreOS’ Container Linux ❏ Immutable, container-optimized OS ❏ https://flatcar-linux.org
my-service spec: selector: app: my-app ports: port: 80 targetPort: 8080 type: LoadBalancer “When creating a service, you have the option of automatically creating a cloud network load balancer. This provides an externally-accessible IP address that sends traffic to the correct port on your cluster nodes provided your cluster runs in a supported environment and is configured with the correct cloud load balancer provider package.” https://kubernetes.io/docs/tasks/access-applicat ion-cluster/create-external-load-balancer/
external to the cluster ❏ Runs in pods in the Kubernetes cluster ❏ Uses standard network protocols ❏ ARP (layer 2 mode) ❏ BGP (layer 3 mode) ❏ Assigns IP addresses from a pool to k8s services ❏ Even traffic distribution using ECMP, leveraging network hardware load balancing capabilities ❏ High availability using standard BGP convergence https://github.com/danderson/metallb Load Balancing
nodes advertising the service IP ⇒ Must balance worker nodes across racks, or get uneven LB ❏ Officially not compatible with Calico BGP peering to ToR ❏ May be possible with some workarounds - would like to see this officially supported in future ❏ https://metallb.universe.tf/configuration/calico/ Considerations with MetalLB
on K8s pods, but it also supports network policies applied to the host itself - i.e. a host-based firewall ❏ Compensates for lack of cloud provider firewall rules ❏ No need to route traffic through a virtual firewall appliance ❏ Policy is enforced automatically on all hosts ❏ Configure using Calico policy model (similar to & extends Kubernetes network policy API) ❏ Policy rules automatically translated into iptables rules ❏ Easy policy updates https://docs.projectcalico.org/v3.7/security/host-endpoints/ Host Protection
allow-ssh spec: selector: nodetype == 'worker' order: 0 preDNAT: true applyOnForward: true ingress: - action: Allow protocol: TCP source: nets: [10.1.2.3/24] destination: ports: [22] Can also define GlobalNetworkSet to manage sets of IP addresses and select them via labels in rules
Ideally would be created by Calico on Kubernetes node creation ❏ XDP/BPF instead of iptables ❏ From Calico 3.7, rules optionally implemented via XDP (development by Kinvolk engineers) ❏ Enforcement at the closest possible to the network, to improve denial-of-service resilience: ❏ Smart NIC ❏ Network driver ❏ Earlier in kernel datapath (BPF) Host Protection: Challenges / Futures
v1.0) ❏ Requires some manual operations both during setup and during day 2 operations. ❏ Still easier than provisioning local storage manually on nodes. ❏ Built-in replication should be disabled when used with an app which already does replication on its own (e.g. Kafka)
months Project payback time $K hundreds Ongoing annual savings Results “Financially this was a no-brainer. More importantly, the combination of Packet’s bare metal hosting and Kinvolk’s Lokomotive sets us up with a flexible multi-cloud architecture for the future, enabling us to cost- effectively deliver superior solutions to our customers.” - Bruno Wozniak Director of Engineering, PubNative