Replacing multiple Vagrant VMs with LXC containers
I’m currently working in an environment where the development setup consists of 6 individual VirtualBox VMs handled by vagrant. This could be a problem on devices with low amount of memory, such as Macbook Air …
This post describes a setup with a single VM which hosts all other VMs as LXC containers with minimal overhead. The tricky part here is the network setup, so that your developer machine is in the same network as the containers, which then allows direct access to and from the containers.
This setup also allows the usage of LXC on developer machines running OSX or Windows, which normally doesn’t support LXC.
The developer machine and all LXC containers are connected to
an ethernet bridge
lxcbr1 inside the base box. This interface acts like a
normal ethernet switch and is the key component of this solution.
. ├── Vagrantfile └── deployment └── Vagrantfile
Vagrantfile in the project root handles the base box and mounts the
deployment/ into the box. Inside the base box, the other
Vagrantfile is used for setting up the LXC containers.
The Base Box
VAGRANTFILE_API_VERSION = "2" Vagrant.configure(VAGRANTFILE_API_VERSION) do |config| config.vm.box = "ubuntu/trusty64" config.vm.hostname = "devbox" config.vm.synced_folder "deployment/", "/deployment" config.vm.network "private_network", ip: "192.168.42.10", auto_config: false config.vm.provider "virtualbox" do |vb| vb.customize ["modifyvm", :id, "--memory", "1024"] vb.customize ["modifyvm", :id, "--nicpromisc2", "allow-all"] end config.vm.provision :shell, :inline => BOX_SETUP_SCRIPT, privileged: false config.vm.provision :shell, :inline => VAGRANT_STARTUP_SCRIPT, run: "always", privileged: false end
Pay attention to the
--nicpromisc2 argument, which tells VirtualBox to put
the second interface into promiscuous mode, so that all traffic reaching the
interface will be transfered. Without this flag, no direct communication
between the developer machine and any LXC-container would be successful.
BOX_SETUP_SCRIPT = <<END #!/bin/bash sudo apt-get update sudo apt-get install -y lxc lxc-templates cgroup-lite redir bridge-utils cat << EOF > /tmp/lxcbr1.cfg auto eth1 iface eth1 inet manual up ifconfig eth1 promisc up down ifconfig eth1 promisc down auto lxcbr1 iface lxcbr1 inet static address 192.168.42.10 broadcast 192.168.42.255 netmask 255.255.255.0 bridge_ports eth1 bridge_stp off bridge_waitport 0 bridge_fd 0 EOF sudo mv /tmp/lxcbr1.cfg /etc/network/interfaces.d/lxcbr1.cfg sudo service networking restart sudo ifup lxcbr1 cd /tmp wget -q https://dl.bintray.com/mitchellh/vagrant/vagrant_1.6.5_x86_64.deb sudo dpkg -i vagrant_1.6.5_x86_64.deb vagrant plugin install vagrant-lxc END
This script is used once for setting up the base box by installing
vagrant-lxc, all dependencies and also configures the
VAGRANT_STARTUP_SCRIPT = <<END #!/bin/bash cd /deployment vagrant up END
This provisioner script is executed every time
vagrant starts the base
box and it boots the LXC containers.
The Container Setup
VAGRANTFILE_API_VERSION = "2" BASE_IMAGE = "fgrehm/precise64-lxc" Vagrant.configure(VAGRANTFILE_API_VERSION) do |config| config.vm.define :lxc1 do |box| box.vm.hostname = "lxc1" box.vm.box = BASE_IMAGE box.vm.provider :lxc do |provider| provider.customize "network.type", "veth" provider.customize "network.flags", "up" provider.customize "network.link", "lxcbr1" provider.customize "network.ipv4", "192.168.42.2/24" end end config.vm.define :lxc2 do |box| box.vm.hostname = "lxc2" box.vm.box = BASE_IMAGE box.vm.provider :lxc do |provider| provider.customize "network.type", "veth" provider.customize "network.flags", "up" provider.customize "network.link", "lxcbr1" provider.customize "network.ipv4", "192.168.42.3/24" end end # .... end
Vagrantfile handles the LXC containers and is the place where you build
your infrastructure with LXC as a provider.
# from inside your project vagrant up # this should be successful ping 192.168.42.2