Hi, I wonder if anyone can help?
We are looking to setup a VM for nested virtualisation, but have run into some problems:
This is our environment:
2 hosts with 8 NICs each: NIC0 & 7 used for vMotion and Management network; NIC1 & 6 used for data and NIC 2,3,4 and 5 are iScasi.
This is the problem as described by one of the engineers that is setting up the VMs within the nested environment:
The problem we see is with VMWare networking when the vSwitch has more than one physical NIC attached to it. In VMWare terms the VMWare virtual machine's network connection is put in promiscuous mode (because we are running nested virtual machines). That network connection is connected to a vSwitch and if that vSwitch has more than one physical NIC attached to it, in certain circumstances we see packets that we transmit to the vSwitch for onward transmission being reflected back to us which corrupts our internal bridging tables.
Our internal network has a bridge which connects to the NIC supplied by VMWare and a number of virtual NICs that we create with different MAC addresses, thus our internal bridge looks something like:
Port 1: VMWare NIC (External Connection)
Port 2: Internal NIC A (Provided to nested virtual machine A)
Port 3: Internal NIC B (Provided to nested virtual machine B)
The Internal NICs are provided to the nested virtual machines that we create, and enabling promiscuous mode should allow this to work correctly.
If I choose an external system, for example your MIS server and ping it so that packets originate from the VMWare NIC, thus having it's MAC address everything works correction. However if I issue the pings from virtual machine A so that they originate from Internal NIC A and have it's MAC address, shortly after the packet is transmitted out on the VMWare NIC, we receive a copy of the packet from the VMWare NIC which shouldn't happen.
Our internal bridge is a learning bridge and so looks at the source address of this reflected packet, and decides that Internal NIC A must have been moved from Port 2 to Port 1 and thereafter sends all packets destined for Internal NIC A to Port 1, thus virtual machine A no longer receives any packets. This is corrected when virtual machine A transmits another packet, but if that packet causes VMWare to reflect it back again, the bridging table is broken and again virtual machine loses network connectivity.
I appreciate that this is a rather unique setup. I thought the solution was simple: create another vSwitch that connects to a single physical NIC, but I soon realised that I don't have any physical NICs available. I was then wondering if it would be possible to map a port group to only one of the physical NICs on the vSwitch, but that's either not possible or I don't know how to do it.
Does anyone have any suggestions?