When it comes to virtual environment, VMware vSphere networking can be one of the most critical components. After all, how can your ESXi hosts and virtual machines communicate? Since the virtual network is key to how everything else operates, we need to have a good understanding of ESXi networking works.
We are going to take a basic look at ESXi hosts, and their network communication needs. Then we are going to break it down into the different components of VMware networking, such as:
- Overview of virtual switch types in VMware vSphere
- Picking the right virtual switch type for your requirements
- Configuring virtual switches
- NIC Teaming Policy
- Virtual Machine port groups
- VMkernel ports and port groups
- VMware virtual switch security
- VMware virtual switch security settings
- Private VLANs
Anatomy of ESXi Networking
When it comes to our ESXi hosts and their networking communication, the first thing we need to take a look at is what they need to do. They need to be able to communicate in several different ways:
- Communicate with each other (think vMotion)
- Communicate with vCenter (think HA)
- Communicate with external resources they depend on, such as a storage array, or Active Directory
- Allow their resident virtual machines to communicate with other virtual machines and external resources they may depend on
Because of this, there are a few different components of ESXi networking in VMware vSphere that must be configured in our environment.
- Virtual Switches on the ESXi hosts
- Port Groups on the Virtual Switches
Within each of these categories there are a number of things to consider. Let’s start with the switches.
Virtual Switches on ESXi Hosts
Think of the virtual switch on the ESXi host as any other switch you may have encountered before. The virtual switch in ESXi is the way to provide connectivity, redundancy, and load balancing to our ESXi hosts when it comes to the network layer.
There are two types of switches, although you may hear them referred to a few different names each.
- vSphere Standard Switch
- vSphere Distributed Switch (Requires vSphere Enterprise Plus licencing)
Let’s take a look at each virtual switch and its characteristics.
vSphere Standard Switch / Standard vSwitch / vSwitch
This is your basic virtual switch, also known as a vSwitch. It was the first switch type available in VMware vSphere. There is nothing wrong with using a standard vSwitch, although they can be more difficult to manage than vSphere Distributed Switches.
The standard vSwitch resides on the ESXi host, and must be configured individually on each host. There are a couple ways to make this process easier, such as using PowerCLI or Host Profiles.
Network switches are connected to network cards in ESXi hosts, and the NICs in ESXi hosts are connected to the virtual switch in ESXi. That’s it!
When we talk about the NIC of an ESXi host, we call them a vmnics, since that is what they are called in ESXi. vmincs always start at 0, so your first physical NIC port will correspond to vmnic0.
You may have multiple vSwitches on your ESXi host, depending on how you design your virtual network.
vSphere Distributed Switch / Distributed Virtual Switch / dvSwitch
On the physical side of things, there is not much of a difference between a standard virtual switch and a distributed virtual switch. We still connect the network cards in our ESXi hosts to network switches in a redundant manner. What does change is the virtual layer, and the switch within VMware vSphere itself.
This is a basic image of a distributed virtual switch.
The switch configuration itself lives on vCenter server instead of the ESXi host, as you can see. Of course, this could be a problem if something happened to vCenter, right?
Each ESXi host has a copy of the distributed virtual switch configuration on it, as you can see with the bright blue switches on each host. This way, if something happens to vCenter our virtual machines will not be impacted.
I remember when distributed virtual switches first came out in vSphere 4, it sort of broke everyone brains, since it was such a massive shift in the way we were used to doing things, so let’s break down our vSphere distributed switch again:
As you can see, each vmnic is connected to a corresponding dvUplink port on the distributed virtual switch. The distributed virtual switch is then managed through vCenter server. The great part about this is that the distributed virtual switch configuration is identical on each and every ESXi host by design.
Similar to standard vSwitches, distributed vSwitches can be can be managed using the vSphere Client, PowerCLI, or Host Profiles. The difference here is that you are configuring the single distributed virtual switch on vCenter sever opposed to individual virtual switches on ESXi hosts.
Which VMware Virtual Switch Is Right for My Environment?
That’s a great question, and unfortunately there is no single answer. Whether you elect to use a standard virtual switch or a distributed virtual switch will depend on the unique requirements of your environment.
When making your choice, it is important to keep the infrastructure design qualities in mind as you determine what meets your requirements. They are:
- Availability
- Manageability
- Performance
- Recoverability
- Security
It may also be helpful to rank these qualities by importance for your project. Be sure to check this post for more information to help you get in the right mind set for Thinking About VMware vSphere Networking Design Choices.
Be sure to extensively test your VMware vSphere networking configuration before declaring your VMware vSphere environment is production ready.
Keep in mind that the vSphere Distributed Switch does require VMware Enterprise Plus licensing.
NIC Teaming Policy in VMware
Another huge component of VMware networking is NIC teaming. NIC teaming is setting up our virtual switches in a way to facilitate load balancing and protect against component failure.
For example, if I had a switch with two physical NICs connected to it, in an ideal world I would want to make sure that traffic flowed across both physical NICs, and if one of the NICs or the upstream switch failed, my switch would remain operational.
There are a number of ways to accomplish this in VMware vSphere Networking. Luckily, I have already written detailed guides on how NIC teaming works in VMware:
- Introduction to NIC Teaming in VMware vSphere Networking – This article reviews why we use NIC teaming, and goes in more detail about what the different NIC teaming mechanisms are.
- The Simple Guide to NIC Teaming in VMware vSphere – This article talks about the simplest NIC teaming methods to implement: Route Based on Originating Virtual Port, Route Based on Physical NIC Load (Load Based Teaming), and Explicit Failover Order
- The Advanced Guide to NIC Teaming in VMware vSphere – This article takes a look at the more advanced load balancing options, Route based on IP hash and route based on source MAC hash.
These discuss VMware NIC teaming in further depth, and examine all of the different NIC teaming options available to us with our vSphere Standard and vSphere Distributed virtual switches.
Now that we have configured our switches themselves, there are a couple more important things we need to think about.
Configuring Port Groups on Virtual Switches
When we connect to our virtual switches, we connect to something called a Port Group. A port group is what it sounds like, a group of virtual ports on our virtual switch.
There are a couple of different types of port groups:
- Virtual Machine Port Groups
- VMkernel Port Groups
Virtual Machine port groups are how we connect our virtual machines. For example, I may have a port group with a single VLAN, multiple VLANs (VLAN trunking), or in the case of a distributed virtual switch, Private VLANs.
As for VMkernel port groups, you may be asking…
What is a VMkernel port?
Great question! A VMkernel port is for non-virtual machine traffic in VMware vSphere. As you can see from this screen shot taken while configuring networking, there are a number of different types of VMkernel ports:
At the absolute minimum, each and every ESXi host will have a VMkernel port for host management. If it is a member of a vSphere cluster, it will also have a VMkernel port for vMotion. If the cluster is a vSAN cluster, there will be a VMkernel port for vSAN.
There will also be VMkernel ports for IP based storage such as iSCSI, NFS, or FCoE, and a VMkernel port for Fault Tolerance if you are taking advantage of that feature.
The big difference between a Virtual Machine port group and a VMkernel port group is the type of traffic it is passing. As you can see, a VMkernel port is passing traffic specific to VMware vSphere. A virtual machine port group is just passing your garden variety virtual machine traffic.
You can read more about the VMkernel system traffic types in the official vSphere Networking documentation.
Each VMkernel port has its own unique IP address. You can also change the default MTU of 1500 on a VMkernel port.
VMware vSwitch Security Settings
Now that we have covered most of the basics of vSphere networking, I want to dig a little deeper into a few areas when it comes to our VMware switch configuration. VMware vSwitch Security Settings are something that are often misunderstood.
Each of these security settings has two options: reject or accept.
By default, these security settings are set to Reject on this distributed virtual switch port group I have just created. Let’s take a look at what each of them really mean.
Promiscuous Mode VMware Security Policy
This one almost sounds exactly like it means. If Promiscuous mode is set to accept, the virtual machine’s network card will accept all frames sent to the active VLANs on that port group.
This is not secure, however it does have its use cases. If you are running a virtual machine that needs to examine all of the packets, like an intrusion detection system, you would set this option to accept.
By default, it is set to reject and should be left that way.
MAC address changes VMware Security Policy
MAC address changes looks at the inbound traffic coming to your virtual machine.
There are a couple of different MAC address when it comes to your virtual machine’s network interfaces.
First, there is the MAC address as specified in the virtual machine’s VMX file. Think of this as if your virtual machine had a physical NIC card, and it was the MAC address of the NIC card as specified by the manufacturer.
There is also the MAC address as specified in the virtual machine, which would be the same as the MAC in the VMX file…unless you have needed to change it for some reason.
In the virtual machine’s properties, there is the ability to change the MAC address. I have changed in in the past to make sure a virtual machine’s MAC address was the same as a physical machine’s MAC address after a P2V, because a software license was tied to a MAC address.
As you can see there are some valid use cases where these MAC addresses may be different, and you must set MAC address changes to accept.
Otherwise, it should be set to reject since a malicious actor could also wreak havoc in your environment with a spoofed MAC address.
Forged transmits VMware Security Policy
Forged transmits also looks at the MAC addresses of your virtual machines, however is operating on outgoing traffic.
It will drop frames if these two MAC addresses do not match, similar to MAC address changes.
Once again, there are valid use cases to set forged transmits to accept, like nested ESXi. In this case, we are going to be sending all sorts of crazy packets! Chris Wahl, who literally wrote the book on VMware networking has a great blog post that explains forged transmits and nested ESXi.
As you can see, forged transmits and MAC address changes are closely related.
Private VLANs in VMware
When we talked about port groups, we also talked about Private VLANs.
Private VLANS are a way to get even more granular than segregating traffic at the VLAN level, which is why I am bringing it up once we have started our security discussion.
Private VLANs are exclusive to distributed virtual switches, and the physical switch your ESXi host is connected to must also support private VLANs.
When we use a PVLAN, the VLAN we are using gets broken up in a few ways:
- Primary PVLAN, which is promiscuous.
- Secondary PVLANs, which have two types, community and isolated.
Let’s look at this illustration. The Primary PVLAN is green, and the secondary PVLANs are blue.
The virtual machine ports on the on the Isolated PVLAN can only communicate with the promiscuous ports on the primary PVLAN. The virtual machine ports cannot communicate with other ports on the Isolated PVLAN.
The virtual machine ports on the Community PVLAN can communicate with each other, as well as the promiscuous ports on the primary PVLAN.
VMware vSwitch Security, A Summary
When it comes to security policy settings on virtual switches, it is best to leave them set to reject unless you have a specific use case in mind. If you do have a reason that you would need to set promiscuous mode, MAC address changes, or forged transmits to accept, you probably already know that, and can modify the security settings accordingly.
You may also want to take a look at the official VMware documentation on vSwitch security policies.
Private VLANs are an additional way to provide traffic segmentation in your virtual networking environment. Be sure to pay special attention to the secondary PVLAN types to ensure things work the way you would expect them to.
The good news is the PVLAN settings are very easy to change for troubleshooting purposes, and it is described further in the VMware vSphere networking documentation.
VMware vSphere Networking Basics
Understanding VMware vSphere networking basics is important for VMware administrators, Network administrators, IT security analysts, and well, pretty much anyone who needs to touch a VMware vSphere environment. While many traditional networking concepts do apply to VMware vSphere, it is important to become familiar with how things translate at the virtualization layer.
Melissa is an Independent Technology Analyst & Content Creator, focused on IT infrastructure and information security. She is a VMware Certified Design Expert (VCDX-236) and has spent her career focused on the full IT infrastructure stack.