When connecting some storage systems to vSphere, vSphere sometime doesn’t select the most optimal multipathing policy.
I came across the above problem when adding an EMC VPLEX storage system to a large vSphere 4.1 installation.
vSphere selected the Fixed path multipathing policy. In smaller environments this isn’t normally a problem , but when a Fixed policy is selected in a large environment with multiple datastores. vSphere selects the first path for each datastore, this results in almost all datastores uses the same path. The result is that the path will be overloaded.
EMC’s best practices for EMC VMAX/VPLEX is to use Fixed policy with static load balancing or to use EMC PowerPath.
In our environment Fixed or PowerPath wasn’t an option as we have multiple hosts and datastores, so in close communication with EMC we chose to change the multipathing policy to Round Robin..
EMC has once now informed us that RR is only an option (not the best) if PowerPath or Fixed is not an option.
Changing the policy for a few LUN’s won’t take long, if you only have a few hosts and a few datastores, but if you have multiple hosts with multiple datastores, the task is massive.
Using Powershell and EsxCli the task is very easy and extremely fast.
Thanks to Arnim van Lieshout (http://www.van-lieshout.com/2011/01/esxcli-powercli/) for the basic script.
The below script illustrates how to set RoundRobin and not Fixed, this will be updated ASAP.
Until then take a look at this post from LucD http://communities.vmware.com/message/1774139
<pre>function FuncMail { param($To, $From, $Subject, $Body, $smtpServer) $msg = new-object Net.Mail.MailMessage $smtp = new-object Net.Mail.SmtpClient($smtpServer) $msg.From = $From $msg.To.Add($To) $msg.Subject = $Subject $msg.IsBodyHtml = 1 $msg.Body = $Body $smtp.Send($msg) } #load Vmware Module if ((Get-PSSnapin | Where-Object { $_.Name -eq "VMware.VimAutomation.Core" }) -eq $null) { Add-PSSnapin VMware.VimAutomation.Core } # ----------- Variables --------------- #vCenter $vcServer = "vcenterserver" $vCenterUser = "vcenterusername" $vCenterPWD = "vcenterpassword" $DC = "*" # Use * for all DC else replace * with datacenter name $cluster = "*" # Use * for all Clusters in DC else replace * with cluster name # LUN settings $LUNType = "EMC Fibre Channel Disk*" $psp = "VMW_PSP_RR" $satp = "VMW_SATP_INV" $iops = 10 # ESX $esxUser = "esxuser-root" $esxPWD = "esxuserpassword" # Email $strEmailTo = "to@mail.com" $strEmailFrom = "from@mail.com" $strEmailSubject = "PSP info '$vcServer' " $strEmailSMTP = "smtpserver.com" # HTML $strHeadHTML = "<STYLE TYPE='text/css'>" $strHeadHTML += "TABLE{border-width: 1px;border-style: solid;border-color: black;border-collapse: collapse;}" $strHeadHTML += "TH{border-width: 1px;padding: 10px;border-style: solid;border-color: black; background-color:thistle}" $strHeadHTML += "TD{border-width: 0px;padding: 0px;padding-right: 5px;padding-left: 5px;border-style: solid;border-color: black}" $strHeadHTML += "</STYLE>" $strBodyHTMLStart = "<H3> Setting PSP multipath & IOPS : " $strBodyHTMLStart += Get-Date -Format g $strBodyHTMLStart += "</H3>" $strBodyHTMLStart += "<TABLE><TR> <TH>vCenter</TH> <TH>Datacenter</TH> <TH>Cluster</TH> <TH>Host</TH> <TH>Device</TH> <TH>SATP</TH> <TH>Old PSP</TH> <TH>New PSP</TH> <TH>IOPS</TH> <TH>Old Paths</TH> <TH>New Paths</TH></TR>" $strBodyHTMLinfoStart = "<H3> PSP multipath & IOPS not set : " $strBodyHTMLinfoStart += Get-Date -Format g $strBodyHTMLinfoStart += "</H3>" $strBodyHTMLinfoStart += "<TABLE><TR> <TH>vCenter</TH> <TH>Datacenter</TH> <TH>Cluster</TH> <TH>Host</TH> <TH>Device</TH> <TH>SATP</TH> <TH>PSP</TH> <TH>Paths</TH></TR>" # ---------- Logic (Don't Change) -------------------- $strBODYHTML = "" $strBodyExcludeHTML = "" #Connect to vCenter Connect-VIServer $vcServer -User $vCenterUser -Password $vCenterPWD| Out-Null #Get Datacenter Clusters $arrDC = Get-Datacenter -Name $DC | Sort Name if($arrDC){ foreach($objDC in $arrDC){ #Get Cluster list $arrCluster = Get-Cluster -Location $objDC -Name $cluster | Sort name if($arrCluster){ foreach($objCluster in $arrCluster){ #Connect to ESX hosts in cluster foreach ($esx in Get-VMHost -Location $objCluster | Sort Name) { Connect-VIServer $esx -User $esxUser -Password $esxPWD | Out-Null #Retrieve the esxcli instances and loop through them foreach($esxcli in Get-EsxCli -Server $esx.name) { #Write-Host $esx.Name -BackgroundColor Red # Change PSP for EMC VPLEX/VMAX devices $arrDevice = $esxCli.nmp.device.list() | where {$_.PathSelectionPolicy -ne $psp -and $_.DeviceDisplayName -like $LUNType} if($arrDevice){ foreach($myDevice in $arrDevice){ #Write-Host "Updating $($myDevice.Device)" -ForegroundColor green $esxCli.nmp.device.setpolicy($null, $myDevice.Device, $psp) $esxcli.nmp.roundrobin.setconfig(0,$myDevice.device,[long]$iops,"iops",$false) $newPSP = $esxCli.nmp.device.list($myDevice.device) $newIOPS = $esxcli.nmp.roundrobin.getconfig($myDevice.device) $strBODYHTML += "<TR> <TD>$($vcServer)</TD> <TD>$($objDC.Name)</TD> <TD>$($objCluster.Name)</TD> <TD>$($esx.Name)</TD> <TD>$($myDevice.Device)</TD> <TD>$($myDevice.StorageArrayType)</TD> <TD>$($myDevice.PathSelectionPolicy)</TD> <TD>$($($newPSP[0]).PathSelectionPolicy)</TD> <TD>$($newIOPS.IOOperationLimit)</TD> <TD>$($myDevice.WorkingPaths)</TD> <TD>$($($newPSP[0]).WorkingPaths)</TD></TR>" } } # Changes not set on $arrInfoDevice = $esxCli.nmp.device.list() | where {$_.PathSelectionPolicy -ne $psp -and $_.DeviceDisplayName -like $LUNType} if($arrInfoDevice){ foreach($myInfoDevice in $arrInfoDevice){ $strBODYHTMLinfo += "<TR> <TD>$($vcServer)</TD> <TD>$($objDC.Name)</TD> <TD>$($objCluster.Name)</TD> <TD>$($esx.Name)</TD> <TD>$($myInfoDevice.Device)</TD> <TD>$($myInfoDevice.StorageArrayType)</TD> <TD>$($myInfoDevice.PathSelectionPolicy)</TD> <TD>$($myInfoDevice.WorkingPaths)</TD></TR>" } } #Change the default PSP for my SATP $esxcli.nmp.satp.setdefaultpsp($psp,$satp) | Out-Null } Disconnect-VIServer $esx.name -Confirm:$false } } } } } #Disconnect from vCenter Disconnect-VIServer $vcServer -Confirm:$false | Out-Null $strBodyHTMLEnd = "</TABLE>" $strBodyHTMLinfoEnd = "</TABLE>" # Collect the HTML $strHTML = "<HTML><HEAD>" $strHTML += $strHeadHTML $strHTML += "</HEAD><BODY>" $strHTML += $strBodyHTMLStart $strHTML += $strBODYHTML $strHTML += $strBodyHTMLEnd $strHTML += $strBodyHTMLinfoStart $strHTML += $strBODYHTMLinfo $strHTML += $strBodyHTMLinfoEnd $strHTML += "</BODY></HTML>" # Email the collected data FuncMail -To $strEmailTo -From $strEmailFrom -Subject $strEmailSubject -Body $strHTML -smtpServer $strEmailSMTP
Script explained
Line 20-24:
Add you vCenter server name and the logon credentionls.
Line 27-30:
LUNTYPE, is the storage type you want to change PSP and SATP for (you can set this to *, if you want you set the PSP and SATP for all datastores.
PSP, is the multipathing policy you want to change to.
SATP, is the storage array type you want to change the default to.
IOPS, is the number of IO’s to be send, before switching to the next path. vSphere default is 1000, EMC recommends 1, but I found 10 to work for me.
Stephen Foskett has explained what is PSP and SATP in plain English http://blog.fosketts.net/2011/06/06/vmware-esx-vsphere-satp-psp-support-matrix/.
Line 33-34:
Change the user to a user with root privileges.
Change the password to correspond with the user.
Line 37-40:
Change the info to receive an report of what has been changed.
Line 92-93:
Sets the PSP and IOPS for each datastore.
Line 95-96:
Retrieves the new settings (for verification) for each datastore.
Line 101-107:
Generates a list of datastores, where the PSP setting wasn’t able to be set.
Run the script again to set them.
Line 111:
Sets the hosts default datastore PSP and SATP.
If your hosts connect to multiple different storage systems, I wouldn’t recommend setting this, but it’s to you.
If you want to see an output on the screen of the progress of the script, uncomment the lines 85, 91
Download the full RoundRobin script here.
An exampel of what a report can look like:
Post is updated after input from Josh Coen, www.valcolabs.com.
Hi Anders,
Good post. Not only does it automate what can be a rather length operation, but it’s a good example of using PowerCLI and ESXCLI in tandum. Well done.
One thing I’d like to point out, you state:
“EMC’s best practices for EMC VPLEX is to use EMC’s own PowerPath or to change the multipathing policy for each datastore (on each host) to Round Robin”
It’s true that the preferred method for multipathing with VPLEX in your environment is to use PowerPath V/E (i’m sure they prefer you use PowerPath for all their gear), but you wouldn’t want to change the native PSP of Fixed to Round Robin. Here is an excerpt from page 26 of EMC’s best practices for EMC VPLEX
“…Therefore, for VMware ESX version 4.x connected to EMC VPLEX, EMC recommends the use of the Fixed policy with static load balancing by changing the preferred path. In addition, the changes to the preferred path should be performed on all of the ESX hosts accessing the VPLEX devices.”
This makes sense when you think about how cache coherency works within VPLEX. If you are doing Round Robin you may be accessing data that’s mapped to one director and then all of a sudden your path changes and your accessing a different director. This could cause disruption if an operation is ongoing and blocks of data have to be remapped in the middle. The same recommendation of choosing a Fixed PSP exists with vSphere 5 as well.
-Josh
Hi Josh,
Thanks for you comment.
I agree, that PowerPath is wat EMC wants you to use with all their storage products, but the performance benefit is only up to about 11%.
So is the cost of PowerPath worth 11%? That is up to each to decide.
I also read that the EMC states that their BP is Fixed, and I state RR, but this is as you say in our environment. So I’ll update the post with this.
RR was chosen in close communication with EMC, to solve our multipathing problem.
So the conclusion must be that you shall go with EMC’s BP, and use Fixed or PowerPath.
And if that isn’t an option due to the size of your environment, then contact EMC for clarification.
BR
A. Mikkelsen
Hi Anders,
Perhaps a usefull addon for those of us that use EMC hardware is the EMC VSI Path Management plugin for vCenter that is part of the Virtual Storage Integrator set from EMC. The complete pluginset is freely available for EMC Customers.
The plugin can manage either the Native VMware NMP or EMC PowerPath V/E and allows the pathing configuration to be set on either a host or clusterlevel. Also seperation can be made between all EMC devices or just Symmetrix/CLARiiON/VPLEX/VNX.
Hi Michel,
Thanks I don’t know that.
I’ll take it for a spin ASAP.
A. Mikkelsen
Hi Anders,
I’d be very interested to know what challenges you faced, as well as the pros vs. cons for Round Robin when compared to Fixed as it relates to VPLEX cache coherency.
-Josh
Hi Josh,
Out challenge was that all most all datastores were using the same path to the storage and therefore the path was maxed to 100% and the other were idle.
This meant that we had high latency on read/write for datastores.
RR meant that we were able to spread the I/O load for each LUN across all paths. So you can say that RR was choosen because of ease of configuration.
I agree that the cache coherency is a problem with this, and therefore you can argue what the IOPS should be. VMware’s default is 1000 and we have set it to 10 but some people say that it should be 1.
So what is the correct setting for VMAX, I don’t know.
A. Mikkelsen
Hi Josh,
EMC has come back and their new recommendation is as you rightly stated, PowerPath or Fixed path with manual load balance.
I’ll update the post and the scripts ASAP.
A. Mikkelsen