For some reason my Solaris 10 Virtual Machines on Xenserver failed when the Distributed Virtual Switch Controller was also running. I didn’t really troubleshoot the issue until recently since I could live without cross-server private networks. This no longer being the case I decided to look into it again.

Scroll forward a couple of hours and after losing quite some time on trying various tricks on the VM (disabling NIC checksum offload, lower MTUs etc) to no avail I concluded that it must be a hypervisor issue. Digging into the openvswitch tools revealed something interesting.


[root@xenserver ~]# ovs-vsctl list-ports xapi25
vif42.0
tap47.0
vif47.0
vif6.0


Specifically, for my Linux VMs only a vifX.Y interface was being added to the bridge, while for my Solaris ones both a tapX.Y and a vifX.Y. Clickety-click.


[root@xenserver ~]# ovs-vsctl del-port xapi25 tap47.0


Et voila! Network connectivity to the Solaris VM works like a charm. Now to make this change permanent:


[root@xenserver ~]# diff /etc/xensource/scripts/vif.orig /etc/xensource/scripts/vif
134c134,138
if [[ $dev != tap* ]]; then
> $vsctl --timeout=30 -- --if-exists del-port $dev -- add-port $bridge $dev $vif_details
> else
> echo Skipping command $vsctl --timeout=30 -- --if-exists del-port $dev -- add-port $bridge $dev $vif_details
> fi


I am not really certain of the ugly side-effects that this may have. But it does the trick for me.

Update 2013/03/10: A better workaround is to have the above behavior apply only to Solaris VMs. For example, assuming that these are based on the “Solaris 10 (experimental)” template, the following snippet skips the offending command only for the Solaris VMs:

if [[ $dev != tap* ]]; then
    $vsctl --timeout=30 -- --if-exists del-port $dev -- add-port $bridge $dev $vif_details
else
    xe vm-list dom-id=$DOMID params=name-description | grep 'Solaris 10' 2>&1 >/dev/null || \
        $vsctl --timeout=30 -- --if-exists del-port $dev -- add-port $bridge $dev $vif_details
fi

following commands let you create/destroy symlinks for vhosts and apache modules:

a2dismod  a2dissite  a2enmod    a2ensite

so blahblah blah i add a new site for supercoolblog.net

create my vhost in /etc/apache2/sites-available/supercoolblog.net

a2ensite supercoolblog.net

Symlink is added, restart apache and you got your cool site.

This was a great help.  Basically I had to take about 8 files with the word “stage” in them and replace it with “dev”

for i in ./*stage*;do mv -- "$i" "${i//stage/dev}";done

Bang, done!

Solution: Install Cntlm between Squid and proxy parent.

Cntlm (http://sourceforge.net/projects/cntlm/): “Is an NTLM / NTLMv2 authenticating HTTP/1.1 proxy”.

In front of my clients, in the proxy server, Dansguardian is installed for content filtering.

Then Squid listen only for dansguardian requests and redirects to a proxy parent.

Because I don’t know if its possible to configure Squid to authenticate with the proxy parent using NTLM, instead, I have used cntlm between Squid and the proxy parent. Good idea huh!

The basic configuration will be like:

Client: Shots to Dansguardian.

Proxy: 192.168.5.7
Port: 8080

Dansguandian: Listen on port 8080 and shots to squid.

filterip = 192.168.5.7
filterport = 8080
proxyip = 192.168.5.7
proxyport = 3128

Squid: Listen on port 3128 and shots to cntlm:

http_port 192.168.5.7:3128
cache_peer 192.168.5.7  parent  4040  0  default no-query proxy-only

Cntlm: Listen on port 4040 and shots to Proxy parent with the credentials:

Username    userofdomain
Domain       MYCOMPANY
Password    supersecretpassword
Proxy        10.0.0.10:8080
Listen        192.168.5.7:4040

Daemons: After configuring, restart every daemon:

# /etc/init.d/cntlm restart
# /etc/init.d/squid3 restart
# /etc/init.d/dansguardian restart

Well, for prevent clients to directly jump to cntlm or squid evading the filter, somo iptables rules:

Redirect traffic to port 3128 (squid) to filter on port 8080 (DansGuardian).
# iptables -t nat -A PREROUTING -p tcp -s 0.0.0.0/0 --dport 3128 -j REDIRECT --to 8080;

Redirect traffic to port 4040 (cntlm) to filter on port 8080 (DansGuardian).
# iptables -t nat -A PREROUTING -p tcp -s 0.0.0.0/0 --dport 4040 -j REDIRECT --to 8080;

Ta Dan!

http://blogs.vmware.com/kb/2013/02/troubleshooting-network-teaming-problems-in-esxesxi.html

A quick summary of what I picked up from this:

• If you are using VST or Virtual Switch Tagging, your physical network must be configured with VLAN trunking using 802.1q protocol on the physical switch
• If you are using ‘Route based on originating virtual port ID’ load balancing type, the physical switch should not be configured for link aggregation (802.3ad or etherchannel or Trunks). Link aggregation or bonding should be used only with IP hash load balancing type.
• When troubleshooting VM network issue, remember to identify which port group and VLAN ID is belongs to
• To determine this which NIC a VM, connect to the host using SSH or via the Local Tech Support Mode console and use a tool called esxtop. From within esxtop, we simply hit ‘n’ for the networking view and it will show you which NIC the VM is using.