none
Hyper-V - Connection Losses / Mac changed Port RRS feed

  • Domanda

  • Hello,
     
    I am currently trying to set up a Hyper-V Enviroment but am struggling with a lot of Connectivity-Problems.
    I also tried some of the Tipps in other Threads in this forum, but could not find any thing that seems to relate to my problem there (or anywhere for this specific one). So maybe you can help me out?

    The Setup looks like this:

    I've got a Hyper-V Core Installation on a Dell Poweredge 2950 with two Broadcom NetXtreme II Adapters.
    I created a Virtual Network attached to the external Networkadapter One, which is used by my Virtual Machines.
    The second Networkadapter is for Remote Management of the Hyper-V Server and not used by the VMs. No Teaming either.

    Currently there are five Servers up and Running, all attached to the Virtual Network, no VLAN-IDs set:

    Server 1 Sharepoint/IIS Server
    Server2 SQL-Server 2008
    Server 3 Dynamics AX 2009
    Server 4 Dynamics AX 2009 
    Server 5 Server 2008 with local Dynamics AX 2009 Instances & SQL Server.

    The problem is that the sporadic but frequent (meaning: i was not able to see a pattern, but my error log is pretty swamped over the day) connection loss is really annoying, because the connection between the AOS of AX and SQL Server or the Connection between an AX-Client and Server get interruptet a lot.

    So, after some search in the Eventlog what might cause this disturbances i found these on the Hyper-V Server:

    free translation of the german Error Code:

    VMSMP, ID 25: MacAdress "00-15-5D-0A-18-02" was switched from Port "64479F0A-18FC-4A0A-A552-0093D26EC9A  to Port "26B01240-6E4F-4353-880C-DFDB52E6965E"




    These Messages occur about 20times per Hour, with every Machine getting its share, switching them between the two ports. And it seems to correlate with the timings of communication loss between AOS and SQL. So - is this what is causing my problems, or is this something not related? Either way - can anyone explain me what exactly the VMSMP is doing there (and get the little cable-switching troll out of my head, which i picture when reading these messages ;)) and why?

    And of course what the solution for my problem might be.

    Thanks a lot.
    martedì 2 dicembre 2008 13:50

Risposte

  • Hello,

    Just to follow up, and to help if the problem occurs for others:

    We finally solved the problem, it was caused by a faulty or incompatible network switch. The Switch was a Netgear 24port Switch, sadly i don't have a detailed name at the moment. After we switched to a new one everything started to work smoothly and connection losses were gone.

    Thanks goes out to the EMEA GTSC Munich for the final clue that it might be a hardware issue in the network and not a problem of the Hyper-V itself.
    • Contrassegnato come risposta Henning Raabe lunedì 12 gennaio 2009 13:48
    lunedì 12 gennaio 2009 13:48

Tutte le risposte

  •     My guess is that you are simply overloading the virtual switch and/or the NIC driver. That sounds like a pretty heavy load for one NIC/swith port! I would add extra NICs to the host and spread the vms over multiple virtual switches.
    Bill
    martedì 2 dicembre 2008 23:21
  • Are all VMs generating these event logs?  If yes - can you confirm that each virtual machine is using different MAC and switch port IDs?
    Cheers,
    Benjamin Armstrong
    ============================
    Windows Virtualization
    Program Manager

    This posting is provided AS IS with no warranties, and confers no rights. You assume all risk for your use.
    mercoledì 3 dicembre 2008 04:36
  • I agree that it sounds like a heavy load, but actually it should not really be the problem for now, since Server 1 and 5 are not really doing anything than idling, Server 4 has no Users, just the Databaseconnections from the AOS, and Server 3 has only one to two Users Developing on it.
    So i would rule out the Load for the NIC/Switch for the moment. Anyways, to rule it out I shut down all but SQL and AX 2009 Server 3 for today, but the Problem persists.
    Additional NICs are planned though.

    For the MAC and switch port IDs:

    Yes, every VM does generate these entries.
    Every VM has its own MAC, dynamically assigned by Hyper-V. As for the port ID - I have to admit i am not sure here. I cannot influence this one anyways, can i? Or is port ID = VLAN ID? This is an exerpt of the Logs from between 7 and 8am this Morning (no users on the system):


    Die MAC-Adresse "00-15-5D-0A-18-03" wurde von Port "64479F0A-18FC-4A0A-A552-0093D26EC9A0" zu Port "84B29213-7C09-43A6-A1C4-EB26B8037F1A" verschoben.
    Die MAC-Adresse "00-15-5D-0A-18-03" wurde von Port "84B29213-7C09-43A6-A1C4-EB26B8037F1A" zu Port "64479F0A-18FC-4A0A-A552-0093D26EC9A0" verschoben.
    Die MAC-Adresse "00-15-5D-0A-18-05" wurde von Port "64479F0A-18FC-4A0A-A552-0093D26EC9A0" zu Port "3DCC02FB-A14B-4AD2-A5EC-0DE0C61C6EBA" verschoben.
    Die MAC-Adresse "00-15-5D-0A-18-05" wurde von Port "3DCC02FB-A14B-4AD2-A5EC-0DE0C61C6EBA" zu Port "64479F0A-18FC-4A0A-A552-0093D26EC9A0" verschoben.
    Die MAC-Adresse "00-15-5D-0A-18-08" wurde von Port "64479F0A-18FC-4A0A-A552-0093D26EC9A0" zu Port "65793909-546A-4426-BAAB-55CA07204FED" verschoben.
    Die MAC-Adresse "00-15-5D-0A-18-00" wurde von Port "D6A365EC-669E-4B3C-9239-F0254385EC3E" zu Port "64479F0A-18FC-4A0A-A552-0093D26EC9A0" verschoben.
    Die MAC-Adresse "00-15-5D-0A-18-08" wurde von Port "65793909-546A-4426-BAAB-55CA07204FED" zu Port "64479F0A-18FC-4A0A-A552-0093D26EC9A0" verschoben.
    Die MAC-Adresse "00-15-5D-0A-18-00" wurde von Port "64479F0A-18FC-4A0A-A552-0093D26EC9A0" zu Port "D6A365EC-669E-4B3C-9239-F0254385EC3E" verschoben.
    Die MAC-Adresse "00-22-19-83-F4-3D" wurde von Port "64479F0A-18FC-4A0A-A552-0093D26EC9A0" zu Port "8F06A799-C675-45F2-89AE-FE39DE55A057" verschoben.
    Die MAC-Adresse "00-15-5D-0A-18-05" wurde von Port "64479F0A-18FC-4A0A-A552-0093D26EC9A0" zu Port "3DCC02FB-A14B-4AD2-A5EC-0DE0C61C6EBA" verschoben.
    Die MAC-Adresse "00-22-19-83-F4-3D" wurde von Port "8F06A799-C675-45F2-89AE-FE39DE55A057" zu Port "64479F0A-18FC-4A0A-A552-0093D26EC9A0" verschoben.
    Die MAC-Adresse "00-15-5D-0A-18-05" wurde von Port "3DCC02FB-A14B-4AD2-A5EC-0DE0C61C6EBA" zu Port "64479F0A-18FC-4A0A-A552-0093D26EC9A0" verschoben.
    Die MAC-Adresse "00-22-19-83-F4-3D" wurde von Port "64479F0A-18FC-4A0A-A552-0093D26EC9A0" zu Port "8F06A799-C675-45F2-89AE-FE39DE55A057" verschoben.
    Die MAC-Adresse "00-15-5D-0A-18-00" wurde von Port "D6A365EC-669E-4B3C-9239-F0254385EC3E" zu Port "64479F0A-18FC-4A0A-A552-0093D26EC9A0" verschoben.
    Die MAC-Adresse "00-22-19-83-F4-3D" wurde von Port "8F06A799-C675-45F2-89AE-FE39DE55A057" zu Port "64479F0A-18FC-4A0A-A552-0093D26EC9A0" verschoben.
    Die MAC-Adresse "00-15-5D-0A-18-00" wurde von Port "64479F0A-18FC-4A0A-A552-0093D26EC9A0" zu Port "D6A365EC-669E-4B3C-9239-F0254385EC3E" verschoben.
    Die MAC-Adresse "00-22-19-83-F4-3D" wurde von Port "64479F0A-18FC-4A0A-A552-0093D26EC9A0" zu Port "8F06A799-C675-45F2-89AE-FE39DE55A057" verschoben.
    Die MAC-Adresse "00-22-19-83-F4-3D" wurde von Port "8F06A799-C675-45F2-89AE-FE39DE55A057" zu Port "64479F0A-18FC-4A0A-A552-0093D26EC9A0" verschoben.
    Die MAC-Adresse "00-15-5D-0A-18-08" wurde von Port "64479F0A-18FC-4A0A-A552-0093D26EC9A0" zu Port "65793909-546A-4426-BAAB-55CA07204FED" verschoben.
    Die MAC-Adresse "00-15-5D-0A-18-08" wurde von Port "65793909-546A-4426-BAAB-55CA07204FED" zu Port "64479F0A-18FC-4A0A-A552-0093D26EC9A0" verschoben.
    Die MAC-Adresse "00-15-5D-0A-18-00" wurde von Port "D6A365EC-669E-4B3C-9239-F0254385EC3E" zu Port "64479F0A-18FC-4A0A-A552-0093D26EC9A0" verschoben.
    Die MAC-Adresse "00-22-19-83-F4-3D" wurde von Port "64479F0A-18FC-4A0A-A552-0093D26EC9A0" zu Port "8F06A799-C675-45F2-89AE-FE39DE55A057" verschoben.
    Die MAC-Adresse "00-15-5D-0A-18-05" wurde von Port "64479F0A-18FC-4A0A-A552-0093D26EC9A0" zu Port "3DCC02FB-A14B-4AD2-A5EC-0DE0C61C6EBA" verschoben.
    Die MAC-Adresse "00-15-5D-0A-18-05" wurde von Port "3DCC02FB-A14B-4AD2-A5EC-0DE0C61C6EBA" zu Port "64479F0A-18FC-4A0A-A552-0093D26EC9A0" verschoben.

    So - every Server/MAC seems to have its own port ID, but they are all being shifted towards Port 64479F0A-18FC-4A0A-A552-0093D26EC9A0 at one point. And shortly after back. (wurde von Port = from port / zu Port = to Port).

    As posted earlier: This may be normal behaviour (since the messages are only info-messages, and no errors) and does not neccessarily have anything to do with my Problem (where i currently more strongly suspect the BroadCom NIC which other people had Problems with), but i am wondering what is going on there anyways.

    mercoledì 3 dicembre 2008 11:54
  •  

    Hi,

     

    Dynamic may be caused by some Network technology, such as "Network Load Balanced Cluster". Have you setup a "Network Load Balanced Cluster" or was any other Networking optimization used?

     

    I suggest we try the steps below to narrow down the cause of this issue.

     

    1.       Start only one server, save or shut down other servers. If the issue appears again, please help to collect "ipconfig /all >c:\ip.log" for further research.

     

    2.       If all server work fine themselves, please start two server once and test. If the issue appears, please help to collect "ipconfig /all >c:\ip.log" for further research.

     

    3.       Detailed Network packages information is very helpful. Please download and install Microsoft Network Monitor 3.2 to capture Network traffic.

     

    http://www.microsoft.com/DownLoads/details.aspx?FamilyID=f4db40af-1e08-4a21-a26b-ec2f4dc4190d&displaylang=en

     

    Close all other programs and stop all other activities on both servers. Start Network Monitor, click New Capture, click Start button. When the issue appears, click Stop. Select latest log content if the log it too long to send. Save the content as a text file and compress it. . Send the ZIP file to tfwst@microsoft.com for research.

     

    Thanks.

    giovedì 4 dicembre 2008 12:23
    Moderatore
  • Hello,

    Just to follow up, and to help if the problem occurs for others:

    We finally solved the problem, it was caused by a faulty or incompatible network switch. The Switch was a Netgear 24port Switch, sadly i don't have a detailed name at the moment. After we switched to a new one everything started to work smoothly and connection losses were gone.

    Thanks goes out to the EMEA GTSC Munich for the final clue that it might be a hardware issue in the network and not a problem of the Hyper-V itself.
    • Contrassegnato come risposta Henning Raabe lunedì 12 gennaio 2009 13:48
    lunedì 12 gennaio 2009 13:48
  • Any chance you can post the switch model? I am seeing identical issues here, and we have a couple of Netgear 24pt switches as well, so would love to know if they're the same models as yours.
    mercoledì 14 gennaio 2009 02:25
  • I'm having a similar issue. Though I have two sets of Hyper-V hosts managed by the same SCVMM 2008 server. The SCVMM server, and the two sets of hosts are on different switches. The two sets of hosts are all on bl460c blade hardware using HP NC373i adapters connected to Cisco 3020 enclosure switches that are configured the same way, however only my most recently setup hosts are behaving this way. The only VMSMP event 25 that I get from the working hosts is about the virtual switch moving from port to port and not the VM MAC addresses. And like Henning my VMs on the affected hosts drop offline during the time they switch between ports. Here is an example log from one of my affected hosts:

    Information    2/20/2009 8:34:39 AM    Service Control Manager    7036    None    The Windows Modules Installer service entered the running state.
    Information    2/20/2009 8:34:00 AM    VMSMP    25    None    The MAC address '00-1D-D8-B7-1C-02' has moved from port 'SCVMM-F3071215-5046-49D4-911B-EA9600F4C621' to port 'EXTERNAL-956197D6-4DB7-48C7-A9B4'.
    Information    2/20/2009 8:30:21 AM    VMSMP    25    None    The MAC address '00-1D-D8-B7-1C-03' has moved from port 'EXTERNAL-956197D6-4DB7-48C7-A9B4-45EF0AC94AA0' to port 'SCVMM-FC7D2B51-8370-4601-83DA'.
    Information    2/20/2009 8:23:16 AM    VMSMP    25    None    The MAC address '00-1D-D8-B7-1C-06' has moved from port 'EXTERNAL-956197D6-4DB7-48C7-A9B4-45EF0AC94AA0' to port 'SCVMM-55CFF9C2-85D6-4741-BE0C'.
    Information    2/20/2009 8:21:14 AM    VMSMP    25    None    The MAC address '00-1D-D8-B7-1C-06' has moved from port 'SCVMM-55CFF9C2-85D6-4741-BE0C-6BAF2AA7B163' to port 'EXTERNAL-956197D6-4DB7-48C7-A9B4'.
    Information    2/20/2009 8:18:21 AM    VMSMP    25    None    The MAC address '00-1D-D8-B7-1C-03' has moved from port 'SCVMM-FC7D2B51-8370-4601-83DA-5A5C3F526D8B' to port 'EXTERNAL-956197D6-4DB7-48C7-A9B4'.
    Information    2/20/2009 8:03:15 AM    VMSMP    25    None    The MAC address '00-1D-D8-B7-1C-06' has moved from port 'EXTERNAL-956197D6-4DB7-48C7-A9B4-45EF0AC94AA0' to port 'SCVMM-55CFF9C2-85D6-4741-BE0C'.
    Information    2/20/2009 8:00:27 AM    VMSMP    25    None    The MAC address '00-1D-D8-B7-1C-06' has moved from port 'SCVMM-55CFF9C2-85D6-4741-BE0C-6BAF2AA7B163' to port 'EXTERNAL-956197D6-4DB7-48C7-A9B4'.
    Information    2/20/2009 7:48:55 AM    VMSMP    25    None    The MAC address '00-1D-D8-B7-1C-06' has moved from port 'EXTERNAL-956197D6-4DB7-48C7-A9B4-45EF0AC94AA0' to port 'SCVMM-55CFF9C2-85D6-4741-BE0C'.
    Information    2/20/2009 7:45:14 AM    VMSMP    25    None    The MAC address '00-1D-D8-B7-1C-06' has moved from port 'SCVMM-55CFF9C2-85D6-4741-BE0C-6BAF2AA7B163' to port 'EXTERNAL-956197D6-4DB7-48C7-A9B4'.
    Information    2/20/2009 7:30:29 AM    VMSMP    25    None    The MAC address '00-1D-D8-B7-1C-03' has moved from port 'EXTERNAL-956197D6-4DB7-48C7-A9B4-45EF0AC94AA0' to port 'SCVMM-FC7D2B51-8370-4601-83DA'.
    Information    2/20/2009 7:26:21 AM    VMSMP    25    None    The MAC address '00-1D-D8-B7-1C-03' has moved from port 'SCVMM-FC7D2B51-8370-4601-83DA-5A5C3F526D8B' to port 'EXTERNAL-956197D6-4DB7-48C7-A9B4'.
    Information    2/20/2009 6:55:15 AM    VMSMP    25    None    The MAC address '00-1D-D8-B7-1C-06' has moved from port 'EXTERNAL-956197D6-4DB7-48C7-A9B4-45EF0AC94AA0' to port 'SCVMM-55CFF9C2-85D6-4741-BE0C'.
    Information    2/20/2009 6:47:46 AM    VMSMP    25    None    The MAC address '00-1D-D8-B7-1C-06' has moved from port 'SCVMM-55CFF9C2-85D6-4741-BE0C-6BAF2AA7B163' to port 'EXTERNAL-956197D6-4DB7-48C7-A9B4'.
    Information    2/20/2009 6:43:18 AM    VMSMP    25    None    The MAC address '00-1D-D8-B7-1C-06' has moved from port 'EXTERNAL-956197D6-4DB7-48C7-A9B4-45EF0AC94AA0' to port 'SCVMM-55CFF9C2-85D6-4741-BE0C'.


    It does this continually for all 3 VMs on this system. Currently the VMs are only being setup by the end users so they don't have much if any activity on them.
    venerdì 20 febbraio 2009 16:40
  • I was having this identical issue.  Turns out that 2 NICS on 2 different hosts were assigned the same MAC by Hyper-V.  After changing the MAC on one, the issue cleared up.

    Take a look at: http://www.virtualizationadmin.com/articles-tutorials/microsoft-hyper-v-articles/networking/mac-address-pool-duplication-hyper-v.html
    • Proposto come risposta Eric Hazen mercoledì 12 agosto 2009 17:47
    giovedì 9 aprile 2009 15:22
  • I was having this identical issue.  Turns out that 2 NICS on 2 different hosts were assigned the same MAC by Hyper-V.  After changing the MAC on one, the issue cleared up.

    Take a look at: http://www.virtualizationadmin.com/articles-tutorials/microsoft-hyper-v-articles/networking/mac-address-pool-duplication-hyper-v.html

    This worked great! I had two guest 2 sets of computers with the same MAC addresses.
    mercoledì 12 agosto 2009 18:13
  • Hello,

    I've got a similar problem. I'm using Windows 2008 server SP1 with Hyper-v. I've got 2 network card, connected to the same switch. One network card for the server administration, one for the VM network. Recently I've got this event message from VMSMP :

    L’adresse MAC '00-21-5A-EE-54-C2' a été déplacée du port '427A66AC-E610-4DD4-AA12-91B44675ACFF' vers le port 'C9456678-08F6-4AA2-B3A4-0921F0C4607C'.

    The VM still alive and online, but I can't ping the Hyper-v anymore with the administrative network card.
    I check, but I haven't got the same MAC Address on my 2 Hyper-v server.

    Where is my problem ?

    Thanks


    austechnique
    lunedì 21 settembre 2009 14:44
  • Hello

    Someone as another way to proceed ?

    because, I've change my netgear 8 ports Giga to a brand new Cisco 8 ports Giga but it's still the same

    there error only appears on the main hyper-v os :

    The MAC address '02-01-00-00-00-00' has moved from port '415ABE07-1DB4-4A16-A9E9-D81FE41F5480' to port '03D95DDF-FF91-44FC-8D63-C6D55B5651FD'.


    I can't find the mac address on any physical or virtual server NIC's

    Any ideas are welcome

    Thanks

    Regards

    Kevin

    venerdì 15 gennaio 2010 12:44
  • I have a far simpler situation that generates the same error (VMSMP 25) with a single VM running:

    Host: is quadproc 2008 r2 + hyper-v

    Single guest is SBS 2003 R2

    Ignore the guest. It is running and is almost totally idle.  It has a mac address that is completely different from my hardware.

    On the host, when I do a large file copy over my 1G/100Mb ethernet, I get the pair of VMSMP errors where a mac address is shifted between ports and connectivity is temporarily lost.

    Hyper-V virt net mgr has a single virtual network called BindingtoHN with connection type external to h/w GBE controller.

    Network Connections shows two: Local Area Connection enabled to h/w GBE controller and Local Area Connection 3 to MS Virt Net Switch

    Going through the registry, I find that the two ports are the virtual network and the hw GBE controller. The Mac address in question is that of "local area connection 3".

    Does anyone (MSFT?) have an idea about this?

    lunedì 6 settembre 2010 06:15
  • Kevin, have you ever find an answer to your problem? I am facing the same problems now and changing the switch did not help.
    domenica 12 dicembre 2010 08:46
  • Just to add:

    In my case this problem started suddenly. The problem was that someone made an ethernet loop on the network ;)

     

    giovedì 30 giugno 2011 15:00