locked
System.Net.HttpWebRequest fails to logon to a website via com-interop in Windows 10 1809 RRS feed

  • Question

  • This problem is described in more detail here on stack overflow, below is a summary.  The question is, how can I solve this, or is this a bug in Windows and/or the .Net framework?

    I have a small piece of C# code which logs in to a website, and

    • Works when called on computer A from Excel 2010 VBA via com-interop
    • Works when called on computer B from a C# console application, but
    • Fails when called on computer B from Excel 2010 VBA via com-interop

    The main difference between computer A and computer B is that computer A has windows 10 version 1803, whereas computer B has windows 10 version 1809. Both computers have Studio 2017, and in all cases the target .Net Framework is 4.6.2.  In all cases, a little JSON object is returned, where the JSON object has a field called "loginStatus". When it works, "loginStatus"="SUCCESS", but when it fails "loginStatus"="CERT_AUTH_REQUIRED".

    I looked at the settings in System.Net.ServicePointManager and they're identical in all cases.  I tried using the .Net Framework 4.7.1 instead of 4.6.2 but the results were the same.  I used Fiddler to review the structure of the https calls that are being made to the website. The two that work look identical, and the one that fails looks slightly different:

    Calls that work OK

    • TLS extension ec_point_formats = uncompressed [0x0]
    • TLS extension encrypt_then_mac (RFC7366) not specified
    • TLS extension renegotiation_info = 0
    • Cipher TLS_EMPTY_RENEGOTIATION_INFO_SCSV not specified

    Failed call

    • TLS extension ec_point_formats = uncompressed [0x0], ansiX962_compressed_prime [0x1], ansiX962_compressed_char2 [0x2]
    • TLS extension encrypt_then_mac (RFC7366) = empty
    • TLS extension renegotiation_info not specified
    • Cipher TLS_EMPTY_RENEGOTIATION_INFO_SCSV specified

    I have no idea how to solve this, please help.

    Friday, February 8, 2019 6:36 PM

Answers

  • AcquireCredentialsHandle exists in "success" case at line [35268] after the security context is initialized, and SSL channel has been established, therefore can proceed to authentication.

    Assuming you've not left out important detail in the logs, I think in the second case it's trying to do without client certs. (Look again for "We have user-provided certificates" in the failed logs)

    If that line does not exists, I think something makes the HttpWebRequest to create connection without attaching the certs in VBA case.

    EDIT: Simon @ StackOverflow might have nailed your case. The web scanner component acts as HTTP proxy on your outgoing traffic and may have influrence on the communication. Try disable it while you test again and see if this will fix the problem.
    • Edited by cheong00 Wednesday, February 13, 2019 10:17 AM
    • Marked as answer by Stochastically Wednesday, February 13, 2019 3:51 PM
    Wednesday, February 13, 2019 10:12 AM

All replies

  • Hi Stochastically,

    Since your question is more related to Excel and VBA, you could post a new thread in Microsoft Office for development forum for suitable support.

    The CLR Forum discuss and ask questions about .NET Framework Base Classes (BCL) such as Collections, I/O, Regigistry, Globalization, Reflection. Also discuss all the other Microsoft libraries that are built on or extend the .NET Framework, including Managed Extensibility Framework (MEF), Charting Controls, CardSpace, Windows Identity Foundation (WIF), Point of Sale (POS), Transactions.

    Best Regards,

    Wendy


    MSDN Community Support
    Please remember to click "Mark as Answer" the responses that resolved your issue, and to click "Unmark as Answer" if not. This can be beneficial to other community members reading this thread. If you have any compliments or complaints to MSDN Support, feel free to contact MSDNFSF@microsoft.com.

    Monday, February 11, 2019 7:21 AM
  • Hi Wendy,

    Thank you for the response.  I am not sure that the forum you suggest is the right one, because it seems to me that this relates to the behaviour of low level Windows 10 and .Net DLLs.  Nonetheless, I have posted the question there.

    Is there a forum that relates to the Windows 10 and .Net DLLs that implement web security, namely protocols such as TLS?  The heart of my problem is that those low level DLLs choose a different implementation in the two cases that work versus the case that doesn't work.  My best guess is that I need a registry setting (or to delete a registry setting) to fix this.

    Thanks.

    Monday, February 11, 2019 7:48 AM
  • Seems you've hit by this problem.

    Try explicitly set the ServicePointManager.SecurityProtocol  and see if it does any help.

    • Edited by cheong00 Tuesday, February 12, 2019 3:09 AM
    Tuesday, February 12, 2019 2:46 AM
  • Thanks for the suggestion, cheong00, but in fact I've already tried setting ServicePointManager.SecurityProtocol  explicitly, and also checked that everything is x64.  Whether I set ServicePointManager.SecurityProtocol explicitly or not, when I use Fiddler to look at the internet traffic being generated, all cases were use security protocol "Version: 3.3 (TLS/1.2)".

    Thanks too for the link to Avoid sending TLS_EMPTY_RENEGOTIATION_INFO_SCSV cipher ... .  But I'm not sure it helps.  On the same computer, the correct internet traffic is generated when the code is called directly, versus the incorrect internet traffic (that uses TLS_EMPTY_RENEGOTIATION_INFO_SCSV) when called from Excel via com-interop.  As I said to Wendy yesterday,  the heart of my problem is that low level windows/.Net DLLs choose to generate different internet traffic in the different cases.  My best guess is that I need a registry setting (or to delete a registry setting) to fix this.

    Please let me know if you have any other ideas, or whether you can recommend a better forum.

    Thanks, Paul

    Tuesday, February 12, 2019 7:46 AM
  • Since I'm unable to setup test for your issue, can you try to add try..catch block to tell us the exact line exception is thrown?

    Also, check ComputerB and see if there exists another code running in Excel with the same certificate. .NET will reuse connection with client certificate specified (something like using certificate.GetHashCode().ToString() as key to try get existing connection from the pool)

    Tuesday, February 12, 2019 8:46 AM
  •  The error doesn't manifest itself by an exception being thrown, cheong00.  In all cases the login request returns a little JSON object, where the JSON object has a field called "loginStatus". When it works, "loginStatus"="SUCCESS", but when it fails "loginStatus"="CERT_AUTH_REQUIRED".  Also, the version of Excel that I'm using for this has all add-ins disabled. so there can't anything else using  the same certificate.  I've deliberately cut everything down to the bare bones here, to keep everything as simple as possible and hence just focus on the core issue.

    Today I did make some progress, because someone on stackoverflow.com told me how to enable debugging for http calls in .Net (see How do I see the raw HTTP request that the HttpWebRequest class sends for what I did).  So I ran the diagnostics on computer B for the two cases (calling the C# code directly from a console EXE, and then calling from Excel 2010 via com-interop).  The "System.Net Information" output for the code which calls directly from a console EXE was as follows:

        System.Net Information: 0 : [35268] Current OS installation type is 'Client'.
        System.Net Information: 0 : [35268] RAS supported: True
        System.Net Information: 0 : [35268] Associating HttpWebRequest#21454193 with ServicePoint#34640832
        System.Net Information: 0 : [35268] Associating Connection#43332040 with HttpWebRequest#21454193
        System.Net Information: 0 : [35268] Connection#43332040 - Created connection from XXX.XXX.XXX.XXX:53002 to YYY.YYY.YYY.YYY:443.
        System.Net Information: 0 : [35268] TlsStream#54444047::.ctor(host=<TargetWebSite>, #certs=1, checkCertificateRevocationList=False, sslProtocols=Tls12)
        System.Net Information: 0 : [35268] Associating HttpWebRequest#21454193 with ConnectStream#20234383
        System.Net Information: 0 : [35268] HttpWebRequest#21454193 - Request: POST /api/certlogin HTTP/1.1
        System.Net Information: 0 : [35268] ConnectStream#20234383 - Sending headers
        System.Net Information: 0 : [35268] SecureChannel#47891719::.ctor(hostname=<TargetWebSite>, #clientCertificates=1, encryptionPolicy=RequireEncryption)
        System.Net Information: 0 : [35268] Enumerating security packages:
        System.Net Information: 0 : [35268]     Negotiate
        System.Net Information: 0 : [35268]     NegoExtender
        System.Net Information: 0 : [35268]     Kerberos
        System.Net Information: 0 : [35268]     NTLM
        System.Net Information: 0 : [35268]     TSSSP
        System.Net Information: 0 : [35268]     pku2u
        System.Net Information: 0 : [35268]     CloudAP
        System.Net Information: 0 : [35268]     WDigest
        System.Net Information: 0 : [35268]     Schannel
        System.Net Information: 0 : [35268]     Microsoft Unified Security Protocol Provider
        System.Net Information: 0 : [35268]     Default TLS SSP
        System.Net Information: 0 : [35268]     CREDSSP
        System.Net Information: 0 : [35268] SecureChannel#47891719 - Attempting to restart the session using the user-provided certificate: [Version]
        System.Net Information: 0 : [35268] SecureChannel#47891719 - Left with 1 client certificates to choose from.
        System.Net Information: 0 : [35268] SecureChannel#47891719 - Trying to find a matching certificate in the certificate store.
        System.Net Information: 0 : [35268] SecureChannel#47891719 - Locating the private key for the certificate: [Version]
        System.Net Information: 0 : [35268] SecureChannel#47891719 - Certificate is of type X509Certificate2 and contains the private key.
        System.Net Information: 0 : [35268] SecureChannel#47891719::.AcquireClientCredentials, new SecureCredential() (flags=(ValidateManual, NoDefaultCred, SendAuxRecord, UseStrongCrypto), m_ProtocolFlags=(Tls12Client), m_EncryptionPolicy=RequireEncryption)
        System.Net Information: 0 : [35268] AcquireCredentialsHandle(package = Microsoft Unified Security Protocol Provider, intent  = Outbound, scc     = System.Net.SecureCredential)
        System.Net Information: 0 : [35268] InitializeSecurityContext(credential = System.Net.SafeFreeCredential_SECURITY, context = (null), targetName = <TargetWebSite>, inFlags = ReplayDetect, SequenceDetect, Confidentiality, AllocateMemory, InitManualCredValidation)
        System.Net Information: 0 : [35268] InitializeSecurityContext(In-Buffer length=0, Out-Buffer length=184, returned code=ContinueNeeded).
        System.Net Information: 0 : [35268] InitializeSecurityContext(credential = System.Net.SafeFreeCredential_SECURITY, context = 227c85a89b0:2449d0deff0, targetName = <TargetWebSite>, inFlags = ReplayDetect, SequenceDetect, Confidentiality, AllocateMemory, InitManualCredValidation)
        System.Net Information: 0 : [35268] InitializeSecurityContext(In-Buffers count=2, Out-Buffer length=0, returned code=CredentialsNeeded).

    However, when running from Excel 2010 via com-interop, instead of those last 4 InitializeSecurityContext lines, there are 6 InitializeSecurityContext lines as follows:

        System.Net Information: 0 : [39988] InitializeSecurityContext(credential = System.Net.SafeFreeCredential_SECURITY, context = (null), targetName = <TargetWebSite>, inFlags = ReplayDetect, SequenceDetect, Confidentiality, AllocateMemory, InitManualCredValidation)
        System.Net Information: 0 : [39988] InitializeSecurityContext(In-Buffer length=0, Out-Buffer length=184, returned code=ContinueNeeded).
        System.Net Information: 0 : [39988] InitializeSecurityContext(credential = System.Net.SafeFreeCredential_SECURITY, context = 8a8e2f0:2449d0def90, targetName = <TargetWebSite>, inFlags = ReplayDetect, SequenceDetect, Confidentiality, AllocateMemory, InitManualCredValidation)
        System.Net Information: 0 : [39988] InitializeSecurityContext(In-Buffers count=2, Out-Buffer length=0, returned code=ContinueNeeded).
        System.Net Information: 0 : [39988] InitializeSecurityContext(credential = System.Net.SafeFreeCredential_SECURITY, context = 8a8e2f0:2449d0def90, targetName = <TargetWebSite>, inFlags = ReplayDetect, SequenceDetect, Confidentiality, AllocateMemory, InitManualCredValidation)
        System.Net Information: 0 : [39988] InitializeSecurityContext(In-Buffers count=2, Out-Buffer length=0, returned code=ContinueNeeded).

    The first two InitializeSecurityContext lines are the identical, so presumably crucial difference is on the third InitializeSecurityContext line, where the console EXE has

    context = 227c85a89b0:2449d0deff0

    but the failed run via com-interop executino has

    context = 8a8e2f0:2449d0def90

    After that the diagnostic output looks different, as one would expect.  Does anyone know what that difference means, and how to make the difference go away so that the com-interop execution behaves in the same way as the com-interop execution?

    Tuesday, February 12, 2019 10:09 PM
  • The context is just a token identifying partial formed security content returned by phNewContext so by itself has not much meaning.

    The relevent part is returned code = ContinueNeeded, where your side would need to make another call to InitializeSecurityContext with the token returned (as in the two-leg security context description on the Remark of linked page).

    Is there AcceptSecurityContext or AcquireCredentialsHandle call appear later in the log?


    • Edited by cheong00 Wednesday, February 13, 2019 1:58 AM
    Wednesday, February 13, 2019 1:56 AM
  • Thanks for helping me on this cheong00.  Here's more of the diagnostics.

    For the case that works (running from console EXE), the trace output (starting with the 4 InitializeSecurityContext lines shown previously) continues as follows:

    System.Net Information: 0 : [35268] InitializeSecurityContext(credential = System.Net.SafeFreeCredential_SECURITY, context = (null), targetName = <TargetWebSite>, inFlags = ReplayDetect, SequenceDetect, Confidentiality, AllocateMemory, InitManualCredValidation)
    System.Net Information: 0 : [35268] InitializeSecurityContext(In-Buffer length=0, Out-Buffer length=184, returned code=ContinueNeeded).
    System.Net Information: 0 : [35268] InitializeSecurityContext(credential = System.Net.SafeFreeCredential_SECURITY, context = 227c85a89b0:2449d0deff0, targetName = <TargetWebSite>, inFlags = ReplayDetect, SequenceDetect, Confidentiality, AllocateMemory, InitManualCredValidation)
    System.Net Information: 0 : [35268] InitializeSecurityContext(In-Buffers count=2, Out-Buffer length=0, returned code=CredentialsNeeded).
    System.Net Information: 0 : [35268] SecureChannel#47891719 - We have user-provided certificates. The server has not specified any issuers, so try all the certificates.
    System.Net Information: 0 : [35268] SecureChannel#47891719 - Selected certificate: [Version]
      V3
    [ Lots of lines describing the certificate that was loaded manually from file by the C# code ]
    System.Net Information: 0 : [35268] SecureChannel#47891719 - Left with 1 client certificates to choose from.
    System.Net Information: 0 : [35268] SecureChannel#47891719 - Trying to find a matching certificate in the certificate store.
    System.Net Information: 0 : [35268] SecureChannel#47891719 - Locating the private key for the certificate: [Version]
      V3
    [ Lots of lines, AGAIN describing the certificate that was loaded manually from file by the C# code ]
    System.Net Information: 0 : [35268] SecureChannel#47891719 - Certificate is of type X509Certificate2 and contains the private key.
    System.Net Information: 0 : [35268] SecureChannel#47891719::.AcquireClientCredentials, new SecureCredential() (flags=(ValidateManual, NoDefaultCred, SendAuxRecord, UseStrongCrypto), m_ProtocolFlags=(Tls12Client), m_EncryptionPolicy=RequireEncryption)
    System.Net Information: 0 : [35268] AcquireCredentialsHandle(package = Microsoft Unified Security Protocol Provider, intent  = Outbound, scc     = System.Net.SecureCredential)
    System.Net Information: 0 : [35268] InitializeSecurityContext(credential = System.Net.SafeFreeCredential_SECURITY, context = 227c85a89b0:2449d0deff0, targetName = <TargetWebSite>, inFlags = ReplayDetect, SequenceDetect, Confidentiality, AllocateMemory, InitManualCredValidation)
    System.Net Information: 0 : [35268] InitializeSecurityContext(In-Buffers count=2, Out-Buffer length=1551, returned code=ContinueNeeded).
    System.Net Information: 0 : [35268] InitializeSecurityContext(credential = System.Net.SafeFreeCredential_SECURITY, context = 227c85a89b0:2449d0deff0, targetName = <TargetWebSite>, inFlags = ReplayDetect, SequenceDetect, Confidentiality, AllocateMemory, InitManualCredValidation)
    System.Net Information: 0 : [35268] InitializeSecurityContext(In-Buffers count=2, Out-Buffer length=0, returned code=ContinueNeeded).
    System.Net Information: 0 : [35268] InitializeSecurityContext(credential = System.Net.SafeFreeCredential_SECURITY, context = 227c85a89b0:2449d0deff0, targetName = <TargetWebSite>, inFlags = ReplayDetect, SequenceDetect, Confidentiality, AllocateMemory, InitManualCredValidation)
    System.Net Information: 0 : [35268] InitializeSecurityContext(In-Buffers count=2, Out-Buffer length=0, returned code=OK).
    System.Net Information: 0 : [35268] Remote certificate: [Version]
      V3
    [ Lots of lines describing a certificate with [Subject]=<TargetWebSite>, [Issuer]=HydrantID SSL ICA G2, etc]
    System.Net Information: 0 : [35268] SecureChannel#47891719 - Remote certificate was verified as valid by the user.

    etc

    However, for the case that fails (running from Excel 2010 via com-interop), the trace output (starting with the 6 InitializeSecurityContext lines shown previously) continues as follows:

    System.Net Information: 0 : [39988] InitializeSecurityContext(credential = System.Net.SafeFreeCredential_SECURITY, context = (null), targetName = <TargetWebSite>, inFlags = ReplayDetect, SequenceDetect, Confidentiality, AllocateMemory, InitManualCredValidation)
    System.Net Information: 0 : [39988] InitializeSecurityContext(In-Buffer length=0, Out-Buffer length=184, returned code=ContinueNeeded).
    System.Net Information: 0 : [39988] InitializeSecurityContext(credential = System.Net.SafeFreeCredential_SECURITY, context = 8a8e2f0:2449d0def90, targetName = <TargetWebSite>, inFlags = ReplayDetect, SequenceDetect, Confidentiality, AllocateMemory, InitManualCredValidation)
    System.Net Information: 0 : [39988] InitializeSecurityContext(In-Buffers count=2, Out-Buffer length=0, returned code=ContinueNeeded).
    System.Net Information: 0 : [39988] InitializeSecurityContext(credential = System.Net.SafeFreeCredential_SECURITY, context = 8a8e2f0:2449d0def90, targetName = <TargetWebSite>, inFlags = ReplayDetect, SequenceDetect, Confidentiality, AllocateMemory, InitManualCredValidation)
    System.Net Information: 0 : [39988] InitializeSecurityContext(In-Buffers count=2, Out-Buffer length=0, returned code=ContinueNeeded).
    System.Net Information: 0 : [39988] InitializeSecurityContext(credential = System.Net.SafeFreeCredential_SECURITY, context = 8a8e2f0:2449d0def90, targetName = <TargetWebSite>, inFlags = ReplayDetect, SequenceDetect, Confidentiality, AllocateMemory, InitManualCredValidation)
    System.Net Information: 0 : [39988] InitializeSecurityContext(In-Buffers count=2, Out-Buffer length=326, returned code=ContinueNeeded).
    System.Net Information: 0 : [39988] InitializeSecurityContext(credential = System.Net.SafeFreeCredential_SECURITY, context = 8a8e2f0:2449d0def90, targetName = <TargetWebSite>, inFlags = ReplayDetect, SequenceDetect, Confidentiality, AllocateMemory, InitManualCredValidation)
    System.Net Information: 0 : [39988] InitializeSecurityContext(In-Buffers count=2, Out-Buffer length=0, returned code=ContinueNeeded).
    System.Net Information: 0 : [39988] InitializeSecurityContext(credential = System.Net.SafeFreeCredential_SECURITY, context = 8a8e2f0:2449d0def90, targetName = <TargetWebSite>, inFlags = ReplayDetect, SequenceDetect, Confidentiality, AllocateMemory, InitManualCredValidation)
    System.Net Information: 0 : [39988] InitializeSecurityContext(In-Buffers count=2, Out-Buffer length=0, returned code=OK).
    System.Net Information: 0 : [39988] Remote certificate: [Version]
      V3
    [ Lots of lines describing a certificate with [Subject]=<TargetWebSite>, [Issuer]=Kaspersky which is my antivirus program, etc]
    System.Net Information: 0 : [39988] SecureChannel#2383799 - Remote certificate was verified as valid by the user.
    etc


    FYI, this is the output from the code that I posted on stackoverflow.com, running on the .Net Framework 4.7.2, and with the line

            System.Net.ServicePointManager.SecurityProtocol = System.Net.SecurityProtocolType.Tls12;

    Added right at the start before the System.Net.WebRequest.Create(new Uri(WEBSITE)) line.

    Regarding AcceptSecurityContext and AcquireCredentialsHandle, AcceptSecurityContext occurs nowhere in either case.  But AcquireCredentialsHandle occurs twice for the case that succeeds (once shown in the previous reply, once shown in this post), and for the case that fails AcquireCredentialsHandle only occurs once (shown in the previous reply).

    Is that enough information?

    Thanks, Paul

    Wednesday, February 13, 2019 8:20 AM
  • AcquireCredentialsHandle exists in "success" case at line [35268] after the security context is initialized, and SSL channel has been established, therefore can proceed to authentication.

    Assuming you've not left out important detail in the logs, I think in the second case it's trying to do without client certs. (Look again for "We have user-provided certificates" in the failed logs)

    If that line does not exists, I think something makes the HttpWebRequest to create connection without attaching the certs in VBA case.

    EDIT: Simon @ StackOverflow might have nailed your case. The web scanner component acts as HTTP proxy on your outgoing traffic and may have influrence on the communication. Try disable it while you test again and see if this will fix the problem.
    • Edited by cheong00 Wednesday, February 13, 2019 10:17 AM
    • Marked as answer by Stochastically Wednesday, February 13, 2019 3:51 PM
    Wednesday, February 13, 2019 10:12 AM
  • Thanks cheong00.  I disabled Kaspersky anti-virus and the problem went away. Completely bizarre, because the identical version of Kaspersky and identical version of Excel with my identical C# program didn't cause the problem on computer A (with Windows 10 1803).  Anyway, thanks again for your help :-).
    Wednesday, February 13, 2019 3:51 PM