none
Read large binary file into byte array - File.ReadAllBytes throws 'System.OutOfMemoryException' RRS feed

  • Question

  • Dim docContent() As Byte = File.ReadAllBytes(path:=CachedFilePath)

    I am trying to read binary files into a Byte array.
    I get a 'System.OutOfMemoryException'.
    I believe that I need to setup a loop and write chunks to the
    byte array. Some vb code would be greatly appreciated.

    Thanks


    Tim

    Thursday, June 22, 2017 2:56 PM

All replies

  • Dim docContent() As Byte = File.ReadAllBytes(path:=CachedFilePath)

    I am trying to read binary files into a Byte array.
    I get a 'System.OutOfMemoryException'.
    I believe that I need to setup a loop and write chunks to the
    byte array. Some vb code would be greatly appreciated.

    Thanks


    Tim

    I don't see how that would help.

    If you were to read in some, process that and incrementally write that out to a file (so it's out of scope per loop), then sure, but you're holding it all in memory. I don't see a way around that.


    "A problem well stated is a problem half solved.” - Charles F. Kettering


    • Edited by Frank L. Smith Thursday, June 22, 2017 3:16 PM ...reworded
    Thursday, June 22, 2017 3:15 PM
  • Tim,

    How about explain your ultimate goal. There might be another way to go about getting it done.


    "A problem well stated is a problem half solved.” - Charles F. Kettering

    Thursday, June 22, 2017 4:37 PM
  • Thanks Charles

    I have a .Net web service (vs 2010).

    One of the function calls downloads the requested file from a file repository application
    (FileNet). It then loads the file into a byte array and returns it to the calling app.

    It has worked for years without any issues, but we have recently added some files
    which are greater than 110 M.

    The server the web service is running on is Server 2003.


    Tim

    Thursday, June 22, 2017 4:44 PM
  • Thanks Charles

    I have a .Net web service (vs 2010).

    One of the function calls downloads the requested file from a file repository application
    (FileNet). It then loads the file into a byte array and returns it to the calling app.

    It has worked for years without any issues, but we have recently added some files
    which are greater than 110 M.

    The server the web service is running on is Server 2003.


    Tim

    I'm Frank.

    I don't have a suggestion other than maybe rethinking it some.

    For example, have you tried it with a FileStream? That's not not a byte array but a stream seems to be what you're eventually after?


    "A problem well stated is a problem half solved.” - Charles F. Kettering

    Thursday, June 22, 2017 4:57 PM

  • It has worked for years without any issues, but we have recently added some files
    which are greater than 110 M.

    That doesn't sound right.

    Const filePath As String = _
                "K:\Photos 4.zip"
    
    Dim test() As Byte = IO.File.ReadAllBytes(filePath)
    
    Stop

    That's a little more than 200 megs and I'm on a 32-bit O/S.

    Do you have a lot else that's loaded in memory at that point in time?


    "A problem well stated is a problem half solved.” - Charles F. Kettering

    Thursday, June 22, 2017 5:05 PM
  • Same here with even a larger file, 32 bit win 10

    Public Class Form1
        Private Sub Form1_Load(sender As Object, e As EventArgs) Handles MyBase.Load
            Const filePath As String = "D:\Downloads\AcronisTrueImage2015_ur_en-US.msi"
            Dim test() As Byte = IO.File.ReadAllBytes(filePath)
            Stop 'test {Length=339881984}
        End Sub
    End Class
    

    File size is 339,881,984, takes a few seconds to read it.

    Thursday, June 22, 2017 7:22 PM
  • Hi Tim_Shaf,

    I use code below to read file quickly,

     Dim filename As String = "D:\test\7.txt"
            Dim fs As System.IO.FileStream = System.IO.File.Open(filename, IO.FileMode.Open, IO.FileAccess.Read)
            Dim buff(fs.Length) As Byte
            If fs.Length > 0 Then
                fs.Read(buff, 0, fs.Length - 1)
                fs.Close()
                buff = Nothing
                fs = Nothing
            Else
                'Empty File
            End If

    If the file size is big, I suggest you to read it in chunks, and put a try catch around in case thing

    hang.

    Best Regards,

    Cherry


    MSDN Community Support
    Please remember to click "Mark as Answer" the responses that resolved your issue, and to click "Unmark as Answer" if not. This can be beneficial to other community members reading this thread. If you have any compliments or complaints to MSDN Support, feel free to contact MSDNFSF@microsoft.com.

    Friday, June 23, 2017 6:45 AM
  • I have separated out the code into a simple web service with only
    one call GetByteArrayFromFile which reads the file and returns a byte array.
    If the web service call fails with an "System.OutOfMemoryException", it reads
    the file within the module.

    Also included a similar routine 'ModuleGetByteArrayFromFile' that performs
    the same function without going thru the web service.
    Note there aren't any OutofMemory Exceptions when ran from the module.

    I am running Visual Studio 2010 using .NET Framework 3.5.

    Looking for ways to avoid these Exceptions in the web service.

    I would prefer to stay with a byte array since I have multiple applications
    that would need to be changed. I have tried using System.IO.FileStream to get
    the contents into a byte array with similar results.

    Is there some setting in the web.config that would increase the memory limit
    of the web service?

    **********************************************************
    Results running on Windows Server 2003

    ModuleGetByteArrayFromFile(FNames:=FNames) - Results
    Read time: 1.10 seconds
    Write time: 0.52 seconds
    MP_M-23-DMP!1u3r02zzz.DOC Read\Write time: 1.64 seconds,size:77582336
    Read time: 2.40 seconds
    Write time: 1.87 seconds
    EDMG_EDG-16u3r00zzz.DOC Read\Write time: 4.27 seconds,size:217624064
    Read time: 2.61 seconds
    Write time: 2.01 seconds
    PEP_29-03u3r00zzz.DOC Read\Write time: 4.62 seconds,size:306111488

    WebServiceGetByteArrayFromFile(FNames:=FNames) - Results
    Web service read time: 3.20 seconds
    Write time: 0.62 seconds
    MP_M-23-DMP!1u3r02zzz.DOC Read\Write time: 3.83 seconds,size:77582336
    A first chance exception of type 'System.OutOfMemoryException' occurred in System.Xml.dll
    OutOfMemoryException in web service, module read time: 4.71 seconds
    Write time: 1.45 seconds
    EDMG_EDG-16u3r00zzz.DOC Read\Write time: 6.19 seconds,size:217624064
    A first chance exception of type 'System.Web.Services.Protocols.SoapException' occurred in System.Web.Services.dll
    OutOfMemoryException in web service, module read time: 3.04 seconds
    Write time: 2.04 seconds
    PEP_29-03u3r00zzz.DOC Read\Write time: 5.09 seconds,size:306111488
    **********************************************************

    'Test Module
    
    Imports System.IO
    
    Module Initial
    
      Private CachedPath As String = "D:\Cache\"
    
      Sub Main()
        Dim FNames As New List(Of String)
        FNames.Add("MP_M-23-DMP!1u3r02.DOC")   '75 Meg
        FNames.Add("EDMG_EDG-16u3r00.DOC")    '212 Meg
        FNames.Add("PEP_29-03u3r00.DOC")      '298 Meg
        'ModuleGetByteArrayFromFile(FNames:=FNames)
        WebServiceGetByteArrayFromFile(FNames:=FNames)
      End Sub
    
      Sub ModuleGetByteArrayFromFile(ByVal FNames As List(Of String))
        For Each FName As String In FNames
          Dim sw As Stopwatch = Stopwatch.StartNew
          Dim docContent As Byte() = New Byte() {}
          Try
            docContent = File.ReadAllBytes(path:=CachedPath & FName)
            Debug.Print("Read time: " & ElapsedTime(sw))
          Catch ex As Exception
            If ex.Message.Contains("System.OutOfMemoryException") Then
              sw.Reset() : sw.Start()
              docContent = File.ReadAllBytes(path:=CachedPath & FName)
              Debug.Print("OutOfMemoryException in web service, local read time: " & ElapsedTime(sw))
            Else
              Stop
            End If
          End Try
          Dim PathFName As String = CachedPath & FName.Replace(".", "zzz.")
          Dim sw1 As Stopwatch = Stopwatch.StartNew
          System.IO.File.WriteAllBytes(path:=PathFName, bytes:=docContent)
          Debug.Print("Write time: " & ElapsedTime(sw1))
          docContent = Nothing
          Debug.Print(Path.GetFileName(PathFName) & " Read\Write time: " & ElapsedTime(sw) & ",size:" &
            FileLen(PathFName).ToString)
        Next
      End Sub
    
      Sub WebServiceGetByteArrayFromFile(ByVal FNames As List(Of String))
        Dim Ws As New ProcWs.ProcSvcs
        Ws.UseDefaultCredentials = True
        Ws.Timeout = (20 * 60 * 1000)
        For Each FName As String In FNames
          Dim sw As Stopwatch = Stopwatch.StartNew
          Dim docContent As Byte() = New Byte() {}
          Try
            docContent = Ws.GetByteArrayFromFile(PathFName:=CachedPath & FName)
            Debug.Print("Web service read time: " & ElapsedTime(sw))
          Catch ex As Exception
            If ex.Message.Contains("System.OutOfMemoryException") Then
              sw.Reset() : sw.Start()
              docContent = File.ReadAllBytes(path:=CachedPath & FName)
              Debug.Print("OutOfMemoryException in web service, module read time: " & ElapsedTime(sw))
            Else
              Stop
            End If
          End Try
          Dim PathFName As String = CachedPath & FName.Replace(".", "zzz.")
          Dim sw1 As Stopwatch = Stopwatch.StartNew
          System.IO.File.WriteAllBytes(path:=PathFName, bytes:=docContent)
          Debug.Print("Write time: " & ElapsedTime(sw1))
          docContent = Nothing
          Debug.Print(Path.GetFileName(PathFName) & " Read\Write time: " & ElapsedTime(sw) & ",size:" &
            FileLen(PathFName).ToString)
        Next
      End Sub
    
      Private Function ElapsedTime(ByVal sw As Stopwatch) As String
        Return (sw.ElapsedMilliseconds / 1000).ToString("0.00 seconds")
      End Function
    
    End Module
    
    '**********************************************************
    'Web Service 
    Imports System.Web.Services, System.Web.Services.Protocols, System.ComponentModel
    
    <System.Web.Services.WebService(Description:="Testing", _
        Namespace:="http://Testing.org/")> _
    <System.Web.Services.WebServiceBinding(ConformsTo:=WsiProfiles.BasicProfile1_1)> _
    <ToolboxItem(False)> _
    Public Class ProcSvcs
      Inherits System.Web.Services.WebService
    
      <WebMethod(Description:="Byte array from File")> _
      Public Function GetByteArrayFromFile(ByVal PathFName As String) As Byte()
        Try
          Dim Content() As Byte = System.IO.File.ReadAllBytes(path:=PathFName)
          Return Content
        Catch ex As Exception
          Throw New Exception(ex.Message)
        End Try
      End Function
    
    End Class
    


    Tim

    Tuesday, June 27, 2017 1:49 PM
  • If you want to trap for a particular exception type then look for the type, not the name.

    Also, why not declare docContent inline?

    None of that will fix the problem here but I'm confused: It looks like you're reading in the byte array then writing to a local file?

    If so then a FileStream is much more appropriate so I'm sure you're doing something else with "docContent"?


    "A problem well stated is a problem half solved.” - Charles F. Kettering

    Tuesday, June 27, 2017 2:35 PM
  • Frank,

    My example is just writing to a local file.
    It would normally return the file byte array via the
    web service to the caller and the caller would save it
    to a local file.

    Every web service example I find that returns binary files
    uses a binary array.

    Do you have an example of a web service returning a FileStream?


    Tim

    Tuesday, June 27, 2017 4:24 PM
  • Frank,

    My example is just writing to a local file.
    It would normally return the file byte array via the
    web service to the caller and the caller would save it
    to a local file.

    Every web service example I find that returns binary files
    uses a binary array.

    Do you have an example of a web service returning a FileStream?


    Tim

    I've never written a web service in my life so no. ;-)

    *****

    If the main objective is transporting the file's data, have you considered binary serialization?


    "A problem well stated is a problem half solved.” - Charles F. Kettering

    Tuesday, June 27, 2017 4:27 PM
  • Thanks Charles

    I have a .Net web service (vs 2010).

    One of the function calls downloads the requested file from a file repository application
    (FileNet). It then loads the file into a byte array and returns it to the calling app.

    It has worked for years without any issues, but we have recently added some files
    which are greater than 110 M.

    The server the web service is running on is Server 2003.


    Tim

    Hi Tim,

    There should be no need to read the source stream into a local array just to write it out to the response stream.  Once the stream is open to the remote file, use the CopyTo method to write it out to the response stream.

    This particular exception is based on available managed memory, so a web application with many users hitting it can experience the error more readily than, say, a desktop application with only a single user and little competition for resources.


    Reed Kimble - "When you do things right, people won't be sure you've done anything at all"

    Tuesday, June 27, 2017 4:42 PM
  • Same here with even a larger file, 32 bit win 10

    Public Class Form1
        Private Sub Form1_Load(sender As Object, e As EventArgs) Handles MyBase.Load
            Const filePath As String = "D:\Downloads\AcronisTrueImage2015_ur_en-US.msi"
            Dim test() As Byte = IO.File.ReadAllBytes(filePath)
            Stop 'test {Length=339881984}
        End Sub
    End Class

    File size is 339,881,984, takes a few seconds to read it.

    It is a web application.  The server is likely under load - we have no idea how many managed applications it is hosting or how many users those applications have.  You can't really compare a web application to a desktop application in terms of managed memory availability since it has to be distributed amongst all applications and user sessions.


    Reed Kimble - "When you do things right, people won't be sure you've done anything at all"

    Tuesday, June 27, 2017 4:46 PM
  • Frank,

    I am using .NET Framework 3.5.

    Looks like CopyTo was added to 4.0.
    I have several hundred functions in this web service and
    would prefer not to update to Framework 4.0.

    Thanks


    Tim

    Tuesday, June 27, 2017 5:03 PM
  • Frank,

    My example is just writing to a local file.
    It would normally return the file byte array via the
    web service to the caller and the caller would save it
    to a local file.

    Every web service example I find that returns binary files
    uses a binary array.

    Do you have an example of a web service returning a FileStream?


    Tim

    Oh, sorry, missed this twist.

    You are talking about classic SOAP web services?  That will make things more difficult since a binary array is actually returned as a BASE64 encoded string within the XML of the body of the web service response.  So a straight stream-to-stream copy wouldn't be possible.

    There may not be a way around this at the code level. If you have no other choice of protocol for getting the document then you may have to go to the ASP.Net forums and/or an IIS forum and see if you can tune the web application to allow more managed memory for this particular use, or if the answer is to add more memory to the server and/or distribute the application across multiple machines.


    Reed Kimble - "When you do things right, people won't be sure you've done anything at all"

    Tuesday, June 27, 2017 5:20 PM
  • Frank,

    I am using .NET Framework 3.5.

    Looks like CopyTo was added to 4.0.
    I have several hundred functions in this web service and
    would prefer not to update to Framework 4.0.

    Thanks


    Tim

    Hi Tim, that was me, Reed, who mentioned CopyTo.

    Even without the method you could implement the same idea with a little loop that reads a small chunk from one stream and writes it to the other.  Its easy enough to do, but it turns out it won't help in this situation.

    I didn't realize you were using an SOAP web service to get the source file when I wrote that reply.


    Reed Kimble - "When you do things right, people won't be sure you've done anything at all"

    Tuesday, June 27, 2017 5:23 PM
  • Tim,

    I'm out of my bailiwick here but what Reed shows makes sense.

    Earlier I kept looking at your code thinking "it acts like it's being loaded over and over".

    It now makes sense that you're getting the OOM exception.


    "A problem well stated is a problem half solved.” - Charles F. Kettering

    Tuesday, June 27, 2017 5:28 PM
  • Reed,

    I appreciate everyone's efforts regarding this issue but I am thinking that you are correct regarding I should post this to an ASP.Net forum.

    Thanks


    Tim

    Tuesday, June 27, 2017 6:28 PM