Sign in
Microsoft.com
 
Microsoft
 
 
 

locked
The performance of the System.IO ReadToEnd RRS feed

 > 

  • 問題

  • Question
    登入以投票
    0
    登入以投票
    I have a lot of large txt file to process , thest txt files are server log file , we want to find the right result we want , we process with the steps below.

    1.Send a report to server
    2.server will create a new file
    3.we find the report in file.

    so we will have a lot of the reports per day , and we have to open the file as ReadToEnd and use Regex to return the result that if the report is in the logfile. the file it's not big,only 6m-10m,but when we read it to memory,it may cause the memory leak.also i dispose and close the object, but it will not work, the memory will increase without stop.

    so i set the string to null , then the memory won't be large to let the computer down , but it still eats a lot of the memory, you can check my codes below.

    private bool FindinLog(string tmid,string mi,string iid)

    {

    bool returnValue = false;

    for (int fileNumbers = 0; fileNumbers < legalFileName.Count; fileNumbers++)

    {

    string fileFullPath = logFilePath + "\\" + legalFileName[fileNumbers].ToString();

    if (File.Exists(fileFullPath))

    {

    using (StreamReader fileReader = new StreamReader(fileFullPath))

    {

    StringBuilder fileAll = new StringBuilder();

    fileAll.Append(fileReader.ReadToEnd());

    fileReader.Close();

    fileReader.Dispose();

    string para = "";

        if (Regex.IsMatch(fileAll.ToString(), para))

        {

            returnValue = true;

            fileAll = null;

            break;

        }

        else

        {

            fileAll = null;

            continue;

        }

    }

    }

    }

    GC.Collect();

    GC.Collect(1, GCCollectionMode.Forced);

    GC.Collect(2, GCCollectionMode.Forced);

    return returnValue;

    }

    Many Thanks..

    2009年2月2日 10:10

所有回覆

  • Question
    登入以投票
    0
    登入以投票

    I think the main problem is, you load and process the file's content all in one time. One way to improve it, is trying to read line by line and process it with the business logic, in your case, it compare it with the para string.

    1 private bool FindinLog(string tmid,string mi,string iid)  
    2 {  
    3     bool returnValue = false;  
    4     string para = "SibhoReport.*TMID=%22" + tmid + ".*MI=%22" + mi + ".*IID=%22" + iid;  
    5     for (int fileNumbers = 0; fileNumbers < legalFileName.Count; fileNumbers++)  
    6     {  
    7         string fileFullPath = logFilePath + "\\" + legalFileName[fileNumbers].ToString();  
    8         if (File.Exists(fileFullPath))  
    9         {  
    10             using (StreamReader fileReader = new StreamReader(fileFullPath))  
    11             {  
    12                 while (fileReader.Peek() > 0)  
    13                 {                   
    14                     if (Regex.IsMatch(fileReader.ReadLine(), para))  
    15                     {  
    16                         returnValue = true;  
    17                         break;  //Exit using loop  
    18                     }  
    19                 }  
    20             }  
    21             if (returnValue)   
    22                 break;  //Exit for loop  
    23         }  
    24     }  
    25     return returnValue;  
    26 } 

    By reading someone's performance test, this will improve 82% of processing time.
    Microsoft MVP, Microsoft Community Star(TW & HK), MCT, MCSD, MCAD, MCSE+I, MCDBA, MCDST, MCSA
    2009年2月4日 7:23
  • Question
    登入以投票
    0
    登入以投票
    OK..I will try.. Thank you very much..
    Jing Guo's Facebook profile
    2009年2月4日 8:01
  • Question
    登入以投票
    0
    登入以投票
    After trying it, please come back and tell us the result. Hope it helps.
    Microsoft MVP, Microsoft Community Star(TW & HK), MCT, MCSD, MCAD, MCSE+I, MCDBA, MCDST, MCSA
    2009年2月4日 8:42