What is the fastest and efficient way to read from a large CSV file in C#?
问题
全部回复
-
You could split the files then process them. During processing you could have a cancellation option and also use a delegate to show progress.
Rough chunking process.
public class FileOperations { private static string _baseFileName = "Your base file name for splitting"; private static bool WriteChunk(IEnumerator<string> lineIterator, int splitSize, int chunk) { // in this case place split file in a folder below the app named ChunkFiles // Then create a file name with chunk param which is the index from the caller for statement var fileName = Path.Combine(AppDomain.CurrentDomain.BaseDirectory, "ChunkFiles", $"{_baseFileName}_{chunk + 1}.txt"); using (var writer = File.CreateText(fileName)) { for (int index = 0; index < splitSize; index++) { if (!lineIterator.MoveNext()) { return false; } writer.WriteLine(lineIterator.Current); } } return true; } /// <summary> /// File to split /// </summary> private static string _incomingFileName = "SomeFileName"; /// <summary> /// Split file by splitSize (lines to split by) /// </summary> /// <param name="splitSize"></param> public static void SplitFileVanilla(int splitSize) { if (!File.Exists(_incomingFileName)) { throw new FileNotFoundException(_incomingFileName); } using (var lineIterator = File.ReadLines(_incomingFileName).GetEnumerator()) { var stillGoing = true; for (int chunkIndex = 0; stillGoing; chunkIndex++) { stillGoing = WriteChunk(lineIterator, splitSize, chunkIndex); } } } }
Please remember to mark the replies as answers if they help and unmarked them if they provide no help, this will help others who are looking for solutions to the same or similar problem. Contact via my Twitter (Karen Payne) or Facebook (Karen Payne) via my MSDN profile but will not answer coding question on either.
NuGet BaseConnectionLibrary for database connections.
StackOverFlow
