Question

in my current ASP.net MVC 3.0 project i am stuck with a situation.

1. I have four .txt files each has approximatly 100k rows of records
2. These files will be replaced with new files on weekly bases.

I need to Query data from these four text files, I am not able to choose the best and efficient way to do this.

3 ways I could think

1. Convert these text files to XML on a weekly basis and query it with Linq-XML
2. Run a batch import weekly from txt to SQL Server and query using Linq-Entities
3. avoid all conversions and query directly from text files.

Can any one suggest a best way to deal with this situation.

update:

---

Url of the Text File

I should connect to this file with credentials.

once i connect successfully, I will have the text file as below with Pipeline as Deliminator
This is the text file

---

Now i have to look up for the field highlighted in yellow and get the data in that row.

Note: First two lines of the text file are headers of the File.

Answer 1

Well As i Found a way my self. Hope this will be useful for any who are interested to get this done.

string url = "https://myurl.com/folder/file.txt";
WebClient request = new WebClient();
                request.Credentials = new NetworkCredential(ConfigurationManager.AppSettings["UserName"], ConfigurationManager.AppSettings["Password"]);
                Stream s = request.OpenRead(url);
                using (StreamReader strReader = new StreamReader(s))
                {
                    for (int i = 0; i <= 1; i++)
                        strReader.ReadLine();
                    while (!strReader.EndOfStream)
                    {
                        var CurrentLine = strReader.ReadLine();
                        var count = CurrentLine.Split('|').Count();
                        if (count > 3 && CurrentLine.Split('|')[3].Equals("SearchString"))
                        {
                            #region Bind Data to Model
                            //var Line = CurrentLine.Split('|');
                            //CID.RecordType = Line[0];
                            //CID.ChangeIdentifier = Line[1];
                            //CID.CoverageID = Convert.ToInt32(Line[2]);
                            //CID.NationalDrugCode = Line[3];
                            //CID.DrugQualifier = Convert.ToInt32(Line[4]); 
                            #endregion
                            break;
                        }
                    }
                    s.Close();
                }
                request.Dispose();