Question

Ok, so I’m working on speeding up the time it takes to load my server. Mainly pre-loading the user accounts from the database, and adding them to a list of player objects with 13 variables apiece.

I had decided that the first step was to see which parts of the system were taking the longest, so I setup DateTime variables to count the number of milliseconds each piece of code took to execute.

We have about 40k accounts in the database, and they all load almost instantly into a list of strings that we then extrapolate into their variables, It’s not the most efficient way to do things, but it’s decent, as far as speed goes, and it both works and is error free.

anyway, it takes about 4 minutes to extrapolate the 40k accounts into their sub-sequent objects, setting the variables and adding the object to the list of preloaded data.

I added DateTime variables around each variable setting, and then ascertained the execution time, there was only one problem, all the numbers were 0, I presumed that I screwed up my code somewhere, so i tested it in a few different places, and it worked.

So, now I’m a little stumped, trying to figure out where all the time and CPU cycles are going.

0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
0 0 0 0 0 0 0 0 0 0 0 0 0 0 0

I’m testing the initial object creation, 13 variables, the adding of the object to the list, then I output the overall time. It’s not meant to be accurate, just to help me understand where I need to change things, I was a little upset at 40k lines of 0’s

Is there something I’m missing? as far as i can tell, it shouldn’t take more then 30 seconds (plus whatever the loop itself costs) to load this data.

Is DateTime just that inaccurate? or did I mess up somewhere, or…

I would appreciate any insight you may have into this.

Thank you for your time:

P.S. Just in case you want to see the whole code, this is it:

int pcounti = 0;
current.Text = "Pre-Loading Players."; bar.Value = 35;
List<string[]> p1 = SDB.get("Select * from players");
if (p1 != null)
{
    int dbplayercount = p1.Count;
    //REMOVE ME
    List<string> tolog = new List<string>();
    foreach (string[] s in p1)
    {
        // Check if this user already has a PDB
        try
        {
            PDB temppdb = PDB.find(s[1].Trim().ToLower());
            if (temppdb != null)
            {
                Console.WriteLine("Deleting");
                SDB.donow("DELETE FROM players WHERE username = '" + s[1] + "' AND UID != " + temppdb.UID);
            }
        }
        catch (Exception e)
        {
            Console.WriteLine("Error:");
            Console.WriteLine(e.Message);
            Console.WriteLine(e.StackTrace);
        }

        // Generate PDB
        try
        {
            //We need the database to be all lowercase :D
            if (HasCapitals(s[1])) SDB.donow("UPDATE players SET username = '" + s[1].ToLower() + "' WHERE username = '" + s[1] + "'");
            DateTime one = DateTime.Now.ToUniversalTime();
            PDB pdb = new PDB();
            DateTime two = DateTime.Now.ToUniversalTime();
            pdb.UID = Convert.ToInt64(s[0]);
            DateTime three = DateTime.Now.ToUniversalTime();
            pdb.username = s[1].Trim().ToLower();
            DateTime four = DateTime.Now.ToUniversalTime();
            pdb.password = s[3];
            DateTime five = DateTime.Now.ToUniversalTime();
            pdb.group = Byte.Parse(s[4]);
            DateTime six = DateTime.Now.ToUniversalTime();
            pdb.ip = s[10];
            DateTime seven = DateTime.Now.ToUniversalTime();
            pdb.color = s[11];
            DateTime eight = DateTime.Now.ToUniversalTime();
            try
            {
                pdb.warn = Convert.ToInt32(s[14]);
            }
            catch
            {
                pdb.warn = 0;
            }
            DateTime nine = DateTime.Now.ToUniversalTime();
            try
            {
                pdb.money = Convert.ToInt32(s[15]);
            }
            catch
            {
                pdb.money = 0;
            }
            DateTime ten = DateTime.Now.ToUniversalTime();
            foreach (string st in s[16].Split(','))
            {
                if (String.IsNullOrEmpty(st)) continue;
                pdb.Allowed.Add(st);
            }
            DateTime eleven = DateTime.Now.ToUniversalTime();
            foreach (string st in s[17].Split(','))
            {
                if (String.IsNullOrEmpty(st)) continue;
                pdb.Denied.Add(st);
            }
            DateTime twelve = DateTime.Now.ToUniversalTime();
            pdb.Title = s[18];
            DateTime thirteen = DateTime.Now.ToUniversalTime();
            foreach (string st in s[19].Split(','))
            {
                if (String.IsNullOrEmpty(st)) continue;
                pdb.Titles.Add(st);
            }
            DateTime fourteen = DateTime.Now.ToUniversalTime();
            PDB.DB.Add(pdb);
            DateTime fifteen = DateTime.Now.ToUniversalTime();
            int one1 = ((two - one).Milliseconds);
            int two1 = ((three - two).Milliseconds);
            int three1 = ((four - three).Milliseconds);
            int four1 = ((five - four).Milliseconds);
            int five1 = ((six - five).Milliseconds);
            int six1 = ((seven - six).Milliseconds);
            int seven1 = ((eight - seven).Milliseconds);
            int eight1 = ((nine - eight).Milliseconds);
            int nine1 = ((ten - nine).Milliseconds);
            int ten1 = ((eleven - ten).Milliseconds);
            int eleven1 = ((twelve - eleven).Milliseconds);
            int twelve1 = ((thirteen - twelve).Milliseconds);
            int thirteen1 = ((fourteen - thirteen).Milliseconds);
            int fourteen1 = ((fifteen - fourteen).Milliseconds);
            int fifteen1 = ((fifteen - one).Milliseconds);
            string sta = one1 + " " + two1 + " " + three1 + " " + four1 + " " + five1 + " " + six1 + " " + seven1 + " " + eight1 + " " + nine1 + " " + ten1 + " " + eleven1 + " " + twelve1 + " " + thirteen1 + " " + fourteen1 + " " + fifteen1;
            tolog.Add(sta);
            pcounti++;
        }
        catch (Exception e)
        {
            Console.WriteLine("Error:");
            Console.WriteLine(e.Message);
            Console.WriteLine(e.StackTrace);
            continue;
        }

                    // write to file
        StreamWriter sw = new StreamWriter(File.Create("test.txt"));
        foreach (string st in tolog)
        {
            sw.WriteLine(st);
        }
        sw.Flush();
        sw.Close();
    }
}

Also, note, After writing most of this, I looked over it one last time, to make sure I didn’t miss anything stupid, and realized i had not checked the first few lines of each loop, I added them to the checks and came out with the same thing.

Also, as a second note, it took 4 minutes “before” I added the performance checking, it doesn’t take a lot longer now.

EDIT
changing Milliseconds to TotalMilliseconds made a differance, but only in showing the results as doubles, rather then int’s, resulting in a test output like this:

0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
0.9765 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
0 0 0 0.9765 0 0 0 0 0 0 0 0 0 0 0 0.9765
0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
0.9766 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0

The first number isn’t added into the total at the end.
also, I should mention that every now and then there’s a 1 mixed in with the 0’s (there was the first time as well) but i’m still as stumped as I was before, time to profile.

EDIT 2
The results of the profiling, note that all the time counters and the progress bar update at the end are not permanent, and were not there in the original course of events that led me to trying to do this in the first place. (ie 4-5 mins load time)

Answer 1

You haven’t really given us any breakdown of where the performance degradation is coming in… or indicated how long the core SQL query takes, but some thoughts:

List<string[]> p1 = SDB.get("Select * from players");

this is buffering the entire set of data; personally, I’d be using a non-buffered API for this, i.e. processing each row as it arrives. I’d also be using typed objects, not a string[] – something like dapper-dot-net would do both very neatly.

I would also be profiling that individual query to see how much of the time is in there vs the processing of the data once it is local.

You seem to be doing a lot (2*N+1) of find (PDB.find(s[1].Trim().ToLower()) and UPDATE / DELETE etc; I think you are drowning in latency. For this volume, I’d be tempted to just throw (SqlBulkCopy) the source data into a table on the DB server and do some set-based queries (3; one update, one delete, one insert – or a MERGE if available) at the db.