Question

I am looking for an efficient way to convert a large int[] into a string[] of csv strings where each csv is limited to a maximum of 4000 characters. The values in the array could be anything between 1 and int.MaxValue.

Here is my final code:

public static string[] GetCSVsFromArray(int[] array, int csvLimit)
{
    List<string> parts = new List<string>();
    StringBuilder sb = new StringBuilder();
    foreach(int id in array)
    {
        string intId = id.ToString();
        if (sb.Length + intId.Length < csvLimit)
            sb.Append(intId).Append(",");
        else
        {
            if (sb.Length > 0)
                sb.Length--;
            parts.Add(sb.ToString());
            sb.Length = 0;
        }
    }
    if(sb.Length>0)
       parts.Add(sb.ToString());
    return parts.ToArray();
}

Is there a more efficient way to do this?

So here is what I am now using (I was able to change the return parameter to the List type to save the ToArray() call at the end):

public static List<string> GetCSVsFromArray(int[] array, int csvLimit)
{
    List<string> parts = new List<string>();
    StringBuilder sb = new StringBuilder();
    foreach(int id in array)
    {
        string intId = id.ToString();
        if (sb.Length + intId.Length < csvLimit)
            sb.Append(intId).Append(",");
        else
        {
            if (sb.Length > 0)
                sb.Length--;
            parts.Add(sb.ToString());
            sb.Length = 0;
        }
    }
    if(sb.Length>0)
       parts.Add(sb.ToString());
    return parts;
}

Performance results:

10,000,000 items csv Limit of 4000 characters

• Original: 2,887.488ms
• GetIntegerDigitCount: 3105.355ms
• Final: 2883.587ms

Whilst I only saved 4ms removing the ToArray() call on my developer machine this seems to make a significant difference on a much slower machine (saved over 200ms on a DELL D620)

Answer 1

You are doing a lot of heap memory allocations when creating a new string for each number just to calculate number of digits. Use following method to calculate number of digits in the number (see method below).

So instead of

string intId = id.ToString();
if (sb.Length + intId.Length < csvLimit)

Just use:

if (sb.Length + this.GetIntegerDigitCount(id) < csvLimit)

Results:

• 2 times faster on 10 million numbers
• Old: 4316ms, New:1983ms, Diff: 2333ms. Faster 217.6%

EDIT: More results on large csv limit

Items:10000000; csvLimit:4000; Old:2091ms, New:1868ms, Diff:223ms
faster = 111.937901498929%

---

Code I’ve used to measure time:

 double elapsedOld = 0;
 double elapsedNew = 0;
 int count = 10000000;
 int csvLimit = 4000;
 var items = Enumerable.Range(0, count).ToArray();
 var watch = Stopwatch.StartNew();
 this.GetCsVsFromArray(items, csvLimit);
 watch.Stop();
 elapsedOld = watch.ElapsedMilliseconds;

 watch = Stopwatch.StartNew();
 this.GetCsVsFromArrayTuned(items, csvLimit);
 watch.Stop();
 elapsedNew = watch.ElapsedMilliseconds;
 var stat = String.Format(
     "Items:{0}; csvLimit:{1}; Old:{2}ms, New:{3}ms, Diff:{4}ms faster = {5}%",                
     count,
     csvLimit,
     elapsedOld,
     elapsedNew,
     elapsedOld - elapsedNew,
     elapsedOld * 100 / elapsedNew);

---

GetIntegerDigitCount:

public int GetIntegerDigitCount(int valueInt)
{
    double value = valueInt;
    int sign = 0;
    if (value < 0)
    {
        value = -value;
        sign = 1;
    }

    if (value <= 9)
    {
        return sign + 1;
    }

    if (value <= 99)
    {
        return sign + 2;
    }

    if (value <= 999)
    {
        return sign + 3;
    }

    if (value <= 9999)
    {
        return sign + 4;
    }

    if (value <= 99999)
    {
        return sign + 5;
    }

    if (value <= 999999)
    {
        return sign + 6;
    }

    if (value <= 9999999)
    {
        return sign + 7;
    }

    if (value <= 99999999)
    {
        return sign + 8;
    }

    if (value <= 999999999)
    {
        return sign + 9;
    }

    return sign + 10;
}