Question

Note:
the question is misleading.. i thought it was using more than one parameter that causes a memory error.. but that’s not the reason.. the reason was an incorrectly formed sql statement.. see the answer below.

if create an sqlite statement that uses the same parameter more than once ie

    NSString* updateStmt = @"INSERT INTO search_email(..., subject, ...)"
    " SELECT ..., :subject, ...,"
    " coalesce((SELECT search_email.threadID "
    " FROM search_email "
    " WHERE search_email.subject MATCH :subject2 "
    " ),"
    " :uid"
    " )";

int subjectIndex = sqlite3_bind_parameter_index(searchEmailInsertStmt,":subject");
int subjectIndex2 = sqlite3_bind_parameter_index(searchEmailInsertStmt,":subject2");

...    
sqlite3_bind_text(searchEmailInsertStmt, subjectIndex, [subject UTF8String], -1, SQLITE_TRANSIENT);        // subject
sqlite3_bind_text(searchEmailInsertStmt, subjectIndex2, [subjectCopy UTF8String], -1, SQLITE_TRANSIENT);        // search_email.subject


if (sqlite3_step(searchEmailInsertStmt) != SQLITE_DONE) {
    NSLog(@"Failed step in searchEmailInsertStmt: '%s', '%i'", sqlite3_errmsg([[AddEmailDBAccessor sharedManager] database]), pk);
}

then it crashes with the following error:
malloc: *** error for object 0x9b6350: pointer being freed was not allocated
*** set a breakpoint in malloc_error_break to debug

any idea why?

update:
if i replace subject and subjectCopy with string constants ie @”subject1″ and @”subject2″ it works just fine.. but for some reason i tried duplicating this programmatically ie

NSString* subjectCopy = [NSString alloc];
subjectCopy = [subject retain];

and none of those work.. also changing SQLITE_TRANSIENT to SQLITE_STATIC has no effect.

update 2: output of bt after breaking at malloc_error_break:

thread #6: tid = 0x2503, 0x99a20815 libsystem_c.dylib`malloc_error_break, stop reason = breakpoint 1.1
frame #0: 0x99a20815 libsystem_c.dylib`malloc_error_break
frame #1: 0x99a21d51 libsystem_c.dylib`free + 346
frame #2: 0x0005d5e8 reMail`sqlite3MemFree + 40 at sqlite3.c:12272
frame #3: 0x0002a53e reMail`sqlite3_free + 126 at sqlite3.c:15653
frame #4: 0x0004e670 reMail`sqlite3Fts3ExprFree + 64 at sqlite3.c:101490
frame #5: 0x0004e665 reMail`sqlite3Fts3ExprFree + 53 at sqlite3.c:101489
frame #6: 0x0003fbf1 reMail`fulltextClose + 49 at sqlite3.c:97401
frame #7: 0x000b48f3 reMail`sqlite3VdbeFreeCursor + 163 at sqlite3.c:47461
frame #8: 0x000aebb8 reMail`sqlite3VdbeExec + 17576 at sqlite3.c:54042
frame #9: 0x00032273 reMail`sqlite3Step + 467 at sqlite3.c:49459
frame #10: 0x00031f5e reMail`sqlite3_step + 78 at sqlite3.c:49531
frame #11: 0x000ff2ae reMail`-[EmailProcessor insertIntoSearch:withMetaString:withUid:withSubject:withBody:withFrom:withTo:withCc:withFolder:] + 1854 at EmailProcessor.m:934
frame #12: 0x001005a1 reMail`-[EmailProcessor addEmail:] + 3153 at EmailProcessor.m:1015
frame #13: 0x000fd673 reMail`-[EmailProcessor addEmailWrapper:] + 4035 at EmailProcessor.m:651
frame #14: 0x0324c1bd CoreFoundation`__invoking___ + 29
frame #15: 0x0324c0d6 CoreFoundation`-[NSInvocation invoke] + 342
frame #16: 0x017c36b5 Foundation`-[NSInvocationOperation main] + 45
frame #17: 0x01738d23 Foundation`-[__NSOperationInternal start] + 736
frame #18: 0x01738a34 Foundation`-[NSOperation start] + 79
frame #19: 0x017c5301 Foundation`__block_global_6 + 150
frame #20: 0x02ec053f libdispatch.dylib`_dispatch_call_block_and_release + 15
frame #21: 0x02ed2014 libdispatch.dylib`_dispatch_client_callout + 14
frame #22: 0x02ec32e8 libdispatch.dylib`_dispatch_root_queue_drain + 335
frame #23: 0x02ec3450 libdispatch.dylib`_dispatch_worker_thread2 + 39
frame #24: 0x99a09e12 libsystem_c.dylib`_pthread_wqthread + 441

Answer 1

I finally found it! after chasing so many red herrings.. the best advice i got was on a forum specialized for sqlite: the point was to keep the guts of sqlite out of my debugging reach, it’s extremely unlikely that it’s sqlite’s fault.

i basically decided to break my sql statement into smaller pieces and run each on it’s own:

original sql statement that caused the memory problem:

    NSString* updateStmt = @"INSERT INTO search_email(docid, meta, subject, body, sender, tos, ccs, folder, threadid)"
    " SELECT ?, ?, ?, ?, ?, ?, ?, ?,"
    " coalesce((SELECT search_email.threadID "
    " FROM search_email "
    " WHERE search_email.subject MATCH ?  UNION SELECT * FROM "
    " (SELECT threadID FROM  (SELECT threadID FROM search_email WHERE search_email.sender MATCH ? "
    " INTERSECT SELECT threadID FROM search_email WHERE search_email.tos MATCH ? ) "
    "  UNION "
    " SELECT threadID FROM (SELECT threadID FROM search_email WHERE search_email.sender MATCH ? "
    "       INTERSECT SELECT threadID FROM search_email WHERE search_email.tos MATCH ?)) "
    " LIMIT 1"
    " ),"
    " ?"
    " )";

the problem happened whenever i supplied strings with special characters to match..
the following are examples of problematic parameters sent to MATCH:

sabaho 🙂
New core-audio questions for Feb 1 – Stack Exchange
Ref; Data Centric testing/ETL Tester oppurtunity at Chicago, IL.

so to go around that i simply replaced MATCH with a normal = comparison.. but cleaned up the parameter using regex first:

NSError *error = NULL;
NSRegularExpression *regex = [NSRegularExpression regularExpressionWithPattern:@"re:(\\s)*"
                                                                       options:NSRegularExpressionCaseInsensitive
                                                                         error:&error];


                            error:&error];

NSString *filteredSubjectFromRe = [regex stringByReplacingMatchesInString:subject
                                                                  options:0
                                                                    range:NSMakeRange(0, [subject length])
                                                             withTemplate:@""];

if(searchEmailInsertStmt == nil) {

    NSString* updateStmt = @"INSERT INTO search_email(docid, meta, subject, body, sender, tos, ccs, folder, threadid)"
    " SELECT ?, ?, ?, ?, ?, ?, ?, ?,"
    " coalesce((SELECT search_email.threadID "
    " FROM search_email "
    " WHERE search_email.subject = ? UNION SELECT * FROM "
    " (SELECT threadID FROM  (SELECT threadID FROM search_email WHERE search_email.sender = ? "
    " INTERSECT SELECT threadID FROM search_email WHERE search_email.tos = ? ) "
    "  UNION "
    " SELECT threadID FROM (SELECT threadID FROM search_email WHERE search_email.sender = ? "
    "       INTERSECT SELECT threadID FROM search_email WHERE search_email.tos = ?)) "
    " LIMIT 1"
    " ),"
    " ?"
    " )";