Question

Note(update): No need to read this entire wall of text.. problem + solution further below.

I would like to, once and for all, find out why I am running into these errors all of the time when creating multithreaded applications. I’m cutting a lot of code out to keep it simple so it won’t work if you try to compile it.

This is a queueing system. Entities will be put in groups of 5.
If, by example, an Entity leaves group 3, group 3 will snatch an entity from another group that has less priority and is not full yet.

All I’m doing at the moment is create entities and queue them, which happens on the main thread. After that, I start a boost thread that will remove 20 entities. This is where the read access violation occurs and I can’t figure out what’s causing it. How do you debug something like this?

main.h

//Externs, so all files can access the objects. Created in main.cpp .

#include "queue.h"
typedef std::map<unsigned int, Entity*> entityMap; //id,entity
extern entityMap entitymap;
typedef std::map<unsigned int, unsigned int> entityPositionMap; //Position,entityID
extern entityPositionMap entitypositionmap;
extern LFDGroups lfdgroups;
extern Queue queue;

main.cpp

typedef std::map<unsigned int, Entity*> entityMap;
entityMap entitymap;
typedef std::map<unsigned int, unsigned int> entityPositionMap; //Position,id
entityPositionMap entitypositionmap;
LFDGroups lfdgroups;
Queue queue;

struct threadStarter2
{void operator()(){
    for(unsigned int i = 0;i<20;i++) queue.removeFakeEntity();
}};

int main()
{
    threadStarter1 startit1;
    boost::thread thrd1(startit1);

    whatNext(); //Prevent program exit

    return 0;
}

queue.cpp

#include "main.h"
#include "queue.h"

Queue::Queue() //Lets add some entities and queue them at application start
{
    for(unsigned int i = 0;i<20;i++){ addFakeEntity(false); }

    //Queue all entities
    std::map<unsigned int, Entity*>::iterator p;
    for(p = entitymap.begin(); p != entitymap.end(); p++) 
    {
        boost::mutex::scoped_lock lock(lfdgroups.access_groupsLookingForMore);
        lfdgroups.addMember(p->second->id);
        lock.unlock();
    }
}

void Queue::addFakeEntity(bool queuenow)
{

    Entity *entity = new Entity; 
    entity->id = entitymap.size();
    entity->LFDGroupID = NULL;
    entity->LFD = false;
    entitymap.insert(std::pair<unsigned int,Entity*>(entity->id,entity)); 

    if(queuenow)
    {
        entity->LFD = true;
        boost::mutex::scoped_lock lock(lfdgroups.access_groupsLookingForMore);
        lfdgroups.addMember(entity->id);
        lock.unlock();
    }
}

void Queue::removeFakeEntity()
{
    //Remove random entity from random group
    unsigned int groupID = getrand(0,lfdgroups.groupHeap.size()-1);
    boost::mutex::scoped_lock lock(lfdgroups.access_groupsLookingForMore); //Thread safety
    lfdgroups.groupHeap[groupID]->removeMember(lfdgroups.groupHeap[groupID]->arrayOfEntityIDs[getrand(0,lfdgroups.groupHeap[groupID]->arrayOfEntityIDs.size()-1)],true);
    lock.unlock();
}

So as you can see, the main thread isn’t doing anything at the moment and the boost thread is removing entities one by one.

Here’s what actually happens when an entitity is removed (lfdgroups.groupHeap[groupID]->removeMember())

bool groupObject::removeMember(unsigned int entityID,bool snatchIt)
{
    boost::mutex::scoped_lock lock(access_group);
    for (unsigned int i = 0;i<arrayOfEntityIDs.size();i++)
    {
        if(arrayOfEntityIDs[i] == entityID) //Messy way of selecting the entity we are looking for.
        {
            std::stringstream ss; 
            ss.str(""); ss.clear(); ss << "Removed member " << entityID << " from group " << id << " which has " << groupSize() - 1 << " members left." << std::endl;
            echo(ss.str());
            arrayOfEntityIDs.erase(arrayOfEntityIDs.begin() + i);
            if(snatchIt){ std::cout << "We have to snatch" << std::endl; snatch();}
            else{ std::cout << "We don't have to snatch" << std::endl;}
            return true;
        }
    }
    lock.unlock();
    return true;
}

snatchit is set to true, so;

void groupObject::snatch()
{
    boost::mutex::scoped_lock lock(snatching);

    groupObject* thisgroup = NULL;
    thisgroup = this;
    if(!thisgroup) return;

    std::cout << thisgroup->id << " need snatch? " << thisgroup->snatchCriteria() << std::endl;
    if(!thisgroup->snatchCriteria()) return; //Do we even need to snatch a player (group < 5?) //This is a little redundant atm

    std::map<unsigned int, unsigned int>::iterator p2;
    for(p2 = entitypositionmap.begin(); p2 != entitypositionmap.end(); p2++) 
    {
        Entity* entity = thisgroup->getEntity(p2->second);
        if(entity != NULL && entity->LFDGroupID != thisgroup->id)
        {
            groupObject* targetgroup = NULL;
            targetgroup = lfdgroups.getGroupObject(entity->LFDGroupID);
            if(targetgroup != NULL && targetgroup->migrateCriteria())
            {
                lfdgroups.getGroupObject(thisgroup->id)->addMember(entity->id,false);

                std::stringstream ss; 
                ss.str(""); ss.clear(); ss << "Snatched " << entity->id << " from " << targetgroup->id << " for: " << thisgroup->id << std::endl;
                echo(ss.str());
                break;
            }
        }
    }
    lock.unlock();
}

What just happened was that the group went over all the entities in the queue until it found one elegible to snatch. After which lfdgroups.getGroupObject(thisgroup->id)->addMember(entity->id,false); takes place (so addmemeber in the group object, not to be confused with lfdgroups’ addmember).

bool groupObject::addMember(unsigned int entityID,bool snatchIt)
{
    Entity* entity = getEntity(entityID);
    if(entity == NULL) return false;

    groupObject* group = lfdgroups.getGroupObject(entity->LFDGroupID);

    if(group != NULL){ if(!group->removeMember(entityID,snatchIt)){return false;} }

    if(elegibleCheck(entityID))
    {
        arrayOfEntityIDs.push_back(entityID);
        entity->LFDGroupID = id;

        std::stringstream ss; 
        ss.str(""); ss.clear(); ss << "Added member " << entityID << " to group " << id << " which has " << groupSize() << " members." << std::endl;
        echo(ss.str());

        return true;
    }
    return false;
}

So we just removed the entity from it’s group with snatchIt set to false to prevent a loop. After that we simply updated the entity->LFDGroupID to the current group’s id and that would be the end of it.

Sorry for the load of code.. not all of it is relevant but it allows you to follow the path the removing-an-entity function takes. My main question is; how do I go about quickly debugging access violations?

When the application crashes, the breakpoint ends up in a file called ‘vector’

size_type size() const
    {   // return length of sequence
    return (this->_Mylast - this->_Myfirst); //Breakpoint here.
    }

UPDATE(Solution):

I started by commenting out the various steps the Queue::removeFakeEntity() function takes to figure out where it all goes wrong. I immediately figured out the problem was with the line lfdgroups.groupHeap[groupID]->removeMember(lfdgroups.groupHeap[groupID]->arrayOfEntityIDs[getrand(0,lfdgroups.groupHeap[groupID]->arrayOfEntityIDs.size()-1)],true);

So I cut it into pieces

unsigned int entityID = lfdgroups.groupHeap[groupID]->arrayOfEntityIDs[getrand(0,lfdgroups.groupHeap[groupID]->arrayOfEntityIDs.size()-1)];
lfdgroups.groupHeap[groupID]->removeMember(entityID,true);

The error still occured. The breakpoint occured at a vector size return, so I isolated that;

void Queue::removeFakeEntity()
{
    std::cout << "size: " << lfdgroups.groupHeap[groupID]->arrayOfEntityIDs.size() << std::endl;
}

The error still occured. So I tried putting in the groupID manually.

void Queue::removeFakeEntity()
{
    std::cout << "size: " << lfdgroups.groupHeap[1]->arrayOfEntityIDs.size() << std::endl;
    std::cout << "size: " << lfdgroups.groupHeap[2]->arrayOfEntityIDs.size() << std::endl;
    std::cout << "size: " << lfdgroups.groupHeap[4]->arrayOfEntityIDs.size() << std::endl;
}

Works just fine! It had to be a problem with the groupID.

void Queue::removeFakeEntity()
{
    std::cout << "Heap size: " << lfdgroups.groupHeap.size() << std::endl;
}

That’s odd, it shows ‘4’. But wait a second! Counting the first element (0) it should be 5, right?

Wrong, I started counting groupIDs at 1 because I wanted to reserve 0 as a return value. I should have just gone for NULL. I even thought about it when I made the decision yesterday. “I’m going to forget I did that!”. Why don’t I ever listen to myself?

So the problem was with unsigned int groupID = getrand(0,lfdgroups.groupHeap.size()-1); which, ironically, is the very first line of the function I was having issues with..

Answer 1

tl;dr

But I bet your problem comes from multiple dealocation, even if you are using locks.

My suggestion is to replace

typedef std::map<unsigned int, Entity*> entityMap;

with

typedef std::map<unsigned int, EntityPtr> entityMap;

where EntityPtr is a smart pointer (you can use Boost‘s, since you’re already using it).

You’re probably freeing the same memory in separate threads.