Question

As suggested in answers to a previous question, I tried using Erlang proplists to implement a prefix trie.

The code seems to work decently well… But, for some reason, it doesn’t play well with the interactive shell. When I try to run it, the shell hangs:

> Trie = trie:from_dict(). % Creates a trie from a dictionary
% ... the trie is printed ...
% Then nothing happens

I see the new trie printed to the screen (ie, the call to trie:from_dict() has returned), then the shell just hangs. No new > prompt comes up and ^g doesn’t do anything (but ^c will eventually kill it off).

With a very small dictionary (the first 50 lines of /usr/share/dict/words), the hang only lasts a second or two (and the trie is built almost instantly)… But it seems to grow exponentially with the size of the dictionary (100 words takes 5 or 10 seconds, I haven’t had the patience to try larger wordlists). Also, as the shell is hanging, I notice that the beam.smp process starts eating up a lot of memory (somewhere between 1 and 2 gigs).

So, is there anything obvious that could be causing this shell hang and incredible memory usage?

---

Some various comments:

• I have a hunch that the garbage collector is at fault, but I don’t know how to profile or create an experiment to test that.

• I’ve tried profiling with eprof and nothing obvious showed up.

• Here is my “add string to trie” function:

add([], Trie) ->
    [ stop | Trie ];

add([Ch|Rest], Trie) ->
    SubTrie = proplists:get_value(Ch, Trie, []),
    NewSubTrie = add(Rest, SubTrie),
    NewTrie = [ { Ch, NewSubTrie } | Trie ],
    % Arbitrarily decide to compress key/value list once it gets
    % more than 60 pairs.
    if length(NewTrie) > 60 ->
            proplists:compact(NewTrie);
        true ->
            NewTrie
    end.

Answer 1

The problem is (amongst others ? — see my comment) that you are always adding a new {Ch, NewSubTrie} tuple to your proplist Tries, no matter if Ch already existed, or not.

Instead of

NewTrie = [ { Ch, NewSubTrie } | Trie ]

you need something like:

NewTrie = lists:keystore(Ch, 1, Trie, {Ch, NewSubTrie})