Question

I have many many files in a folder, and I want to process them one after another, I need to have a global dictionary to record the user identifier and the flowcount, but if my code is like this, when the second or third file is processed, the user_dict for last file will lose.

Because if an user id in the second file is as same as in the first file, then if should be assigned the same flowcount instead of a new one, how can I make one dictionary keep growing when open files one by one?

 for line in fd.readlines():
    obj = json.loads(line)
    user = obj["host_dst"]["addr"] + '_' + str(obj["host_dst"]["port"])
    if user not in user_dict:
       user_dict[user] = []
    user_dict[user].append(obj["params"]["flowcount"])

Since the size of each file is very big, I merged them all into one file, then ran the script to process, the computer will kill the process after a while, I have to process them one by one instead

Answer 1

You can open multiple files in your python script, and use your for loop to take care each of them

for filename in os.listdir(folderpath):
    filepath = os.path.join(folderpath, filename)
    fd = open(filepath, 'r')

    # here is your code
    for line in fd.readlines():
        ....