Home/ Questions/Q 9311043
In Process

The Archive Base Latest Questions

Editorial Team
  • 0
Asked: 2026-06-19T01:15:56+00:00

I have large mbox files and I am using third party API like mstor

  • 0

I have large mbox files and I am using third party API like mstor to parse messages from mbox file using hadoop. I have uploaded those files in hdfs. But the problem is that this API uses only local file system path , similar to shown below

MessageStoreApi store = new MessageStoreApi(“file location in locl file system”);

I could not find a constructor in this API that would initialize from stream . So I cannot read hdfs stream and initialize it.

Now my question is, should I copy my files from hdfs to local file system and initialize it from local temporary folder? As thats what I have been doing for now:

Currently My Map function receives path of the mbox files.

Map(key=path_of_mbox_file in_hdfs, value=null){

String local_temp_file = CopyToLocalFile(path in hdfs);
MessageStoreApi store = new MessageStoreApi(“local_temp_file”);
//process file

}

Or Is there some other solution? I am expecting some solution like what If I increase the block-size so that single file fits in one block and somehow if I can get the location of those blocks in my map function, as mostly map functions will execute on the same node where those blocks are stored then I may not have to always download to local file system? But I am not sure if that will always work 🙂

Suggestions , comments are welcome!

Share

Leave an answer

You must login to add an answer.

Need An Account,

1 Answer

  1. Editorial Team

    For local filesystem path-like access, HDFS offers two options: HDFS NFS (via NFSv3 mounts) and FUSE-mounted HDFS.

    The former is documented under the Apache Hadoop docs (CDH users may follow this instead)

    The latter is documented at the Apache Hadoop wiki (CDH users may find relevant docs here instead)

    The NFS feature is more maintained upstream than the FUSE option, currently.

    • 0