Home/ Questions/Q 8218625
In Process

The Archive Base Latest Questions

Editorial Team
  • 0
Asked: 2026-06-07T12:51:50+00:00

I have a question on Pig when performing what seems like two levels of

  • 0

I have a question on Pig when performing what seems like two levels of groupings. As an example, let’s say I had some example input data like:

email_id:chararray    from:chararray        to:bag{recipients:tuple(recipient:chararray)}
e1                    user1@example.com     {(friend1@example.com),(friend2@example.com),(friend3@myusers.com)}
e2                    user1@example.com     {(friend1@example.com),(friend4@example.com)}
e3                    user1@example.com     {(friend5@example.com)}
e4                    user2@example.com     {(friend2@example.com),(friend4@example.com)}

So each line is an email from a user “from” to user(s) “to”.

And I ultimately want a list of all senders and all the people they’ve sent emails to, including the # of emails sent for each person, sorted from highest to lowest, for example:

user1@example.com     {(friend1@example.com, 2), (friend2@example.com, 1), (friend3@example.com, 1), (friend4@example.com, 1), (friend5@example.com, 1)}
user2@example.com     {(friend2@example.com, 1), (friend4@example.com, 1)}

Ideas on the best way to tackle this in Pig would be appreciated!

Share

Leave an answer

You must login to add an answer.

Need An Account,

1 Answer

  1. Editorial Team

    Here is one version of the script:

    inpt = load '/pig_data/pig_fun/input/from_senders.txt' as (email_id:chararray, from:chararray, to:bag{recipients:tuple(recipient:chararray)});

pivot = foreach inpt generate from, FLATTEN(to);
pivot = foreach pivot generate from, to::recipient as recipient;
dump pivot;
/*
(user1@example.com,friend1@example.com)
(user1@example.com,friend2@example.com)
(user1@example.com,friend3@myusers.com)
(user1@example.com,friend1@example.com)
(user1@example.com,friend4@example.com)
(user1@example.com,friend5@example.com)
(user2@example.com,friend2@example.com)
(user2@example.com,friend4@example.com)
*/

grp = group pivot by (from, recipient);
with_count = foreach grp generate FLATTEN(group), COUNT(pivot) as count;
dump with_count;
/*
(user1@example.com,friend1@example.com,2)
(user1@example.com,friend2@example.com,1)
(user1@example.com,friend3@myusers.com,1)
(user1@example.com,friend4@example.com,1)
(user1@example.com,friend5@example.com,1)
(user2@example.com,friend2@example.com,1)
(user2@example.com,friend4@example.com,1)
*/

to_bag = group with_count by from;
result = foreach to_bag {
    order_by_count = order with_count by count desc;
    generate group as from, order_by_count.(recipient, count);
};
dump result;
/*
(user1@example.com,{(friend1@example.com,2),(friend2@example.com,1),(friend3@myusers.com,1),(friend4@example.com,1),(friend5@example.com,1)})
(user2@example.com,{(friend2@example.com,1),(friend4@example.com,1)})
*/

    Hope it helps.

    • 0