Home/ Questions/Q 7965639
In Process

The Archive Base Latest Questions

Editorial Team
  • 0
Asked: 2026-06-04T06:12:57+00:00

I’m using the ff and ffbase packages to combine two ffdf objects but when

  • 0

I’m using the ff and ffbase packages to combine two ffdf objects but when I use the merge statement it goes from 1 million lines in the target ffdf to 8 million lines.

ffdf1 is 1 million rows by 6 columns:

> summary(ffdf2)
              Length  Class     Mode
userid        1000000 ff_vector list
V2            1000000 ff_vector list
V3            1000000 ff_vector list
V4            1000000 ff_vector list
V5            1000000 ff_vector list
V6            1000000 ff_vector list

ffdf2 is ~20 million rows X 3 columns like this:

  userid      gender      age
    1         1          3
    2         1          2
    3         2          5
    4         0          4
    5         2          3
    ...       ...       ...

I use the following code to merge the two:

ffdf3 <- merge(ffdf1, ffdf2, by.x="userid",by.y="userid",all.x=T,sort=F)

The result is this:

> summary(ffdf3)
                  Length  Class     Mode
    userid        8000000 ff_vector list
    V2            8000000 ff_vector list
    V3            8000000 ff_vector list
    V4            8000000 ff_vector list
    V5            8000000 ff_vector list
    V6            8000000 ff_vector list
    gender        8000000 ff_vector list
    age           8000000 ff_vector list

Any ideas why the length is going from 1mm to 8mm?

EDIT:

When I try this:

ffdf3 <- merge(ffdf1, ffdf2, by.x="userid",by.y="userid",all.x=F,sort=F)

I get:

> summary(ffdf3)
                      Length  Class     Mode
        userid        740383 ff_vector list
        V2            740383 ff_vector list
        V3            740383 ff_vector list
        V4            740383 ff_vector list
        V5            740383 ff_vector list
        V6            740383 ff_vector list
        gender        740383 ff_vector list
        age           740383 ff_vector list

Also here is the output from running the merge:

2012-05-13 14:49:06, x has 2 chunks, y has 8 chunks
2012-05-13 14:49:06, working on x chunk 1:500000
2012-05-13 14:49:07, working on y chunk 1:2958661
2012-05-13 14:49:16, working on y chunk 2958662:5917322
2012-05-13 14:49:32, working on y chunk 5917323:8875983
2012-05-13 14:49:45, working on y chunk 8875984:11834644
2012-05-13 14:49:57, working on y chunk 11834645:14793305
2012-05-13 14:50:09, working on y chunk 14793306:17751966
2012-05-13 14:50:20, working on y chunk 17751967:20710627
2012-05-13 14:50:30, working on y chunk 20710628:23669283
2012-05-13 14:50:40, working on x chunk 500001:1000000
2012-05-13 14:50:41, working on y chunk 1:2958661
2012-05-13 14:50:52, working on y chunk 2958662:5917322
2012-05-13 14:51:03, working on y chunk 5917323:8875983
2012-05-13 14:51:14, working on y chunk 8875984:11834644
2012-05-13 14:51:24, working on y chunk 11834645:14793305
2012-05-13 14:51:36, working on y chunk 14793306:17751966
2012-05-13 14:51:47, working on y chunk 17751967:20710627
2012-05-13 14:51:58, working on y chunk 20710628:23669283

Also ffdf1 contains 677840 unique userid‘s. So there are some duplicates in the 1mm rows.

Share

Leave an answer

You must login to add an answer.

Need An Account,

1 Answer

  1. Editorial Team

    The merge.ffdf contains a bug and currently only allows to do correctly inner joins and not all.x=TRUE and all.y=FALSE. The function is worked upon @ http://code.google.com/p/fffunctions/. Problem is that when you don’t have a matching record, when doing a left outer join, the vmodes need to be changes in order to correctly allow NA’s. This is being worked upon.

    FYI. This is now solved in the development version at http://code.google.com/p/fffunctions/ and will be uploaded to CRAN in the coming weeks.

    • 0