Question

Below is an example of choosing a set number of random permutations from a dataset, how can I create a new dataset with the remaainder. For example below I choose 49402 (roughly 10%) and create a dataset named UnseenTestdata after this is chosen I want the remainder to go into a new dataset called testdata.

pointsToPick = 49402;  %# Numbers to pick
rVec = randperm(494021);   %# Random permutation of datapoint indices (N=494021 in this case)  

UnseenTestdata = fulldata(rVec(1:pointsToPick),:); %# Random sample

Unseentestdata minus fulldata = remainder of the dataset aptly named testdata.

Dimensions of fulldata set is 494021×6 of which I choose at random 49402×6 from fulldata. I then need to get whats left from fulldata minus the unseentestdata.

Barnabas Szabolcs added a test case answer of:

fulldata = [1 2; 3 4; 5 6; 7 8];
rVec = randperm(4);  
pointsToPick=2;
unseen = fulldata(rVec(1:pointsToPick),:); 
testdata = fulldata(rVec(pointsToPick:length(rVec)),:); 

However this does not work, I have screen dumped the results:

If you notice in the screen dump unseen data = 3,4 and 7,8 however if you notice in testdata 7,8 remain.

If fulldata =

1,2
3,4
5,6
7,8

And we choose two random rows in this case the rows in unseen are:

row
3,4
7,8

Then whatever remains should be:

1,2
5,6

However if you notice in the sreen dump from the example test testdata has the row:

7,8

showing that the example test does not work.

Answer 1

If I understand your question correctly, the solution is

testdata = fulldata(rVec((pointsToPick+1):length(rVec)),:);

Simple test case:

fulldata = [1 2; 3 4; 5 6; 7 8;10 9];
rVec = randperm(4);  // gives me first time [4 2 3 1 5]
pointsToPick=2;
unseen = fulldata(rVec(1:pointsToPick),:); // [7 8; 3 4]
// length(rVec) is 5
testdata = fulldata(rVec((pointsToPick+1):length(rVec)),:); // [5 6; 1 2; 10 9]

you can clearly see that in a sense fulldata = unseen (setplus) testdata.
Note that we need “+1” because arrays are indexed from one up unlike say in c++, so the last index is length not length-1.

You can verify if things are correct using this:

 isequal(sort([unseen; test]), sort(full_data)) // should be true