Question

The problem I am having is with trying to find an efficient way to find swappable elements in a matrix in order to implement a swap algorithm for null model creation.

The matrix consists of 0’s and 1’s and the idea is that elements can be switched between columns so that the row and column totals of the matrix remain the same.

For example, given the following matrix:

   c1 c2 c3 c4
r1  0  1  0  0 = 1
r2  1  0  0  1 = 2
r3  0  0  0  0 = 0
r4  1  1  1  1 = 4
   ------------
    2  2  1  2

columns c2 and c4 in r1 and r2 can each be swapped in such a way that totals are not altered i.e.:

   c1 c2 c3 c4
r1  0  0  0  1 = 1
r2  1  1  0  0 = 2
r3  0  0  0  0 = 0
r4  1  1  1  1 = 4
   ------------
    2  2  1  2

This all needs to be done randomly so as not to introduce any bias.

I have one solution that works. I randomly select a row and two columns. If they yield a 10 or 01 pattern then I randomly select another row and check the same columns to see if they yield the opposite pattern. If either of them fail I start over and select a new element.

This method works but I only “hit” the correct patterns about 10% of the time. In a large matrix or in one with few 1’s in the rows I waste a lot of time “missing”. I figured that there had to be a more intelligent way of choosing elements in the matrix but still doing it randomly.

The code for the working method is:

def isSwappable(matrix: Matrix): Tuple2[Tuple2[Int, Int], Tuple2[Int, Int]] = {
  val indices = getRowAndColIndices(matrix)

  (matrix(indices._1._1)(indices._2._1), matrix(indices._1._1)(indices._2._2)) match {
    case (1, 0) => {
      if (matrix(indices._1._2)(indices._2._1) == 0 & matrix(indices._1._2)(indices._2._2) == 1) {
        indices
      }
      else {
        isSwappable(matrix)
      }
    }
    case (0, 1) => {
      if (matrix(indices._1._2)(indices._2._1) == 1 & matrix(indices._1._2)(indices._2._2) == 0) {
        indices
      }
      else {
        isSwappable(matrix)
      }
    }
    case _ => {
      isSwappable(matrix)
    }
  }
}

def getRowAndColIndices(matrix: Matrix): Tuple2[Tuple2[Int, Int], Tuple2[Int, Int]] = {
  (getNextIndex(rnd.nextInt(matrix.size), matrix.size), getNextIndex(rnd.nextInt(matrix(0).size), matrix(0).size))
}

def getNextIndex(i: Int, constraint: Int): Tuple2[Int, Int] = {
  val newIndex = rnd.nextInt(constraint)
  newIndex match {
    case `i` => getNextIndex(i, constraint)
    case _ => (i, newIndex)
  }
}

I figured a more efficient way to handle this was to remove any rows that could not be used (all 1’s or 0’s) and then choose an element randomly. From there I could filter out any columns in the row that had the same value and the choose from the remaining columns.

Once the first row and column are chosen I then filter out the rows that can not provide the required pattern and then choose from the remaining rows.

This works for the most part but the problem that I can’t figure out how to deal with is what happens when there are no columns or rows to choose from? I don’t want to loop infinitely trying to find the pattern I need and I need a way of starting over if I do get an empty list of rows or columns to choose from.

The code that I have so far that sort of works (until I get an empty list) is:

def getInformativeRowIndices(matrix: Matrix) = (
  matrix
    .zipWithIndex
    .filter(_._1.distinct.size > 1)
    .map(_._2)
    .toList
  )

def getRowsWithOppositeValueInColumn(col: Int, value: Int, matrix: Matrix) = (
  matrix
    .zipWithIndex
    .filter(_._1(col) != value)
    .map(_._2)
    .toList
  )

def getColsWithOppositeValueInSameRow(row: Int, value: Int, matrix: Matrix) = (
  matrix(row)
    .zipWithIndex
    .filter(_._1 != value)
    .map(_._2)
    .toList
  )

def process(matrix: Matrix): Tuple2[Tuple2[Int, Int], Tuple2[Int, Int]] = {
  val row1Indices = getInformativeRowIndices(matrix)
  if (row1Indices.isEmpty) sys.error("No informative rows")

  val row1 = row1Indices(rnd.nextInt(row1Indices.size))
  val col1 = rnd.nextInt(matrix(0).size)
  val colIndices = getColsWithOppositeValueInSameRow(row1, matrix(row1)(col1), matrix)
  if (colIndices.isEmpty) process(matrix)
  val col2 = colIndices(rnd.nextInt(colIndices.size))
  val row2Indices = getRowsWithOppositeValueInColumn(col1, matrix(row1)(col1), matrix)
    .intersect(getRowsWithOppositeValueInColumn(col2, matrix(row1)(col2), matrix))
  println(row2Indices)
  if (row2Indices.isEmpty) process(matrix)

  val row2 = row2Indices(rnd.nextInt(row2Indices.size))
  ((row1, row2), (col1, col2))
}

I think the recursive methods are wrong and don’t really work here. Also, I am really just trying to improve the speed of cell selection so any ideas or suggestions would be greatly appreciated.

EDIT:

I have had a chance to play with this little more and have come up with another solution but it does not seem to be much faster then just randomly choosing cells in the matrix. Also, I should add that the matrix needs to be swapped about 30000 times in succession in order for it to be considered randomised and I need to generate 5000 random matrices for each test of which I have at least another 5000 to do so performance is kind of important.

The current solution (besides random cell selection is:

1. Randomly select 2 rows from the matrix
2. subtract one row from the other and put it in an Array
3. if the new Array contains both a 1 and -1 then we can swap

The logic of the subtraction looks like this:

  0  1  0  0
- 1  0  0  1
---------------
 -1  1  0 -1

The method that does this looks like this:

 def findSwaps(matrix: Matrix, iterations: Int): Boolean = {
   var result = false

   val mtxLength = matrix.length

   val row1 = rnd.nextInt(mtxLength)
   val row2 = getNextIndex(row1, mtxLength)

   val difference = subRows(matrix(row1), matrix(row2))

   if (difference.min == -1 & difference.max == 1) {
     val zeroOne = difference.zipWithIndex.filter(_._1 == -1).map(_._2)
     val oneZero = difference.zipWithIndex.filter(_._1 == 1).map(_._2)

     val col1 = zeroOne(rnd.nextInt(zeroOne.length))
     val col2 = oneZero(rnd.nextInt(oneZero.length))

     swap(matrix, row1, row2, col1, col2)
     result = true
   }
   result
 }

The matrix row subtraction looks like this:

 def subRows(a: Array[Int], b: Array[Int]): Array[Int] = (a, b).zipped.map(_ - _)

And the actual swap looks like this:

 def swap(matrix: Matrix, row1: Int, row2: Int, col1: Int, col2: Int) = {

   val temp = (matrix(row1)(col1), matrix(row1)(col2))
   matrix(row1)(col1) = matrix(row2)(col1)
   matrix(row1)(col2) = matrix(row2)(col2)

   matrix(row2)(col1) = temp._1
   matrix(row2)(col2) = temp._2
   matrix
 }

This works much better than before in that I get have between 80% and 90% success for an attempted swap (it was only about 10% with the random cell selection) however… it is still taking about 2.5 minutes to generate 1000 randomised matrices.

Any ideas on how to improve the speed?

Answer 1

I’m going to assume the matrices are big so that storage of the order of (matrix size squared) is not viable (for reasons of either speed or memory).

If you have a sparse matrix, you can enter the index of each 1 in each column in a set (here I show the compact way to do things, but you may wish to iterate with while loops for speed):

val mtx = Array(Array(0,1,0,0),Array(1,0,0,1),Array(0,0,0,0),Array(1,1,1,1))
val cols = mtx.transpose.map(x => x.zipWithIndex.filter(_._1==1).map(_._2).toSet)

Now for each column, a later column contains compatible pairs (at least one) if and only if only the following two sets are nonempty:

def xorish(a: Set[Int], b: Set[Int]) = (a--b, b--a)

So the answer will involve computing these sets and testing whether they’re both nonempty.

Now the question is what you mean by “sample randomly”. Randomly sampling single 1,0 pairs is not the same as randomly sampling possible swaps. To see this, consider the following:

1 0       1 0
1 0       1 0
1 0       1 0
0 1       1 0
0 1       1 0
0 1       0 1

The two columns on the left have nine possible swaps. The two on the right have only five possible swaps. But if you are looking for (1,0) patterns, you will sample only three times on the left vs. five on the right; if you are looking for either (1,0) or (0,1), you will sample six and six, which again distorts the probabilities. The only way to fix this is either to not be clever, and randomly sample a second time (which in the first case will work out with a usable swap 3/5 of the time, while only 1/5 in the second), or to basically compute every possible pair for swapping (or at least how many pairs there are) and select from that predefined set.

If we want to do the latter, we note that for each pair of nonidentical columns, we can compute the two sets to swap among, and we know the sizes and the product is the total number of possibilities. In order to avoid instantiating all the possibilities, we can create

val poss = {
  for (i<-cols.indices; j <- (i+1) until cols.length) yield 
    (i, j, (cols(i)--cols(j)).toArray, (cols(j)--cols(i)).toArray)
}.filter{ case (_,_,a,b) => a.length>0 && b.length>0 }

and then count how many there are:

val cuml = poss.map{ case (_,_,a,b) => a.size*b.size }.scanLeft(0)(_ + _).toArray

Now to pick a number at random, we pick a number between 0 and cuml.last and pick out which bucket this is and which item within the bucket:

def pickItem(cuml: Array[Int], poss: Seq[(Int,Int,Array[Int],Array[Int])]) = {
  val n = util.Random.nextInt(cuml.last)
  val k = {
    val i = java.util.Arrays.binarySearch(cuml,n)
    if (i<0) -i-2 else i
  }
  val j = n - cuml(k)
  val bucket = poss(k)
  (
    bucket._1, bucket._2, 
    bucket._3(j % bucket._3.size), bucket._4(j / bucket._3.size)
  )
}

This ends up returning (c1,c2,r1,r2) selected randomly.

Now that you have the coordinates, you can create the new matrix however you wish. (Most efficient is probably to do an in-place swap of the entries, and then swap back when you want to try again.)

Note that this is only sensible for a large number of independent swaps from the same starting matrix. If you instead want to do this iteratively and maintain independence, you are probably best off doing this randomly after all unless the matrices are extremely sparse, at which point it’s worth simply storing the matrices in some standard sparse matrix format (i.e. by index of nonzero entries) and doing your manipulation on those (probably with mutable sets and an update strategy, since the consequences of a single swap are confined to about n of the entries in an n*n matrix).