Question

set.seed(1234) 
mydata <- data.frame (
 individual = factor(1:10), 
 M1a = factor (sample (c(1,2),10, replace = T)),
 M1b = factor (sample (c(1,2),10, replace = T)), 
  pop = factor (c(rep(1, 5), rep (2, 5))), 
 yld = rnorm(10, 10, 2))

Here M1a, M1b are fixed however individual is random.

    require(lme4)
    model1 <- lmer(yld ~  M1a + M1b + pop + (1|individual), data = mydata)
    model1
     Error in function (fr, FL, start, REML, verbose)  : 
      Number of levels of a grouping factor for the random effects
    must be less than the number of observations    

Can we do this in lme4. These are known as animal model and asrmel can do some of such things (link).

EDITS: I forget to mention the relationship matrix is rquired. The following is pedigree structure to do so. To fit the example to size, I reduce the sample size to 10.

 peddf <- data.frame (individual = factor(1:10), 
   mother = c(NA, NA, NA, 1, 1, 1, 1,3, 3,3), 
    father = c(NA, NA, NA, 2, 2, 2, 2, 2, 2, 2))

  individual mother father
1           1     NA     NA
2           2     NA     NA
3           3     NA     NA
4           4      1      2
5           5      1      2
6           6      1      2
7           7      1      2
8           8      3      2
9           9      3      2
10         10      3      2

In term of matrix is following (only lowerhalf triangle plus diagnonal is shown):

1   NA  NA  NA  NA  NA  NA  NA  NA  NA
0   1   NA  NA  NA  NA  NA  NA  NA  NA
0   0   1   NA  NA  NA  NA  NA  NA  NA
0.25    0.25    0   1   NA  NA  NA  NA  NA  NA
0.25    0.25    0   0.25    1   NA  NA  NA  NA  NA
0.25    0.25    0   0.25    0.25    1   NA  NA  NA  NA
0.25    0.25    0   0.25    0.25    0.25    1   NA  NA  NA
0   0.25    0.25    0.125   0.125   0.125   0.125   1   NA  NA
0   0.25    0.25    0.125   0.125   0.125   0.125   0.25    1   NA
0   0.25    0.25    0.125   0.125   0.125   0.125   0.25    0.25    1

In picture form:

Answer 1

Try the kinship package, which is based on nlme. See this thread on r-sig-mixed-models for details.

For non-normal responses, you’d need to modify lme4 and the pedigreemm package; see this question for details.