Question

 def append(msg: Msg, session: OutputChannel[Any]) {
    changeSize(1) // size always increases by 1
    val el = new MQueueElement(msg, session)

    if (isEmpty) first = el
    else last.next = el

    last = el
  }

the append method of the MQueue(message queue of the actor) have no maximum size. Won’t this cause outOfMemory?

And look into the changeSize(1)

private var _size = 0

  def size = _size
  final def isEmpty = last eq null

  protected def changeSize(diff: Int) {
    _size += diff
  }

why no @volatile with private var _size ?
what if append times exceed the Int.maxValue?
do we just expect those will never happend?

Answer 1

For the first part of your question: yes, see also this related question Actors Mailbox Overflow. Scala

I think the _size variable is not marked as volatile because it is expected here that the calling method take care of the synchronization. I briefly scanned through the code and various parts of the actor library that call this method are indeed marked as synchronized.
For the integer overflow: I guess it is indeed expected that this will never happen.
The actor library that is most used is Akka, which does have support for bounded mailboxes, see this link for semantics and configuration. Apart from that they will replace/be merged with the actors library in the scala distribution: as discussed recently on the mailing list.