Just by messing around a little it seems that the video stream is not ascii. i tested by downloading the stream. It would be insane if it was. Theres so many videos. So that couldnt be it. Youtube seems to not work with javascript disable (not counting mobile if true).
How is it being done? is it javascript magic? is the SWF running the video through a filter in realtime? (I doubt its a native filter so how is the filter compiled) its really cool. I cant imagine how this is running realtime yet it is!
MPlayer uses AALib for this type of effect. YouTube have probably ported it to ActionScript.