The most important insight during that discussion of dreams came from Jessica's son, who informed us of a perhaps common childhood belief -- namely, that a child will have fewer bad dreams when he becomes a grownup.
You seem to be pessimistic about the development of a theory that will answer all questions regarding human nature in one fell swoop. There is of course good reason for that. It's not the way human advances are made. Advances arise incrementally, with steps forward being followed with backward steps and leaps into new unexpected directions. That is how it has been in the physical sciences, and it undoubtedly will be true with psychology as well.
Long ago, during a trial, the lawyer Clarence Darrow stated the case better than I ever could. During summary remarks he said, "I have done the best I could through many years to search for truth. Sometimes I have thought I had a gleam of truth; sometimes I felt that I had in my hands the truth, a truth that could not be disputed, but that would be true forever. Sometimes I thought I had found it; and then again I thought I had lost it; and the truth I so fondly held in my hands was only an empty dream, and not the truth that all; and I have searched again and again, and here I find it and there I lose it; and I expect it will be this way until the end. It is not given to man to be sure of the truth. There are no standards, there are no measures; everything is dumped in on his imperfect brain. He weighs it the best he can and finds out the best way he can whether it is true or false; and he never knows."
This is the way of all searches for truth, and there is no predicting the course of advancement. That is why Darrow went on to say, "Therefore, gentlemen, above everything else on earth, men should cling fast to their right to examine every question; to listen to everyone, no matter who he is; to hear the spoken words and read the written words; because if you shut men's mouths and paralyzed their minds, then the greatest truth that is necessary for the welfare of the human race may die." In other words, the researcher with the biggest grant and the most prestigious academic digs will not necessarily be the one to make the next telling breakthrough.
As for the "black hole" of dream analysis, I don't see advances being made within a clinical context. The reason relates to the fact, as you state, that therapists don't get to know the whole person because they must focus on the "problem" to the exclusion of everything else. You cannot analyze dreams that way because dreams address the whole person.
It is nevertheless possible to be systematic in studying dreams by demanding internal consistency in one's approach. That does not guarantee correctness, but it does help avoid black-hole subjectivity. For instance, the assumption that REM dreams are rational creations designed to help us meet our present life situations carries with it a host of logical consequences, all of which must be incorporated in one's interpretation of such dreams for the attempts at analysis to have any chance of being correct.
Let me give you a simple example. The notion that a REM dream has the purpose of helping us to rationally meet our life situations in the present means that they must somehow bring the lessons derived from similar situations in the past to bear on a person's analysis of the present situations. The past has nothing else to offer.
What it means in practice is that dream interpretation should begin by listing a complete set of the unresolved situations a person is facing in his present life. Associations from the dream should then be gathered with an eye toward the lessons derived from each past situation. Next, present and past life situations should be matched on the basis of similarity, with the lessons derived from each past situation encapsulating the wisdom that the dream offers in helping the person to successfully meet the challenges associated with the corresponding present situation. That's the way dreams should be interpreted, according to the assumptions made. No one but me has ever tried. I have news--it works.
Such an approach of course does not do away with subjective interpretation. How can this subjectivity be handled? The idea that a REM dream experience is physiologically experienced in a literal sense helps. Without going into details, it means that a REM dream should be open to analysis on a moment-to-moment basis from the point of view of REM dream structure. To accomplish this, one must make some suppositions, but here again the field is limited by the assumption of real-life experience. Further, the interpretation of the dream that flows from such a moment-to-moment analysis must jive with that derived by the method outlined above.
Such a program has been carried out completely with respect to Freud's Irma dream and partially with respect to Freud's other dreams from his "The Interpretation of Dreams," with the latter being the result of limited information. Even if all of Freud's dreams could be analyzed completely in this way, would it necessarily mean that REM dreams were completely understood? Of course not. But to pooh pooh the accomplishment as nothing would be wrong too.
You seem to be waiting for a guaranteed-by-God complete answer. I sympathize with the tendency; however, please recognize that the rest of us must grub around the best we can with the information at hand.