How important is traceable lineage compared to demonstrated competence of current leadership?
In Hakuin's own time, around the 17-18th century, the Rinzai school of Zen was in a state of decline for 300 years before he came about and revitalized it. A continuous, traceable lineage looks pretty on paper, but it's a prestige based on appearances and romantic assumptions. Of these romantic assumptions, the most widespread is by far the notion that the original teacher found a competent heir who was either his equal or surpassed him, this heir in turn finding a new heir who was either his equal or superior, and so on.
In reality, this is hardly the case and I think when scouting for magical and mystical orders to join that one keeps this possibility in mind - that perhaps, somewhere along the line, the order fell under bad leadership and that it's name means nothing. It happens in politics, it happens in companies, it happens in family clans. Why do we believe these orders to be exempt from this natural fluctuation? Do we think all Emperors equal?
While a prestigious university such as Harvard may have a long history, it's actual merit is solely and entirely dependent on the competence of the current teaching staff and the culture of the students. You may of course tell people that you have graduated from Harvard and piggyback on this prestige in your professional life, but it's a glamour and doesn't speak to your actual competence in the workplace.
In addition, what internal signs should a serious student look for?
A magical and mystical order is by it's a nature a social structure. We all have an innate need for attention, recognition and acceptance from our peers, and we may indeed find it in such and such order. We may adopt our belonging in such a tribe as part of our self-image and identity, and indeed feel that we are part of something very special, exclusive, rare and unique, setting us apart from the "others" - the non-initiates, the mundane, the unenlightened, the muggles or whatever not-so-subtly derogatory term one may use.
None of this has anything to do with magical or mystical practice. The primary engagement with the group becomes less about schooling and more about social mingling.
At worst, even when magic proper is taught, this self identity can foster a sense of "magical elitism", in which an overinflated estimation of ones self in the area of magic is assumed to translate equally well to completely unrelated fields. The solution therefore becomes "do more wealth magic" and not "maybe I should curb my spending habits and put together a budget". You see this especially with certain academics, who speak with the same gravity and authority about things they know nothing about as they do in their specialized field of study.
There is magical practice, yes, but there is also the "cultivation of the heart". How we live and move and have our being. Any order which only addresses the former but not the latter is going to produce very imbalanced students, the kind that will neglect the mundane aspect of life and glorify magic as a panacea that one sweet day will surely do it's wonders.
Indeed, you must look at the senior students and not just the teacher. The students are the fruits of the teacher and his competency lies not in his own success but in his ability to reproduce the same in others. "By their fruit ye shall know them". If you see several students that have been there for a good while who are all out of shape, broke and socially maladapted - if there is an underlying pattern of delusional beliefs and resentment of the world, using the group as a place to vent - this is a very clear indicator that what you're about to get into is a very one sided form of development.
Magic especially has this against it, that as it takes time to learn, there is this underlying and continuous sense of a great promise. The potential lies there, just out of reach, right beneath the surface, waiting to gush forth with transformative blessings. It is not uncommon to see students spending years and years at this hopeful stage, with just a modicum of success. If that's the type of senior practitioners that constitutes the group, that is a very clear indicator that what is being taught is subpar.
We shouldn't judge the teachings of an order based on their potential - that's what solo experimentation is for - but on their actuality; living proof in the pudding among the seniority.
Are teachings consistent over time, or do they constantly shift depending on trends or the leader’s preferences?
I think this an important point too and why a familiarity with primary sources is essential for making a proper judgment call before joining any order.
How can one possibly know that what is being taught is actual Grimoire magic, and not modern make believe that skips 90% of the prescribed classical procedure?
How can one know that the "four elemental tools" of the magician are a completely fabricated Golden Dawn invention, and that such tools in the Grimoires were indeed planetary, if one has not read the source material?
How can one know that the seven chakras were popularized by Arthur Avalon from a 16th century manuscript - which itself mentioned several systems of chakras, none of them considered innately inborn - if one has not investigated these matters from a scholarly perspective, leaving behind deeply ingrained assumptions in the pop-cultural sphere?
How can one possibly know that what is being taught is genuine Daoist alchemy, if one is completely unaware of the traditional Daoist worldview and it's sources?
Because we believe in the goodwill of the teacher and assume that Divine Providence has brought us to this reputable school? That we are somehow special on our spiritual path and are by default secluded from wandering astray on crooked paths?
Yes, "armchair magicianship" is a serious investment of time, but it allows us to see what is in line with a given tradition and it's underlying reasoning, what is a deviation, what is an improvement, and what is purely make believe. It also allows us to ask annoying questions of the teacher and investigate the foundations on which the teachings are given.