Your ancestors are all the people responsible for you being alive now. Some are by water, some are by blood, some are by culture.
The people you are literally descended from are indeed your ancestors, but so are the family friends (or the godfathers/mothers, or the cumpari) who were considered to be family.
But how far does the bloodline go? How far do you want it to go? In the Norse tradition, for example, the Disir are female spirits who protect the family, and are the uninterrupted line of mother to daughter that in the end brought you to life.
And also, your ancestors are technically those who came before you, like in magical lineage, or even those who fought for your right to exist if you are a minority.
You don't need to like your immediate family to work with ancestors. Probably, you won't like all your ancestors.
Do they care about your relationship with your family? Depends: some absolutely will, some absolutely not (especially if they left the family in the first place or were disowned).
They are family: you can visit them for the holidays or chat with them every evening, or you can decide it's better for you to go no contact.