What you're saying is correct, but there's a missing piece. Yes, there can be an infinite number of directions in a 3D scene. But that doesn't mean that there are actually an infinite number of orientations for the edges actually existing within a scene. There can be a lot - too many to draw them all - but given that we're talking about actual sets of edges that we have to draw in our scene, they are not infinite. But that's kind of splitting hairs, in the case that we're using "infinite" to mean "a hell of a lot" and "more than would be reasonable to plot out on the page".

It being more than reasonable is why in this course we try not to focus on vanishing points. We introduce what they represent and the rules they adhere to in order to be useful to us, and then we shift more and more away from focusing on these far-off points to instead the actual relationships they govern. A vanishing point gives us something to refer to as we add more edges to a scene, but the edges themselves that point to it imply its position. They give us enough information to add more edges to that set, even without having the VP itself marked out.

It can be useful to have a limited set of VPs plotted out to give us our major axes (that's why perspective is often talked about in terms of 1, 2, and 3 point perspective, and not often more than that outside of very specific stylistic effects) - not because they're the only relevant VPs governing the scene, but because they're enough to define 3 dimensions (in the case of 1 and 2 point perspective, there's still 3, it's just that one or two are forced to infinity as we discuss in Lesson 1).

These limited VPs are useful in providing us some sort of reference to help in our estimation of those sets of edges that don't converge towards them directly, but ultimately we're still focusing more on how those edges within a given set converge with one another, ensuring that what we draw at least maintains some consistency between those things that are meant to be parallel to one another. It can throw things off considerably if there's surfaces or edges you know are supposed to be parallel in 3D space, but very clearly don't converge towards a shared VP.