"Wasn't the Scriptural canon just chosen by privileged men?"
The Bible's authority rests on God.
We must acknowledge one key fact about Scripture's authority: The early church didn't create, compile, or stand as the final authority on the Bible. Early Christians simply recognized the legitimacy of the 66 books: Is it reliable? Is it consistent? Is it corroborated? The Bible is, therefore, not a man-made construct contrived by a council or determined by men. Rather, its authority rests on God.
For example, time and again—despite disputes and disagreements—the same 27 books of the New Testament have passed the litmus test of reliability: 1) Did the literature originate from an apostle or colleague of an apostle? 2) Were the teachings doctrinally sound according to the teachings of Christ and His apostles? 3) Did the churches of early Christianity universally accept the material as accurate and trustworthy? God, in His sovereignty, has protected His Word throughout the ages. If man had been the ultimate orchestrator of the Bible, it would have fallen apart.
Ultimately, Jesus taught the full authority of the Old Testament (see Matt. 5:17-19, 12:40). He affirmed His words were equally authoritative (see Matt. 5:21-44). The events and teachings of His earthly ministry were also penned by those close to Him while other eyewitnesses were still alive to refute them. Any documents that fell outside of these boundary lines never made it into the lasting canon of Scripture, and rightly so.