(1) Define the problem correctly
A libertarian government (let's avoid the anarchist-minarchist debate for now: a government that doesn't aggress, doesn't tax, and allows secession is fine with me) would consider the protection of the people in its jurisdiction to be its defense responsibility. That doesn't require nearly as much money as being the world's policeman. The US government spends nearly $1 trillion on "national defense" right now, while the second place country, China, spends less than $100 billion, with a population more than 4 times the US.
(2) Reduce the number of enemies
Probably 99% of those people who hate America enough to want to engage in violence against Americans are mad about the US government's foreign policy (significantly less than 1% only hate us for our freedom). Are there SOME people who will want to engage in massive violence on US soil in the absence of intervention? Maybe, but they will be isolated madmen without the sympathy of any large group of foreigners, and as much a threat as common criminals.
We also reduce hatred with free trade: commerce makes two people benefit from each other's existence (free travel and migration could help as well). In the final analysis, there is no reason for any large group of people to hate Americans who don't intervene militarily and who exchange goods and services with them.
(3) Use a variety of voluntary methods of financing
It is quite unlikely that a purely defensive libertarian government that covered the entire territory of the present US would require more than, say, $50 billion for the special forces and intelligence needed to deal with the relatively small number of threats, along with modest repairs, maintenance, and upgrading of military equipment. Although it is impossible to predict how a free society would finance the need, here are a few thoughts:
(a) Charity - People already give $300 billion a year to charities. With the enormous prosperity of a society free of regulation and taxes, it would be even more generous. Raising $50 billion from 300 million people for a service most of them believe is protecting their lives and liberty shouldn't be very hard. Heck, people TIP close to $50 billion a year. Will there be free riders? Sure. There are free riders now.
(b) Advertising - "When you buy a Big Mac, you're also helping protect America." Businesses will pony up cash to sponsor activities seen by the public as defending the country.
(c) Insurance - People will insure their own life and property (rich people will pay more than poor people). Insurance companies want to minimize losses, and with sufficient numbers of policies at risk, will finance measures that reduce their risk. Multiple insurers will cooperate with each other to fund activities for mutual benefit. Social norms will ostracize those insurance companies that offer lower prices by not helping defend people.
(d) User fees - "National defense" and "defense from terrorism" are not generic services of one type: there are degrees. Businesses can pay for more security at their establishments. People can pay for background checks.
(e) Volunteers - Citizen militias, made up of retired military and others, can provide the reserves for local defense in the highly unlikely event that any large group is crazy enough to try to invade and rule. We can start challenging the Swiss dominance in the biathlon at the Olympics.
Every legitimate service that benefits large numbers of people can be financed voluntarily with a little imagination. The reason taxes are needed today is that nobody in their right mind would voluntarily finance most of what passes for government services.
On the broad issue of funding public goods without coercion, I strongly recommend Roderick Long's Funding Public Goods: Six Solutions. In addition, Dr. Ruwart has addressed the issue of financing government without taxation in various Ask Dr Ruwart columns.
Update: Thanks, Steve. I corrected my typo on China.
