By Jason Zweig

June 23, 2017 1:55 p.m. ET

In less than two decades, more than half of all publicly traded companies have disappeared. There were 7,355 U.S. stocks in November 1997, according to the Center for Research in Security Prices at the University of Chicago’s Booth School of Business. Nowadays, there are fewer than 3,600.

A close look at the data helps explain why stock pickers have been underperforming. And the shrinking number of companies should make all investors more skeptical about the market-beating claims of recently trendy strategies.

Back in November 1997, there were more than 2,500 small stocks and nearly 4,000 “microcap” stocks, according to the Center for Research in Security Prices. At the end of 2016, fewer than 1,200 small and just under 1,900 microcap stocks were left.

Most of those companies melted away between 2000 and 2012, but the numbers show no signs of recovering.

Several factors explain the shrinking number of stocks, analysts say, including the regulatory red tape that discourages smaller companies from going and staying public; the flood of venture-capital funding that enables young companies to stay private longer; and the rise of private-equity funds, whose buyouts take shares off the public market.

For stock pickers, differentiating among the remaining choices is “an even harder game” than it was when the market consisted of twice as many companies, says Michael Mauboussin, an investment strategist at Credit Suisse Group AG in New York who wrote a report this spring titled “The Incredible Shrinking Universe of Stocks.”

That’s because the surviving companies tend to be “fewer, bigger, older, more profitable and easier to analyze,” he says, making stock picking much more competitive.

Consider small-stock funds. Often, they compare themselves with the Russell 2000, an index of the U.S. stocks ranked 1,001 through 3,000 by total market value. “Twenty years ago, there were over 4,000 stocks smaller” than the inclusion cutoff for the Russell 2000, says Lubos Pastor, a finance professor at the University of Chicago. “That number is down to less than 1,000 today.”

So fund managers have far fewer stocks to choose from if they venture outside the index, the very area in which the best bargains might be found. More money chasing fewer stocks could lead some fund managers to buy indiscriminately, regardless of value.

Eric Cinnamond is a veteran portfolio manager with a solid record of investing in small stocks. Last year, he took the drastic step of shutting down his roughly $400 million mutual fund, Aston/River Road Independent Value, and giving his investors their money back.

“Prices got so crazy in small-caps, I fired myself,” he says. “My portfolio was 90% in cash at the end, because I couldn’t find anything to buy. If I’d kept investing, I was sure I’d lose people their money.”

He adds, “It was the hardest thing I’ve ever done professionally, but I didn’t feel I had a choice. I knew my companies were overvalued.”

Mr. Cinnamond hopes to return to the market when, in his view, values become attractive again. He doesn’t expect recent conditions to be permanent.

The evaporation of thousands of companies may have one enduring result, however, and it could catch many investors by surprise.

Most research on historical returns, points out Mr. Mauboussin, is based on the days when the stock market had twice as many companies as it does today. “Was the population of companies so different then,” he asks, “that the inferences we draw from it might no longer be valid?”

“Factor investing,” also known as systematic or smart-beta investing, picks hundreds or thousands of stocks at a time based on common sources of risk and return. Among them: how big companies are, how much their shares fluctuate, how expensive their shares are relative to asset value and so on.

But the historical outperformance of many such factors may have been driven largely by the tiniest companies, exactly those that have disappeared from the market in droves.

Before concluding that small stocks or cheap “value” stocks will outrace the market as impressively as they did in the past, you should pause to consider how they will perform without the tailwinds from thousands of tiny stocks that no longer exist.

The stock market has more than tripled in the past eight years, so the eclipse of so many companies hasn’t been a catastrophe. But it does imply that investing in some of the market’s trendiest strategies might be less profitable in the future than they looked in the past.

Write to Jason Zweig at intelligentinvestor@wsj.com