By Liz Hoffman

June 11, 2015 5:39 p.m. ET

Big shareholders for years have grumbled about the rise in executive pay. Now, activist investors are taking up the compensation cause, focusing less on how much corporate leaders earn and more on whether they deserve what they get.

Case in point: Shutterfly Inc.[], where an activist hedge fund is seeking three board seats at the online photo retailer at a shareholder vote set for Friday. The founder of Marathon Partners Equity Management LLC said once the fund “started peeling back the onion” on Shutterfly’s pay plans, it found “a compensation scheme that had run amok.”

Marathon’s main complaint, that the company rewards scale over profits, is finding increasing resonance among activists lately. Shutterfly has defended its pay plan, while it recently tweaked the metrics it uses to determine pay. Changes to the plan “appropriately reflect stockholder views while also balancing the critical importance of retaining key employees,” the company has said.

Once left to governance hounds, unions and academics, executive pay is getting a closer look from activist investors, which take stakes in companies and push for measures to boost share prices. These funds are scanning corporate filings for what they see as skewed incentives and generous formulas, increasingly moving what had long been a back burner issue to the fore.

“Compensation was not something activists cared about a great deal, unless they could use it as a wedge to get something else done,” said Francis Byrd, a former TIAA-CREF corporate-governance expert. “That’s starting to change.”

Activists have zeroed in on pay recently at Qualcomm Inc.[], DuPont Co.[] and Perry Ellis International Inc.[] Some activists argue that ill-designed plans encourage the wrong kinds of growth—for example, boosting revenue at the expense of profitability. Others point to nonstandard financial metrics they say reward executives even when business falters. Criticisms tend to focus less on the size of CEOs’ paychecks than on the yardsticks that determine them.

Companies say such critiques are off-base and that handcuffing pay packages makes it hard to retain talent. They also note executive pay is mostly stock, which ties executives’ fortunes to those of all shareholders.

Most activists are unlikely to pick fights based solely on pay. Arguments about corporate operations are considered more powerful, investors and advisers say. But more are embracing the idea that incentives matter.

One factor teeing up the issue for activists is the complexity of compensation, experts say.

“I’ve seen bonus plans that would take a Ph.D. in physics to figure out,” said Kevin McManus, vice president at shareholder-advisory firm Egan-Jones Ratings Co.

That complexity presents opportunity for activists, who typically research a handful of companies, rather than monitor hundreds. “You’ve got someone who has the time and the incentive to dig into the numbers” Mr. McManus said.

Take Trian Fund Management LP, which narrowly lost a DuPont [] shareholder vote for board seats in May. It has made a small cause out of executive compensation, regularly seeking a seat on the compensation committee of boards it goes on, according to people familiar with its practices.

At DuPont, however, Trian criticized how the company’s board calculated bonuses, drawing on scores for individual performance and overall corporate performance. DuPont’s board in 2014 assigned zero points for the latter, after earnings inched up just 3%, but gave executives higher personal scores.

“How can it be that the company is doing poorly operationally but management as individuals are each doing great?” Trian said in shareholder materials.

DuPont recently changed its plan to give a heavier weighting to the company’s overall performance and de-emphasize individual scores. A spokeswoman said the plans “are designed to align pay with performance and the achievement of annual goals and objectives.”

Jana Partners LLC, which recently took a $2 billion stake in Qualcomm[], has urged the company to tie executive pay to measures like return on invested capital, rather than its current yardsticks of revenue and operating income, according to a Jana investor letter. Such changes “would eliminate the incentive to grow at any cost,” said the letter, reviewed by The Wall Street Journal.

Qualcomm has said its pay plans were aligned with stockholders’ interests and cited the “increasing competitive threat Qualcomm faces for talent.” An example: Its current chief executive was courted by Microsoft Corp., according to a person familiar with the matter.

Another red flag for activists: metrics that change on the fly.

In 2010, Perry Ellis linked bonuses for its father-and-son management team to earnings before interest, taxes, depreciation and amortization, a common measure of cash flow known as Ebitda. By 2013, the retailer had fallen short of its three-year goal.

Meanwhile, a new phrase had appeared in the company’s filings: “adjusted Ebitda.” Adding back in certain costs, the tweaks pushed the company just above the threshold necessary for the two men to receive their maximum bonuses of $1.4 million each.

Activist hedge fund Legion Partners LLC said in May the change “undermines confidence” and wasn’t adequately disclosed to shareholders. Legion, along with a California pension fund, had been seeking board seats but dropped the fight last month after Perry Ellis added new directors and announced a CEO succession plan.

A spokeswoman for Perry Ellis declined to comment on the change. She said the executives didn’t receive a bonus in 2014 or 2015 after the company fell short of the new targets.

Perry Ellis recently revamped its bonus metrics again. Executive bonuses will soon be based on total shareholder return and a mix of earnings before taxes and return on invested capital—both “as adjusted.”

The larger argument from activists—that pay too often rewards size over profitability—echoes a common push among activists to get companies to slim down and focus.

In its early years as a fledgling technology company Shutterfly needed scale, Marathon’s managing partner Mario Cibelli said, so tying bonuses to revenue growth, among other measures, made sense. Now, Shutterfly is a $1.7 billion company in a maturing industry and should reward profitability, he argues.

Starting next year, Shutterfly will tie bonuses to measures that more closely track profits, including free cash flow and total shareholder return.

—Michael Rapoport contributed to this article.

Write to Liz Hoffman at liz.hoffman@wsj.com