NEW YORK -- B.J. Upton's left ankle bothered him throughout Wednesday night's 4-2 loss to the Yankees, causing the Rays center fielder to come up short on fielding Derek Jeter's fifth-inning double and prompting Rays manager Joe Maddon to take him out of the game after the fifth inning.

Jeter's second hit of the game, on a night when he tied Lou Gehrig's Yankees record of 2,721 hits, came via a double he hit over Upton's head in center field. To many Upton appeared not to hustle on the play, but the ankle he sprained last week was at the root of the problem.

"Just couldn't get there," Upton said. "If I'm 100 percent, maybe 95, I would have gotten there. But [his ankle] just wouldn't let me get there."

In addition, Upton barely got to a shallow fly ball by Brett Gardner to make a catch in the fourth, two innings after he misplayed Robinson Cano's routine fly ball..

"I thought [Jeter's double], you would have seen him go after it harder if he'd had a better ankle under him," Maddon said. "I think it was a hit regardless. But I don't think he went after it as hard as he could because his ankle isn't up to speed. And then [Gardner's hit] he came in on a shorter pop up and he had to turn it on at the end. But I was watching him come off the field, and his gait, he had a little hitch in his giddy-up, so I got him out of there."

Upton couldn't shake the pain Wednesday night.

"At the [beginning] of the game it wasn't too bad," Upton said. "But inning by inning, just standing on it and being in a ready position, it just got more sore as the game went on."

Maddon plans to give Upton some time off to let his injured ankle heal.

"Ankle was bothering him once again, so we got him out of there," Maddon said. "Looks like we're going to have to rest him for a bit.

"I'm going to give him some time to get it well. ... Everybody's playing with a lot of intensity among our group. If you get to the point where he can't really get after it, I would prefer he just gets well."

Upton said he wants to play again.

"Hopefully, the next couple of days, we knock it out," Upton said. "Get some treatments on it and it gets better. We'll see."