Forwards Rashard Lewis and Hedo Turkoglu added 19 points apiece as the Magic took the first meeting between two of the Eastern Conference's top teams.
James had 23 points but was only 10-for-27 shooting from the field. Cleveland took a 56-55 lead into halftime but managed just 32 points in the second half to lose for the first time in five contests.
"We had to tighten up the defense (in the second half)," Orlando point guard Jameer Nelson told reporters.
"LeBron was getting too many dunks in the first quarter and everything they did was really easy."
Nelson scored 18 points on the day he was named an Eastern Conference All-Star reserve -- the first appearance of his career.
Cavaliers guard Mo Williams, who was coming off a career-high 43 points against the Sacramento Kings on Tuesday, had 12 points on just 4-for-15 shooting.
Cleveland are still without center Zydrunas Ilgauskas, who has missed 13 games with an ankle injury.
The Southeast Division-leading Magic improved to 35-10 with the victory.
Cleveland sits one half-game ahead of Orlando at 35-9.
