After a judge removed him from his daughter’s controversial conservatorship on Wednesday, Jamie Spears has complained about the “tremendous amount of daily worry and work” he went through. People magazine reported that Jamie called the decision a “loss for Britney.” (Britney’s temporary conservator until Dec. 31 will be a California accountant named John Zabel.) Jamie’s lawyer claimed he had been “biting his tongue and not responding to all the false, speculative, and unsubstantiated attacks on him by certain members of the public, media, or more recently, Britney’s own attorney.” Britney herself called the conservatorship abusive, and numerous reports have detailed creepy, manipulate behavior by Jamie, who controlled the pop star’s life and multimillion-dollar empire.