Indeed filmed in Portugal, as I love the film and have been trying to find the exact location of the castle ruins. Hope to have a holiday and pack in a visit at some point. This film is badly flawed, but it is creepily wonderful all the same and I dread a studio ever planning a remake.
I can't give you a definitive answer, but you may want to start with these buildings or buildings here, which can be glimpsed in the film.
Monasterio del Cescon, Madrid, Spain
Estorial, Cascais, Lisbon, Portugal
Faculty of Medicine, Lisbon University, Lisbon, Portugal
Good luck and happy hunting. I have had the good fortune to visit Spain, but I have yet to make it to Portugal. One of those places I hope to see sometime in the future.