This new gallery with rotating exhibitions of modern and contemporary art is housed in the stunning Castle Garden Kiosk (Várkert Kioszk), a lovely little building with a fountain designed by Miklós Ybl in 1878. It was once a pump house for Castle Hill; come here if for no other reason than to view the lovely renovated interiors.
There's a fine cafe with outside seating here too.