Should filters collapse and expand with space or stay a fixed height?

I'm doing an interface where the user is allowed to filter down by attributes. The attributes are filled in by the users, so there could be 1 filter in the list or hundreds.

Do I let the filter area stay a fixed height all the time even if there are no attributes or do I let it grow and shrink depending on the number of filters available?

Photos for reference: