Display year selection dropdown in a calendar

I have a calendar control which currently has "Previous Year" and "Next Year" button to switch years. I want to improve it, as most of the time a user has to go 3-4 years back.

Currently, I plan to remove both buttons, and put a dropdown there. Problem is the dropdown size gets too large for comfort if I add +/- 20 years range. If I set fix height, then there is a scroll bar. Sure, it is still fewer amounts of clicks if I have to use dropdown to go five years back. However, it is more time consuming with a scroll bar, and finicky to deal without fix height.

I am thinking of other options, like using an auto complete box, but that requires user to move from mouse to keyboard. Or I can use multiple dropdown, one to choose a decade and second to choose a year, which will cause extra clicks (especially for selecting a next year's date in 2019).

What other ways can I implement this?

P.S. I can't use the default calendar control as that requires a lot of changes with the application (way too many to justify to management).