Hit it with the hammer and screwdriver, nothing at first then pop out they came
Happy days, job done.
Now, I would suggest you treat yourself to a proper engineers punch (large). It will help you a great deal as the handle of the screwdriver will absorb a lot of the hammer effort and ultimately make the job harder. The proper punch will transmit the hammer energy straight through to the job.
A worthwhile addition to your toolbox