Ask users to set a password or pin during the registration process

We are going to give our users the ability to set a password on their apps to protect their personal data such as their transactions, wallet balance and etc. We decided to ask them to set a password, after their initial sign-up process that includes giving OTP (one-time password) after they entered their mobile number.

The problem is we are afraid of reducing conversion rate out of pushing users to set a password in their sign-up process. Is this the right time to ask the user to set up this password?