Overstaying means allowing your visa, or BRP, to expire whilst you remain in the UK. Overstaying is a criminal offence. There is no period within which you can lawfully overstay.
If you submit a valid application for further leave to remain before your current visa/BRP expires, and your application is still being decided when your current visa/BRP expires, you will not be an overstayer in this post-expiry period. Instead your previous immigration permission and the conditions attached to it will continue while your application is pending.
If you become an overstayer, and then wish to make a new Student application in order to remain in the UK, your Student sponsor may not issue you a CAS in order for you to do this and/or it may withdraw any unused CAS that you were issued before you became on overstayer. Seek advice from your Student sponsor about its policies on issuing CAS to overstayers, and on whether someone who makes such an application can study.
Overstaying will have serious consequences for any future immigration applications that you make. If you overstay by more than 30 days you will normally be prohibited from coming back to the UK for at least 12 months from the date you leave the UK. Overstaying in the UK could also affect applications you make for entry to countries other than the UK. You must tell the truth in immigration applications and declare any periods of overstay if asked about them. If you are found to have used deception in an immigration application you are likely to be barred from the UK for ten years.
If you need more time in the UK to complete your studies, see Making a Student route application in the UK.