For the first part, why he doesn’t have a formal education: Nog is living on a space station with his uncle and father. Not in a Ferengi society. The Uncle doesn’t want to shell out for an education for the kid cause he’s Ferengi and doesn’t want to pay for things, and the father just goes along because he’s kind of a doormat. There’s not enough opportunity for job-shadowing that allows Nog to learn from a Ferengi mentor that’s capable of teaching such things as literacy, so he remains ignorant. Also explains why he gets into so much trouble (aside from the bad influence of his uncle), as he has too much free time from lack of mentorship.

He also shows that he’s very hard working when he applies himself. So, in addition to tutoring from Jake and probably Keiko, he just buckles down and does the work. Especially when he gets to the academy. He specifies in “Homefront” (Or “Paradise Lost”, I don’t remember exactly) that he’s doing all right at the Academy. Not excelling, so he’s struggling. Probably not the least of which is because he’s behind in education as compared to his classmates.