A few of my students will be testing for promotion this weekend, and if all goes well they'll get new belts, or new patches on their existing belts, and get to start learning new material next week.
The idea of kyu and dan ranks, much less of various colored obi, was not part of martial arts training in the old days. If you were a rank-and-file soldier you probably learned a few basics through drilling with your unit, and maybe a veteran would show you an advanced technique or two. If you had the good luck to study an art directly with a teacher, he (or maybe in a handful of instances she, but we are talking about a time long before gender equality was a widespread idea) would show you stuff when he thought you were ready for it.
As the arts developed in Japan, particularly jujutsu (the unarmed fighting art of the samurai) and kenjutsu (the sword art), and systems of schools became organized, the people at the top of the various ryu needed to be able to acknowledge and distinguish authorized teachers. So a system of certificates or licenses, menkyo, was created. Each school might use them a little differently, with different schools having different numbers and titles for them, but even the first came only after many years of training, and they were designed more as a matter of certifying teachers than as a way to break down a student's training. So if you were a master fighter but had no interest in opening your own school, you might not have any rank.
Zen began to be associated with the martial arts in Japan starting around 1200, for various political reasons, but the martial arts were only available to the warrior class until after the Meiji restoration of the 1860s . If you were a commoner or a merchant, you couldn't study martial arts at all. I'm sure some teachers occasionally made secret exceptions, caste systems have never been ironclad, but exclusion was the rule. So the idea of martial arts as a path of personal development for the average person was not possible in Japan at that time. Martial arts were for soldiers to kill enemies. If meditation helped you keep your mind clear to be a better killer, good. While there was an stringent ethical code, at its heart was killing or dying at the command of one's feudal lord.
However, as Japanese society was changing very rapidly in the late 1800s and early 1900s, it became possible for people outside the warrior caste to study the arts. I don't want to paint an overly rosy picture, so we have to acknowledge that part of the reason for that was that Japanese society was becoming militaristic and imperialist and wanted to use the arts to instill these values in the wider populace. This was part of the process that would lead to ultimately to Pearl Harbor, Nanking, Hiroshima, and Nagasaki, and we have to acknowledge it as a dark element of the martial arts' history.
But it was also a step in making the arts available as a means of self-improvement to people outside the warrior caste. One of the most historically significant of these people was Jigoro Kano, the founder of judo. Kano's father was a Shinto priest and businessman who later became a government clerk, and his mother's family were sake brewers, so in the old days he would never have been able to train. But in the new social order he was able to become a master of jujutsu.
More than that, Kano made his living as a educator, and his interests included a blend of Western-style progressive education and Japanese classicism's Confucian roots. (If you're wondering why a Chinese philosopher like Confucius would be so influential on Japanese thought, it's for the same sort of reasons that Greek and Roman philosophers have been so influential on Anglo-American thought .)
As Kano developed his ideas of martial arts and education into a new art which could serve as a means for personal development for all people, he saw a need to break training down in the same way that students in school system progress through the various grades. He borrowed the ranking system from the game of go, and eventually the tradition of using colored belts for kyu ranks and dan ranks was developed.
When Gichin Funakoshi came to mainland Japan from Okinawa and started to popularize karate, he saw the wisdom of Kano's approach. (Funakoshi was also an educator, and he and Kano seem to have gotten along quite well.) He adopted the same ranking system for his karate school, and it eventually spread throughout Japanese karate (and from there to Korean tae kwon do).
Used properly this system is a fine teaching tool. But it's important for students to understand what it's for. These ranks are a way of breaking down a student's progress. So they have to come more often earlier in the process, when we really have no idea what's going on and need more constant feedback. As we grow as students and become more able to direct our own study and training, these checkpoints become further and further apart. Your first few kyu promotions might be a few months apart, while dan promotions come only at intervals of several years. (I took my godan promotion in 2013 and won't even be eligible to take the next until, I think, 2020.)
If you don't pass a test, it doesn't mean you're a failure or that we don't like you or anything like that. It just means that you're not ready yet to build your training higher. You have to build the bottom of a building before you build the top, and if there's a problem in the lower levels and you build up anyway it's all going to fall down eventually. Even if you pass your promotion test usually we'll find some weak areas that need to be addressed and fixed up.
So think of a promotion test as an opportunity to have your karate inspected. If it passes inspection then you can build more on top of it, and if it doesn't then you've found areas to rebuild. Either way it's a chance to improve.