FIN 579 Jumble Improvement
Jumble Difficulty Model
A working prototype for my FIN 579 individual project. The model studies 27 past Jumble puzzles, learns how much weight each trait gets in making a puzzle easy, medium or hard, and then labels new jumbles before they're published, so players know what to expect.
The Pitch
Labeling Jumble Difficulty with Machine Learning
A Jumble is a classic word puzzle that was common in daily newspapers. Players would solve clues about a person, place, or thing. As they wrote out their answers to each clue, some of the letters in each answer would fit into circular spots. All of the letters from the Jumble's circular spots would then be rearranged to provide the answer for a final clue, which often had something to do with a theme for the day. I have created an AI skill that makes a daily Jumble puzzle and prepares it to be interacted with on a public website.
Not many people visit my website or play my Jumble puzzles. Perhaps the Jumbles are too hard or too easy. I don't yet know if difficulty is the reason, and the solve data I ask to collect below will tell me. I want to increase engagement with my Jumble puzzles, encourage more people to play them, and have more people return to the website and play them on a daily basis. Specifically, my business problem is that I want to know how difficult my Jumble puzzles are, and label them, so users aren't turned off by unexpectedly easy or hard puzzles. The business problem would be owned by the game designer or data analyst, and I fill both roles. I will know the labels are a success if Jumbles labeled easy are solved 80% or more of the time, Jumbles labeled medium are solved 50–79% of the time, and Jumbles labeled hard are solved under 50% of the time. These ranges are starting targets that I'll adjust once data comes in. I also hope more people will play, but the solve rates are the measure.
My matrix cell is External Facing × Machine Learning. The model must correctly classify each new Jumble as easy, medium, or hard, and display the label on the Jumble for players to see. To solve the problem I created a table of all 45 past Jumbles in my archives. Each Jumble was labeled using rules based on my own judgment of which subjects are harder and how long the answers are. Each Jumble also had a column for each variable I felt contributed to its difficulty level. The variables were: Subject Difficulty, Topic Difficulty, Total Number of Words in the Jumble, and Total Number of Letters in the Jumble. There are 7 possible Jumble subjects. Each subject has 5 to 7 clue topics to fit into 4 total clues in each puzzle. I rated which subjects are most difficult, as well as which clue topics are most difficult. For instance, I have rated Screen Jumbles as less difficult than Historical Events Jumbles, and clue topics about actors as less difficult than clues about historical figures. It is also my initial belief that Jumbles with more words and more letters are more difficult than Jumbles with fewer words and fewer letters.
A linear regression learns from those labeled Jumbles how much weight each of the four variables should get, and uses that equation to label each new Jumble. The current equation is:
Difficulty = −0.99 + 0.86 × Subject Difficulty + 0.32 × Topic Difficulty + 0.01 × Words + 0.015 × Letters
A score under 1.5 is easy, 1.5 up to 2.5 is medium, and 2.5 or more is hard. For example, the October 1 Jumble about Minecraft scored 0.92, so it was labeled easy. The label players see comes from the model. My labels are proxy data. They stand in for real solve results, because earlier Jumbles didn't record whether players solved them. In my prototype, the model gave the same label as my rules on 44 of 45 Jumbles when each one was hidden from it during training, and it found that subject difficulty carries the most weight. The model shows it can reproduce my rules. It doesn't show that my rules are right, which only real solve data can show. The prototype is at ihavefour.com/jumble-difficulty-model.html.
As solve data becomes available, real results will replace my rule-based labels as what the model learns from. I hope to be able to capture how difficult a Jumble actually is by comparing how many people were able to successfully solve each Jumble.
My six one-sentence closed-circle answers are below:
- What is the exact business problem, and who owns it?
As the game designer and data analyst, I want to label the difficulty of my Jumble puzzles so players aren't turned off by unexpectedly easy or hard puzzles.
- What will people do differently once the system runs?
Players will use the label to decide whether to play today's Jumble, and a label they can trust encourages them to come back the next day.
- What data feeds the model, and where does it come from?
Four variables for each of the 45 Jumbles in my archive (subject difficulty, topic difficulty, number of words, and number of letters), with difficulty labels from my own rules as proxy data until the site has collected enough real solve results.
- What is the AI task, precisely?
Classification: a linear regression learns how much weight each variable gets and uses that equation to label each new Jumble as easy, medium, or hard.
- What does the model output, and in what form?
A text label, such as "Easy difficulty," at the top of each Jumble.
- What metric proves we have solved it?
Returning Jumble players would show that the labeling has been a success.
(Right now I can infer how many Jumble players are returning from Google Analytics. Currently, it is rare that I have anyone returning to the website for more than a few days at a time. I also don't have a steady flow of first-time visitors. With a small and steady player base, I might look to calculate a metric like a 10% increase in returning Jumble players.)
A key risk is that an incorrect difficulty label could discourage a player and make them not want to play again. Both my rules and the model rest on my own judgment, not on players. We'll mitigate the risk in two ways. When the model and my rules disagree on a new Jumble, the system flags it so I can review the label before publishing. For example, the system flagged the October 5 Jumble about the Mona Lisa, which the model labeled medium and my rules labeled hard. After reviewing it, I kept the model's label. Solve data will eventually replace my judgment with real results. The Jumble has recorded solves since October 1. My ask is approval to collect solve data for several weeks and then retrain the model on it, so the model can learn from real players in place of my initial rules. Collecting solve data also mitigates the risk.
The Jumble now records whether a player solved it, how many attempts it took, and whether players gave up and opened the answer key. The results are kept on the site and in Google Analytics. I estimate multiple weeks of collecting data on a variety of Jumbles with different subjects, topics, and lengths, and then about a day of work to retrain the model on what we see. In the future, players could also get more "points" for solving harder Jumbles. We'll know we've been successful if solve rates land in the ranges above.
What the model learns from
The site only began recording whether players solve a jumble on October 1, 2026, so there isn’t enough real data yet. For now, the model learns from the Easy / Medium / Hard labels my rules gave 27 past jumbles. The model is a linear regression with four variables:
- Subject difficulty: my rating of the jumble’s subject (1 Easy, 2 Medium, 3 Hard).
- Topic difficulty: the average of my ratings for its four clue topics (1 Easy, 2 Medium, 3 Hard).
- Words: the total number of words in its four answers.
- Letters: the total number of letters in its four answers.
It works out how much weight each variable should get, then uses that equation to label new jumbles. The difficulty label on each new puzzle on the Jumble page comes from this model, trained on every earlier jumble. Once enough plays are recorded (see “How players did” below), the model would learn from those real results instead.
How well it works
The equation the model learned
A score under 1.5 is Easy, 1.5 up to 2.5 is Medium, and 2.5 or more is Hard.
“Share of the weight” compares the variables fairly. Letters run from about 30 to 80 while Subject difficulty only runs from 1 to 3, so the raw weights can’t be compared directly. The share shows how much each variable typically moves the score.
Label a new jumble
The 27 training jumbles
Jumbles come from the Jumble archive as of September 30, 2026. Labels come from my rules: the subject sets the level (Screen, Music and Video Games are Easy; Sports, Food & Drink and Historical Events are Medium; Arts & Literature is Hard), and 11 or more words across the four answers is Hard.
How players did
Scroll the table sideways to see every column, and down to see every jumble.
The table has four groups. Model inputs: the jumble’s subject, topics, words and letters. Label: the difficulty label the jumble was published with, which is what players saw. It was the model’s answer on the day, not one of its inputs, so it can differ from what the model says today as it learns from more jumbles. Today the model learns only from the labels my rules gave the archived jumbles, not from any player result. Player results: what actually happened, and the only part that comes from players. Check: whether the published label matched the results.
Solve rate is solves divided by starts. A start is the first letter a player types, and a solve is every square correct without opening the answer key. A label is “in range” when Easy jumbles are solved 80% or more of the time, Medium 50–79%, and Hard under 50%. Jumbles with fewer than 5 starts aren’t judged yet. An attempt is one click on Check Answers, so a check with a wrong answer counts as an attempt. Attempts per solve is the average number of attempts players needed to solve the jumble. Max is the most attempts by any one player, whether or not they solved it, and Min is the fewest attempts by a player who solved it. A dash means no data yet. Counts are anonymous and began October 1, 2026, so earlier jumbles have no plays recorded. Since October 18, 2026, past jumbles can also be played from the archive. Those plays are included in every count, and “From archive” shows how many of the starts came from the archive rather than the day the puzzle was published. A caution about these results: since October 10, 2026, every player sees the difficulty label before playing, and seeing “Hard” or “Easy” may change how they play. From October 1 to October 10, the label appeared on a separate copy of the Jumble, so those counts mix players who saw it with players who didn’t. I merged the two pages so that every play from here on is collected the same way.