Reduce a problem to its lowest sensible abstraction
Framing a problem in its lowest sensible abstraction to help you understand and develop accurate solutions to a problem.
For example, for a business issue: I want to book an appointment rapidly on a website.
You can frame the problem in many different ways. From higher to lower abstractions:
I need to...
- click buttons on my mouse and move it to the right point across my screen very fast.
- click buttons on this website and fill out forms fast.
- parse the DOM to access the website and submit the forms.
- parse HTML forms and submit them via post requests.
- parse an HTTP response, submit an HTTP response
- parse TCP packets, submit a TCP response
- parse the electrical signals and submit an electrical response.
- Greatly increases chances of project success, as you're working at the optimum level to solve the problem.
- It helps you understand what capabilities are required to solve a problem. The team required to parse computer electrical signals would be very different from the team required to click buttons on a website.
When working on building a performance-critical web scraper, the problem in my mind was, "I need to click buttons on this website and fill out forms fast."
To solve that problem, I decided that the best way to solve that problem was using an automated headless browser, controlled by Puppeteer.
However, I soon found out it wasn't fast enough, and scaling would require a lot of computational resources.
Because I thought of the problem in terms of clicking buttons, I became stuck. My thought process revolved around solving the problem I thought I had. "What is faster than headless chrome?" "Years of development had gone into it. How could I possibly make it run faster?"
After a while, I realized the problem was not clicking buttons on a website; it was: "I just need to parse HTML forms and submit them via post requests.". It over-simplified the problem. But I'd found the lowest sensible abstraction.
After I'd stated the problem correctly, I realized I didn't need to use a browser at all. I re-built the scraper in Go, then parsed the HTML and submitted the forms via post requests. Without the complexity of the browser, I moved page processing down from around 50ms to 0.5ms - A 100x improvement.
If I'd stated the problem at the correct abstraction initially, I would have saved myself weeks of work.
Throughout my career, I've seen large projects based upon the wrong framing of a problem.
If someone at the beginning had sat down and stated the problem more accurately, the project would have developed in a completely different and more successful direction. It's also why having many experts with diverse backgrounds will often provide better results with this principle.