An asymmetric bet on deliberate practice to pass a series of difficult exams
How I used Anki to pass the Canadian CPA exams, by studying less
I’ve been unable to shake an ear worm over the last few months, ever since I read musings on deliberate practice and improving as a knowledge worker. If I were a concert pianist, the path to improving as a pianist is clear - scales, drilling on particular parts of pieces, etc. If I were a baseball player, the path to improving is also clear - strength training, drills, etc.
Putting aside whether or not those things are effective at improving the work of concert pianists or baseball players, the thing that differentiates that from knowledge work is that most knowledge workers don’t even think about improvement. As a concert pianist, the question I ask is “how do I get better?”. As a knowledge worker, generally the question I ask is “when do I get off work?”.
The first step for me was to define a set of goals. Getting “better” as a knowledge worker is ill-defined. What does it mean to be “better”? I’m paid now for my ideas and for code that I write, so ‘better’ is more along the lines of being able to better develop ideas over time, writing better code, retaining more of what I read, etc.
I have been exploring this idea for the last few years without having formulated it so well. I wanted to explore one particular example of how I applied deliberate learning in the past, while studying for my CPA exams with Anki. I made an asymmetric bet that I could better prepare for the exam, relative to ‘the way’ you study, with less time input.
The CPA Way
In a past life, I worked at a public accounting firm as an auditor. As part of my work, I had to become a Chartered Professional Accountant (CPA). Being a CPA is mostly a set of signals that you are sending to others that you have:
completed a set of rigorous exams, which itself is a signal for you being a competent accountant; and
pay lots of money to CPA on an annual basis to continue being a CPA
The requirements to becoming a CPA in Canada are stricter than becoming a lawyer. You need 30 months of work experience, and need to pass a series of exams along with ‘self study’ modules. This is all capped off by a three-day mega exam the Common Final Exam (CFE).
The exams are case-based. They present you with a lengthy situation and are expecting an answer in a certain format. CPA tells you exactly how they want you to write their exams, by giving a statement of the problem, stating the facts, stating the section of the accounting standards that apply and the recommendations. Most candidates who fail probably fail to give CPA what they want. Therefore the CFE is not primarily a test of your accounting knowledge but instead is a test of your ability to follow instructions.
The problem for studying is that there is a lot of material you need to learn, and it’s all very technical language. For example, here is a portion of the legalese in one section of the ‘Handbook’:
You are incentivized by CPA for instant recall. The exams are tightly timed, and you are expected to read the case, understand a series of hooks (this part of the case expects me to talk about concept x
), and write your answer. During the CFE, I wrote about 15k words for each day, so there really isn’t a lot of time during the exam for research. The longer you spend durdling, the less time you have for actual writing, and you need to do a lot of writing.
Preparing for the CFE with Anki
The typical advice for the typical public accountant is to take at least eight weeks off work in order to study. Studying should be treated like a full time job, so you spend 40 hours a week studying. Three to four of the days should be spent writing cases, with one to two days spent studying technical materials (which everyone just calls the “technical”).
The issue with this is that studying the technical isn’t actually preparing you for the exam. Remember, the exam tests your ability to write in the manner CPA wants you to write, not your technical ability. The technical is necessary but not sufficient. I made a bet to myself that I could more efficiently prepare for the exams by studying with Anki instead of the usual path. Optimizing for better technical knowledge is optimizing for the wrong thing.
Anki is spaced repetition software (SRS). You define a set of flash cards, and when reviewing cards, the algorithm attempts to show you what you need to see, rather than everything. It helps to flatten the ‘forgetting curve’, by challenging your memory on a regular basis. Med students use it extensively, but it hasn’t caught on in other fields.
I spent a few weeks or months leading up to my leave from work building my deck of cards. Instead of spending an entire day a week reviewing technical material, I spent about 30 minutes a day reviewing the cards, and was able to spend more time overall writing cases. I don’t think I ever really studied for 40 hours a week, probably more like 20-25, but a larger percentage of that time than usual was spent writing practice cases instead of studying.
Results
Most importantly, I passed the exam. Taking a bet like this felt risky at the time, and in the four months or so after the exam I had plenty of time to consider whether or not I messed up by moving off the beaten path. In hindsight, I knew that I knew the technical material well enough, so the risk wasn’t that Anki was a mistake, and more general risk that I didn’t pass.
Unfortunately, the exam is pass/fail, so I don’t actually know how well I did. It is possible that I barely passed, but since there is no reward for doing better than ‘just’ passing, anything more would be inefficient.1
What I love about Anki though is that it is an asymmetric bet. The potential upside (greater recall with less time put in) is way greater than the downside, because since you need to start preparing weeks in advance, at any point you can quit and go back to ‘normal’ practice.
How to prepare for exams with Anki
There are well defined resources online for getting started with Anki. The subreddit is great. The SuperMemo wiki (another SRS) is also a font of knowledge. In particular, things to keep in mind:
You need to start preparing weeks or months in advance. Cramming last minute is not going to help you. Remember, we’re looking for deliberate practice and not a silver bullet solution.
You can find decks of flash cards online, but that is not a substitute for building them yourself. The act of writing the cards yourself helps encode the knowledge.
Anki is also only a tool for remembering, not for learning. You will not remember answers for cards you don’t understand, trust me.
The content of the cards matters a great deal. You will remember a lot more by following a set of rules developed for SuperMemo. Essentially, cards should be atomic, and not cramming full of information. If you need to make a detailed card, it probably should be multiple cards instead.
Today
My needs have changed now that I work on solving problems with undefined solutions, rather than exams. So, I don’t use Anki anymore. I probably studied daily on Anki for about five years, but I have transitioned in the last six months to a zettelkasten. I plan to blog about how I use that in the future, but I actually use index cards and pen instead of a digital solution.
There is an honour roll, but its for the top 100 people writing the exam, or something like that. Don’t get me wrong, I would have liked the recognition, but I don’t think it would have furthered any of my career goals, so it didn’t seem worth it. Given how well I felt prepared for the CFE technically, getting on the honour roll probably would involve better preparation for case writing as opposed to technical, so it isn’t relevant for this discussion.