Reimagining the feedback experience
context
what does
newton school do?
Learn More
Newton School offers a 6 month bootcamp in fullstack development. It’s a Pay after Placement model, which means that students don’t have to pay anything until they are placed. Since this case study
focusses mainly on the placement journey of students, I have attached a student journey roadmap below for your context :)
The placement journey of a student at Newton School
context
How newton makes
revenue
pay after
placement model
In simple terms, the company makes revenue only after the student is placed.
Better interview performances would increase the probability of placements which would further translate to an increase in revenue
context
so what was the
problem
Feedback response
During placement season, 100+ daily interviews are scheduled
On an average, we received feedback responses of less than 2 students on a daily basis.
team
bandwidth
The number of interviews were projected to grow by 5-10x
Since scheduling & obtaining interview feedback were being done manually, the current team wouldn’t be able to handle such high number of interviews.
Placement
& revenue
The no of students, teachers and mentors were increasing rapidly
The revenue was not increasing at the same rate...
research
to dig deeper into the problem, we did an
experiment
📄 WHAT DID WE DO
Using our WhatsApp chatbot, we messaged around 190 students who
had their interviews in the last 48 hours.
We used the same WhatsApp message as used by our operations team.
🎯 The result
What we found out was unusual and left us confused...
All of this was done without a single follow-up.
Within 16 hours, 36 students submitted more than 100 questions as feedback
This was an
unusual result
The data that we started with conveyed that only 2 students fill the feedback on a daily basis. But the result was completely different. It compelled us to dive deeper and conduct more research.
Research
The script we used for user interviews
Key finding #1
Timing of the message
The students have a hectic timetable and give 2–3 interviews daily. Due to the recency effect, they are able to best recall the questions asked in their last interview.
Recency Effect
When you try to memorize a list of items, you are more likely to recall the items from the list that you studied at the last.
What was going wrong before?
We were sending messages to them after 8–48 hours of their interview.
By this time, the students were already preparing for their next interview instead of thinking about the previous one.
Key finding #2
Trust factor
This was one of the things we missed, the messages to students were being sent from different, personal WhatsApp numbers of the team members.
What is wrong with Personal WhatsApp numbers?
Since they show up as unknown numbers on WhatsApp and no prior relationship exists between students and our ops team, the students have a hard time trusting the message.
solutioning
concept #1
Recorded voice notes
No need to waste time typing, simply record a voice note sharing your feedback about the interview.
problems
Technical constraints, difficult to execute speach to text
More bandwidth required to do quality check of audio files
Slow internet can create problems while submitting the form
concept #2
Simple Form
No distractions, a simple form to obtain feedback.
problems
Does not align with our current design language
We wanted to optimize for speed and keep development
efforts minimum
concept #3
Form with Newton DS
A form that aligned with already existing information architecture and design language. But not so simple anymore...
problems
Poor layout, difficult to scan information
The question input fields don't look actionable
Poor placement of CTAs
solutioning
the final
solution
aftermath
measuring the
impact
number of students submitting feedback daily (a 15x increase)
Number of unique questions filled daily (a 10x increase)
some second order effects
As we started receiving large number of interview questions. We decided to turn them into open for all, company specific question banks. This was done to help students prepare.
Students who opened the question bank (at-least once) had
73% more chances of getting placed as compared to other students.
aftermath
wrapping it up
My learnings
I was fortunate to have got the opportunity of creating this product from 0 to 1. I learnt a few lessons which I’ve found to be widely applicable.
User interviews
They worked great for insights. But in order to dig out the pain points, we had to study the user behavior through experiments.
Documentation
Documenting our decisions and assumptions helped us tremendously in taking feedback and brainstorming. It helped me write this case study,
including devs
Involving devs in brainstorming from the start helped us evaluate our ideas
