Certainty Based Marking (CBM): Measuring Training Success

Certainty Based Marking (CBM): Measuring Training Success

Creating effective learning is a continuous challenge for corporate businesses. The introduction of new technologies has eased this by opening up more ways for employees to take part in workplace training, such as e-learning platforms, video learning, and virtual reality. Formal training has vastly progressed over the years, but there is room for improvement through assessments, and that’s where Certainty Based Marking (CBM) can have a huge impact on training and development.


Multiple-choice assessments remain the most common and convenient method of evaluation; scoring is quick and simple, multiple versions can be created and simple metrics highlight who knows the subject matter and who doesn’t. However, these traditional metrics aren’t completely reliable. They fail to demonstrate how training impacts the employees’ performance, their confidence, or how effectively knowledge is retained over time. Despite the convenience of multiple-choice assessments in terms of distribution and the metrics it provides, they are problematic. Multiple-choice assessments make it easier for the learner to guess the answer, rather than demonstrate that they know specific information. 

Multiple-choice also tests the learner’s ability to recall information, as opposed to showing that the learner understands the answer’s application in real-life situations. For example, a salesperson might be taught their organization’s sales processes and be able to identify them correctly in a multiple-choice question, but that doesn’t necessarily mean that they’ll be able to remember that information independently and put it into practice in the workplace. Assessments help organizations understand their employees’ knowledge levels. By tracking the correlation between performance and training results, organizations can clearly see the impact training has had on their workforce. Additionally, there is a third metric that can also contribute to how effective training is: employee confidence. We have found that measuring a learner’s confidence in their knowledge is not only a good way to test how solid their knowledge really is, but it also provides a strong indication of how well a learner can put that knowledge into practice.


What is Certainty Based Marking?

Certainty Based Marking (CBM) can be utilized to evaluate assessments. Not only must learners indicate what they think the correct answer is, they must also rate how certain they are of it (from ‘High’ to ‘Low’). For example, if a learner gets the answer right with high certainty, they are sufficiently knowledgeable. However, if a learner gets the answer wrong with high certainty, the unique scoring system alerts organizations to any ‘confidently incompetent’ employees. CBM was theorized by Tony Gardner-Medwin, a professor of physiology at University College London in 1994. According to him, past experiments show that asking learners to indicate their confidence levels stimulate a higher level of thinking, which assists learning and recall:

“When properly set up, it motivates students to think carefully about whether they can see reasons for either sound justification or uncertainty about the answers they give in tests. It is particularly helpful as a study aid in self-tests, providing a stimulus and challenge, and a warning signal when misconceptions have arisen. But it has also been clearly shown to improve the statistical reliability and validity of assessment data.” CBM enhances traditional assessment approaches by measuring both competence and confidence. It also avoids false indications of accuracy, as learners are unable to simply guess the right answer. To encourage honesty from learners, the CBM assessment scoring system does not penalize for incorrect answers with low confidence. Therefore, organizations can be more confident that they’re getting an accurate reflection of a learner’s knowledge. In addition, unsure but careful thinkers are not deflated by receiving a minus score – they are actually encouraged because more considered responses receive higher marks.


Using Certainty Based Marking with Assessments

Employee competence is invaluable to an organization, but only as long as it lasts for the long term, otherwise, the organization could face negative consequences. Employee evaluation and assessments are commonly implemented as a one-off event when in reality they should be implemented consistently in order to successfully track employee’s knowledge levels. Assessments also help inform employers’ training decisions as it highlights the subject areas employees are struggling with. This allows training to be refined and for relevant modules to be delivered in a cost-effective way. To do this, Wranx uses pre and post-assessments in conjunction with CBM.

A pre-assessment determines the learner’s existing knowledge to act as a benchmark. Spaced Repetition training is then delivered over a given period and upon conclusion, the learner is assessed again to determine the impact of training on their knowledge. This is particularly useful for those falling below the pass-rate, as their progression becomes more clear. On average, using the AMA approach, we’ve found that learners improve knowledge levels by 25 points from pre- to post-assessment. By adding CBM into the mix means we see increased confidence, with high-engagement learners improving by 38% and even low-engagement learners improving by 20%.

So, as opposed to making assessments a one-off event, they can be delivered at certain time periods to continuously refresh and assess employees’ knowledge. Assessments also help determine the type and frequency of training delivered for each individual on the Wranx platform. Using analytics with assessments Another advantage of using CBM assessments is that they safeguard organizations by exposing employee knowledge gaps that could potentially be harmful. Wranx’s performance analytics can visually demonstrate to organizations a learner’s accuracy against their confidence. We call this the ‘knowledge quadrant’ – this places each learner into a specific category, depending on the two variables: Aligned – Learners who are accurately answering and doing so with high certainty and will likely be applied correctly in practice. These individuals would be considered masters of that knowledge domain, resulting in higher performance and productivity.

Misaligned – Learners who are somewhat confident in their answers. The individual believes their answer to be correct, but an element of doubt exists that may cause them to not act on that knowledge. Uninformed – Learners who have gaps in their knowledge but are aware of it. They are uninformed and are unlikely to act. Misinformed – A ‘confidently incompetent’ employee. This individual has knowledge that they confidently believe to be correct, but which is actually incorrect. Those who have confidence in wrong information will very likely make mistakes on the job, which could potentially put the organization at risk. Being able to see what category employees fit into will help inform future training decisions for an organization. More focused, relevant learning content can be delivered and close any previous knowledge gaps employees may have – which results in more highly and broadly skilled workforces.


Success in the Workplace with Certainty Based Marking

While new methods of learning and extending knowledge retention are being developed all the time, knowledge assessments remain one of the key areas that can be improved. Naturally, it’s important to guarantee employees learn effectively, but the next step should be to assess how well they can apply their newfound knowledge in the workplace – both effectively and with confidence. Wranx offers a unique approach to employee assessment through its use of Certainty Based Marking. As explored above, this method requires those being assessed to indicate how certain they are of an answer before proceeding. Not only does this highlight the learner’s ability to recall specific information, but it also provides employers with an understanding of how likely their staff are to implement that knowledge in a workplace scenario. One of our clients who has successfully implemented CBM is The North Face®; they used this technique to assess their employees’ knowledge levels and see the measurable benefits of training with Wranx.

Following a month-long module, The North Face® saw 80% of their learners improve their CBM score, meaning they were answering more questions correctly and with greater confidence. Being a retail organization, The North Face® focus was primarily on delivering product knowledge training. With such a high volume of information on a wide range of products, it’s crucial that their employees are able to successfully recall key information when speaking with customers. Through the addition of CBM assessments, The North Face® gained a deeper insight into how confident their staff were and which areas needed improvement. In a retail environment, this confidence level can be the key factor in guaranteeing sales.

The Wranx Approach to CBM

For a learner to be truly confident in their knowledge, they must be able to use that knowledge in a real-life situation – where it counts most. There is empirical evidence to prove that effective training, which supports knowledge retention and boosts confidence, has a measurable impact on performance and can vastly improve an organization’s success in the long term. The inclusion of Certainty Based Marking assessments fits perfectly into a Wranx training program. By delivering a month-long module between a pre and post-assessment, employers are provided with detailed insight into their learners’ abilities to perform their duties correctly in the workplace. This is a powerful metric – one that helps to safeguard against unnecessary and easily avoidable mistakes. Most importantly, it demonstrates the true ROI of Wranx training.

Interested in learning more about Certainty Based Marking (CBM) and how it can enhance your training initiatives?


Read more about Wranx’s CBM feature here.

Make learning easier with the Wranx mobile app for Android and iOS

Make learning easier with the Wranx mobile app for Android and iOS

Following our introduction of Certainty Based Marking to the Wranx platform, we’re pleased to say that the platform is now available as a mobile app for both Android and iOS. If you’ve used the mobile version of our website, the Wranx mobile app will look very familiar. There are a number of new features that have been included to improve the learning experience too, though.


How mobile training can effectively train dispersed workforces in the gig economy

How mobile training can effectively train dispersed workforces in the gig economy

Gig Economy BikeThe Gig economy introduces a new way of working in modern society, characterised ‘by the prevalence of short-term contracts or freelance work, as opposed to permanent jobs.’ It is continuing to prove itself as an increasingly popular option for people seeking work, with an estimated 5 million employed within this area. Popular organisations that are involved within the gig economy include Uber, Airbnb, HomeAway and Deliveroo.


The rise of mobile

Ever since the advent of the Internet and the proliferation of mobile devices like smartphones and tablets, the way we access and acquire new information and knowledge has changed dramatically. This is especially true for millenials (otherwise known as Generation Y), who are taking advantage of technology to increase their business acumen through e-learning.

The marriage of technology and learning doesn’t look like it will disappear anytime soon either. In five years time, the digital natives that grew up with devices like the iPhone will come of age and enter the world of work. Therefore, it is fair to say that digital and mobile technologies are the future of workplace training.

As a result, workforces in years to come won’t feel like referencing a textbook or passively sitting in a classroom to boost their skillset. They could probably access much of this learning online anyway, so will want training to feel familiar and be enjoyable.

For these reasons, businesses must realise that traditional training methods won’t be effective for much longer. But how can you shift your organisation’s approach towards mobile learning (m-learning) if obstacles concerning implementation stand in the way?

The increasing adoption of mobile

According to the report Mobile Learning at Work by Towards Maturity, tomorrow’s generation of mobile-orientated workers will demand learning tools that they can relate to. While just 25 per cent of top learning companies are currently using mobile devices as a means to deliver learning and development content, many organisations are planning to follow suit in the near future.

What’s more, 47 per cent of managers report additional business benefits to mobile learning, which include improvements in organisational productivity and an increase in organisational revenue.

However, doubts still remain. Amit Garg, Director of Custom Learning Solutions at Upside Learning says “even though some large organisations have started using mobile technology to empower their workforce, for most others the question still remains – how do we actually use it in the workplace?”

The challenges relating to m-learning

For both formal and informal learning programmes, m-learning can provide access to content, encourage communication and collaboration, act as an alternative mode of delivery to personal computers, help with the application of new skills in the workplace, and deliver support when it is required.

Even so, L&D teams face a number of challenges in terms of introducing a new approach to learning. Obstacles include but are not limited to organisational culture and policy, getting buy-in from management at all levels, IT infrastructure and security issues, as well as a lack of skills and knowledge by the L&D team on how to introduce m-learning.

However, Towards Maturity states: “The barriers to success need not be insurmountable and it is certainly worth persevering with finding the right mobile solution. When these issues are successfully addressed the benefits both in terms of business results and in staff satisfaction and engagement are clear.”

Making the move over to m-learning

Instead of a drastic changeover of teaching methods and training techniques, most organisations will make the move to m-learning gradually at no fixed point in time. This is because the entire organisation will need to prepare for a self-directed learning approach, especially in cultures where formal face-to-face training is the norm.

Blended solutions are the key here, as engaging and interesting content can encourage thought as well as action. In turn, a cultural shift should eventually start to happen as the benefits of m-learning become clear.

However, it is important to keep moving forward with m-learning, due to the rapid rate in which technology changes. After all “mobile users are three times more likely to use cloud-based content than non-mobile users and using web 2.0 widgets to personalise learning environments,” according to Towards Maturity.

Possible m-learning approaches

Content delivery – The easiest and most straightforward way to implement m-learning is to make content available on smartphone screens. However, it might be more beneficial to favour tablets instead, as this can reduce the need to redesign courses specifically for smartphones, while content will be much easier to view and interact with. Mobile devices can also double up as content browsers or media players for the purposes of providing company information such as price lists, product manuals or job aids. While these resources aren’t unique to mobile, the speed of delivery alongside the ability to provide learning at the point of need is what makes smartphones and tablets so invaluable.

Informal learning – With the correct planning and careful design, there is no reason why the extensive capabilities and functionality of smartphones cannot be an intrinsic part of mobile learning apps, which increase learning interest, boost retention rates, and provide performance support for practical tasks. To give an example, accelerometers and motion sensors can be used to create immersive simulations or gamified scenarios. Then there is the option of incorporation social networking too, as mobile users are 50 per cent more likely to use learning communities such as action learning sets.

Bring Your Own Device (BYOD) – Towards Maturity reveals “1 in 2 organisations cite the wide variation in learners own personal technologies as a real barrier to implementation.” For this reason, 31 per cent are now providing learners with mobile devices that are compatible with the company’s IT architecture. But in the future, the increasing use of HTML5 will allow access to the same learning content from any device, regardless of its operating system or screen dimensions. However, due to security issues with BYOD, there needs to be a clear framework and policy that encompasses privacy concerns, terms of usage and access rights.

The rise of mobile

M-learning is something that organisations cannot ignore or expect to go away like other previous training fads. It might seem like several obstacles stand in the way of implementation, but these can be overcome gradually rather than all at once.

What’s more, by striking the right balance between content delivery, informal learning and BYOD, your employees will feel a lot more positive about training and achieve better learning outcomes too.

How to create a great content strategy for mobile

One of the reasons why e-learning took a little bit of time to be accepted as a genuine training solution is because it introduced new and novel technologies that had never been seen before in teaching environments. Some employers were rather sceptical as to whether it would work, while members of staff had to get used to the learning techniques on offer. But before long, e-learning was the training solution of choice for countless organisations.

Even so, it would be wrong to compare the integration of e-learning with the future potential of m-learning or mobile learning. This is because we are already familiar and au fait with mobile technology, as it dominates and dictates modern society.

Therefore, when it comes to adopting mobile learning as a training technique, you shouldn’t be looking for ways to get content onto smartphones and tablets. Instead, you must think of how to marry up learning and development with the digital lives of your employees.


Modern day mobile learning

Most people in today’s digitally connected society will probably say that their smartphone or tablet is extremely personal to them. Along with functionality to help the user go about their daily routine, mobile devices also feature applications that deliver the latest news stories, social media trends, and information about every subject imaginable.

But it is important to make the distinction between learning and finding things out. Therefore, a mobile learning strategy must abide by the consumption habits of mobile users and not force training upon them. Mobile learning should also provide users with content they actually want to receive.

This should involve the “micro-moments” that employees are experiencing on their mobile devices. Google defines these micro-moments as “I-want-to-go, I-want-to-do, I-want-to-buy, or I-want-to-know moments when people are turning to devices to find answers, discover new things, or make decisions.”

Although these micro-moments will differ depending on the learner’s position, from new hires to senior executives, it is fair to say that each and every staff member experiences them. For this reason, you can’t expect just one person or even a small team of employees to create relevant content for everyone.

Instead, the content creation process should be contextual, which changes to the wants and needs of the workforce over time. Inspiration for this approach can be taken from YouTube and its “CCC content framework.” This can help you to create, collaborate on, and curate content to produce appropriate learning materials and be there for employees in their various micro-moments of need.


Screen Shot 2015-07-07 at 11.45.46Create

The first aspect of the CCC framework is to create your own content. Think carefully about the I-want-to-do and I-want-to-know micro-moments your employers are experiencing and how to address them. Try to create content that captures the brand’s voice and embraces its core values.

Although your focus will be on educating employees, you should also try to make this content entertaining and inspiring. For example, a short video that addresses significant moments can get the audience’s attention and keep their interest levels high for the duration.

Several businesses are adopting this approach for induction days, performance management, selling tips, and other types of internal training.



The second aspect of the CCC framework is collaborating on your content with others. Along with broadening the relevance and reach of your mobile learning footprint, it also opens up the possibility of leveraging the experience and expertise of other creators.

These additional architects might be more obvious than you think too. For example, speak to your finance department to discover what content is needed to satisfy their micro-moments. This could involve processing payments, future forecasts or allocating budgets.

Across the business, approach employees from different departments and in different positions to discover what their most critical micro-moments are and how to solve them. They may even be willing to provide the content for your mobile strategy, as a recent study by Towards Maturity revealed that 84 per cent of respondents were willing to use technology to share what they know with their peers, which could include webinars or podcasts.



The third and final aspect of the CCC framework is to curate content that already exists. This can be anything from previous training materials, information on your company website, employee onboarding handbooks, or any other content that is easy to locate and curate.

The Internet is arguably the most important and extensive resource at your disposal. Use tools to find and filter relevant resources or give others the opportunity to point you towards valuable resources online.

After all, 87 per cent of people feel they should be able to communicate, share opinions, and interact with brands in real time. Although the same can be said for your organisation’s employees, any recommended content should still be referenced against an organisational context.


Piecing together a great content strategy for mobile

In contrast to e-learning, which required employers and employees to adapt to a new way of training, the prospect of mobile learning seems much easier to introduce and implement. However, it still presents a number of challenges, as organisations can’t get away with simply migrating existing training materials over to mobile devices.

Modern-day learners are using their smartphones and tablets to actively seek out the information they want to consume based on micro-moments of need. Therefore, training materials need to be available on mobile devices but also capitalise on the habits of their users.

But thanks to technology, this is a distinct possibility. In addition to creating content, which can include interactive infographics and entertaining videos, businesses have the ability to collaborate with others to identify what is required and transform existing materials to meet these needs. Make sure you are there for your employees’ micro-moments of need, enabling them to quickly and easily find contextually relevant content.