Effectiveness of CTF Scoreboards

[Don Murdoch <dmurdoch () REGENT EDU>]

Greetings,

              TL;DR short question: Is there a study that can be pointed to that shows "CTF style user interfaces have 
positive measurable impact on knowledge acquisition and learning permeance"? On a related question, is anyone aware of 
a CTF event coming up in the next 3-4 months within a 150 mile radius of the Hampton Roads, VA region, so perhaps I 
could design a study and actually measure effectiveness?

              Long Question w/ background:

              I've been searching a bit for a study that measures the effectiveness of a CTF style scoring system, with 
the goal of measuring effectiveness of using the CTF UI tool itself on the adult learner. The phrase >> Effectiveness 
of CTF Scoreboards site:*.edu << in Google finds some really nice papers that explain "this is what we did, how we 
collected, here is the amazing infrastructure we built to support asynch decentralized competition, lessons 
learned,.etc.", but I haven't seen an answer to the question that measures if the CTF game tool (such as CTFd or the 
FaceBook tool) had a measurable positive impact on learning - and more importantly, an improvement on fact data and 
process to solve recall 30d after the event.

              I often work with a Senior High teenager in a local NJROTC unit, where AFCEA competition is all the rage, 
and I have anecdotally observed that some things "he gets", some things "he gets are hard", and some things "he should 
have got he did not", and more importantly, there is variable results in the coaching I've offered him in his ability 
to apply a lesson in a CTF. This is highly anecdotal and and an error prone observation, but it does prompt the 
question "does the CTF tool and environment measurably improve performance", and then "how can we measure performance 
30d later"?

              I'd say that there is certainly anecdotal evidence that people "like CTF's", that "CTF's help provide a 
score", and that organizations like SANS have used the NetWars platform to great effect. After having done one myself, 
it was enormously satisfying to see that my team got 510 of the 511 points 42 minutes before the next team, that we had 
less people, and we had more people depart mid game. That's "cool", and it's a permanent memory. However, that 
"feeling" was not measured for effectiveness in a longitudinal manner.

              On a related question, is anyone aware of a CTF event coming up in the next 3-4 months within a 150 mile 
radius of the Hampton Roads, VA region, so perhaps I could design a study  and actually measure effectiveness? My 
initial thought is that you would want to measure fact knowledge ahead of time, emotional and cognitive impact right 
after the event so as not to disrupt the event, and then measure knowledge permeance 7d and 30d later by asking a 
question using the same UI elements and measuring the analysis and response time. (following the philosophy expressed 
in Make It Stick).

Don Murdoch, GSE #99
Assistant Director, Institute for Cybersecurity Direct: US 757 352 4588
Regent University<https://www.regent.edu/> | Christian Leadership to Change the World
New Book "Blue Team Handbook: SOC, SIEM, and Threat Hunting Use Cases: A condensed field guide for the Security 
Operations team (Volume 2)" is now Live on 
Amazon<https://www.amazon.com/gp/product/1726273989/ref=oh_aui_detailpage_o00_s00?ie=UTF8&psc=1>