Educause Security Discussion mailing list archives
Re: Effectiveness of CTF Scoreboards
From: Valerie Vogel <vvogel () EDUCAUSE EDU>
Date: Fri, 22 Mar 2019 15:45:50 +0000
Hi Don, I would recommend contacting the folks at the Virginia Cyber Range: https://virginiacyberrange.org/ (other states have similar efforts – like MERIT’s cyber range in Michigan: https://www.merit.edu/cyberrange/). David Raymond and Thomas Weeks will be presenting a session: Using Capture the Flag for Cybersecurity Education and Training at our Security Professionals Conference on May 13 in Chicago: https://events.educause.edu/special-topic-events/security-professionals-conference/2019/agenda/using-capture-the-flag-for-cybersecurity-education-and-training-separate-registration-is-required – and they might be aware of studies Here is a list of a few upcoming events in Virginia, as well: https://virginiacyberrange.org/events Also, this blog on CTFs and the National Cyber League may be of interest to you: https://er.educause.edu/blogs/2017/11/how-to-develop-cybersecurity-athletes Thank you, Valerie Valerie Vogel Senior Manager, Cybersecurity Program EDUCAUSE Uncommon Thinking for the Common Good direct: 202.331.5374 | Follow HEISC on LinkedIn<https://www.linkedin.com/showcase/higher-education-information-security-council-heisc-/> | twitter: @HEISCouncil | vvogel () educause edu<mailto:vvogel () educause edu> From: Security Discussion Group List <SECURITY () LISTSERV EDUCAUSE EDU> on behalf of Bryce Porter <0000008467c11b85-dmarc-request () LISTSERV EDUCAUSE EDU> Reply-To: Security Discussion Group List <SECURITY () LISTSERV EDUCAUSE EDU> Date: Friday, March 22, 2019 at 7:15 AM To: Security Discussion Group List <SECURITY () LISTSERV EDUCAUSE EDU> Subject: Re: [SECURITY] Effectiveness of CTF Scoreboards The folks at MAGIC, Inc. might have some thoughts to share: https://magicinc.org/<https://nam03.safelinks.protection.outlook.com/?url=https%3A%2F%2Fmagicinc.org%2F&data=02%7C01%7C%7C7e6aee74910d4c1cd0b508d6aed0cd98%7Cdd4b037fe626495db0170cc0f7dddb37%7C0%7C0%7C636888609168526745&sdata=ooe4MVz9%2BIlY4s0MUi4AcRb418vQ1zsIUpfUOYqyi%2FI%3D&reserved=0> They put on a semi-annual (twice each year) multi-site CTF competition for education and municipal/community locations with a scoreboard that shows team scores and site scores. I believe their next event (CTF007 - https://magicinc.org/event/capture-the-flag-007<https://nam03.safelinks.protection.outlook.com/?url=https%3A%2F%2Fmagicinc.org%2Fevent%2Fcapture-the-flag-007&data=02%7C01%7C%7C7e6aee74910d4c1cd0b508d6aed0cd98%7Cdd4b037fe626495db0170cc0f7dddb37%7C0%7C0%7C636888609168536753&sdata=lnSqx%2B3TKZzNfvGIHOsaJ09%2BeCZ2zFAdjuP4xnsakQk%3D&reserved=0>) is on April 13th, and there may be a site that is close enough for you to attend and observe. They may also be willing to share their scoreboard technology with you. We took part in the November CTF event last year, and I observed it to be very well assembled and managed. We will likely participate again next year, but we are skipping the Spring event this year due to other priorities for our program. Bryce Porter Chief Information Security Officer Information Technology Services UNC Greensboro On Fri, Mar 22, 2019 at 9:33 AM Don Murdoch <dmurdoch () regent edu<mailto:dmurdoch () regent edu>> wrote: Greetings, TL;DR short question: Is there a study that can be pointed to that shows “CTF style user interfaces have positive measurable impact on knowledge acquisition and learning permeance”? On a related question, is anyone aware of a CTF event coming up in the next 3-4 months within a 150 mile radius of the Hampton Roads, VA region, so perhaps I could design a study and actually measure effectiveness? Long Question w/ background: I’ve been searching a bit for a study that measures the effectiveness of a CTF style scoring system, with the goal of measuring effectiveness of using the CTF UI tool itself on the adult learner. The phrase >> Effectiveness of CTF Scoreboards site:*.edu << in Google finds some really nice papers that explain “this is what we did, how we collected, here is the amazing infrastructure we built to support asynch decentralized competition, lessons learned,.etc.”, but I haven’t seen an answer to the question that measures if the CTF game tool (such as CTFd or the FaceBook tool) had a measurable positive impact on learning – and more importantly, an improvement on fact data and process to solve recall 30d after the event. I often work with a Senior High teenager in a local NJROTC unit, where AFCEA competition is all the rage, and I have anecdotally observed that some things “he gets”, some things “he gets are hard”, and some things “he should have got he did not”, and more importantly, there is variable results in the coaching I’ve offered him in his ability to apply a lesson in a CTF. This is highly anecdotal and and an error prone observation, but it does prompt the question “does the CTF tool and environment measurably improve performance”, and then “how can we measure performance 30d later”? I’d say that there is certainly anecdotal evidence that people “like CTF’s”, that “CTF’s help provide a score”, and that organizations like SANS have used the NetWars platform to great effect. After having done one myself, it was enormously satisfying to see that my team got 510 of the 511 points 42 minutes before the next team, that we had less people, and we had more people depart mid game. That’s “cool”, and it’s a permanent memory. However, that “feeling” was not measured for effectiveness in a longitudinal manner. On a related question, is anyone aware of a CTF event coming up in the next 3-4 months within a 150 mile radius of the Hampton Roads, VA region, so perhaps I could design a study and actually measure effectiveness? My initial thought is that you would want to measure fact knowledge ahead of time, emotional and cognitive impact right after the event so as not to disrupt the event, and then measure knowledge permeance 7d and 30d later by asking a question using the same UI elements and measuring the analysis and response time. (following the philosophy expressed in Make It Stick). Don Murdoch, GSE #99 Assistant Director, Institute for Cybersecurity Direct: US 757 352 4588 Regent University<https://nam03.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.regent.edu%2F&data=02%7C01%7C%7C7e6aee74910d4c1cd0b508d6aed0cd98%7Cdd4b037fe626495db0170cc0f7dddb37%7C0%7C0%7C636888609168536753&sdata=c6VknJU6XRnF%2BJG4eEWUwZQFkybrete2v0lGeK%2FF5%2BM%3D&reserved=0> | Christian Leadership to Change the World New Book “Blue Team Handbook: SOC, SIEM, and Threat Hunting Use Cases: A condensed field guide for the Security Operations team (Volume 2)” is now Live on Amazon<https://nam03.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.amazon.com%2Fgp%2Fproduct%2F1726273989%2Fref%3Doh_aui_detailpage_o00_s00%3Fie%3DUTF8%26psc%3D1&data=02%7C01%7C%7C7e6aee74910d4c1cd0b508d6aed0cd98%7Cdd4b037fe626495db0170cc0f7dddb37%7C0%7C0%7C636888609168546761&sdata=%2FgaVB1manbbozuhZQBY2oTCBNmfw8IjkWIErSzIbJPs%3D&reserved=0>
Current thread:
- Effectiveness of CTF Scoreboards Don Murdoch (Mar 22)
- Re: Effectiveness of CTF Scoreboards Bryce Porter (Mar 22)
- Re: Effectiveness of CTF Scoreboards Valerie Vogel (Mar 22)
- Re: Effectiveness of CTF Scoreboards Bryce Porter (Mar 22)