Developing AI Literacy for Academic Success - Canvas Course

Students in Higher Education face many challenges these days.  From navigating college life to staying on top of course requirements, to learning the materials needed for academic success.  The onset of various Large Language Models (LLMS) and Generative Artificial Intelligence (Gen AI) has served to level the playing field, making it easier for students to successfully navigate their higher education career.  At the same time, with the introduction of tools like ChatGPT and other Gen AI tools, students are increasingly using AI to offload their cognitive abilities.  As a result, students are not developing the necessary skills that an undergraduate degree would help them develop.  Instead, their educational growth in many ways is being stunted by these new tools.  If left unaddressed, the use of Gen AI by students could severely undermine their academic success, and future career success. 

Additionally, many students are hesitant to use AI, for fear of plagiarism or not understanding how to use it correctly.  These students are being provided a disservice because AI is continually shaping both the educational atmosphere as well as the outside world. 

With this in mind it became clear that students needed to learn how to utilize Gen AI for their academic career.  To help address the problem of generative AI overuse, and lack of AI understanding, the course we set out to develop a course that would help students develop basic AI literacy.

NEEDS, TASK, AND LEARNER ANALYSIS

To help us to be able to analyze how far reaching the problem was, we began with a Need Analysis. Needs Analysis aim at helping Instructional Designers to know what the needs of the stakeholders are (in these cases the students and the academic institutions), while also addressing whether instruction is the best means for implementing the desired outcome.

In our Needs Analysis, we recognized the above problem of AI overuse and also lack of AI literacy, while at the same time understanding how those in Higher Education (our stakeholders) were viewing the use of AI by students. 

The Needs Analysis highlighted that instruction was the most appropriate means of addressing the lack of AI literacy.  Additionally, the Needs Analysis helped to address unseen issues. For instance, our Needs Analysis highlighted that although institutions are quickly working on coming up with rules and regulations regarding the use of generative AI in Higher Education, these were often reactionary responses to perceived threats (student’s cheating, liability issues, etc.).  The Needs Analysis helped us to understand that a more productive means of tackling the issue of student’s lack of AI Literacy, and thus instruction was the best choice for implementing learning.

The next step we utilized was the creation of the Task and Learner Analysis in creating student personas to better understand our learners.  Because the course had not yet been created, or tried, we decided to utilize generative AI to render basic foundational demographic knowledge of our learners.  In this way, we were able to humanize the participants, while also painting a picture of our target demographic’s specific needs. 

Some of the aspects that came up were conducive to our previous expectations.  For example, most students had had experience with AI but lacked formal training on how to verify generative AI output.  Additionally, the Task and Learner Analysis made clear the specific challenges and  steps that students would need to take to be able to gain measurable understanding of AI literacy.  They helped to understand the academic pressures and concerns that the students were facing.  These ranged from desire to implement generative AI while being concerned about cheating, to not understanding how to cite and vet answers.  By understanding our learners, we were able to better come up with specific measurable instructional goals and objectives for our modules.

INSTRUCTIONAL GOALS AND OBJECTIVES

With a strong understanding of our learners in mind, it came time to put together the specific measurable Goals and Objectives that we wanted the students to walk away with.  While the overarching goal was to have students move from knowing about AI to being able to critically use AI, we broke the steps down into four scaffolded goals:

1. Understand the Evolution and Mechanics of AI.

2. Master AI Tool Literacy and Prompt Engineering.

3. Navigate Ethics, Bias, and Academic Integrity.

4. Apply AI Learning to Discipline Specific Integration.

Within each goal was a subset of specific measurable objectives.  These objectives allowed for us to be able to quantify the student’s learning.  

Each goal with its objectives were presented as a module, which each module building off of the previous one.  In this way we were able to use a scaffolded approach for moving the students from “knowing about AI” to “knowing how to use AI responsibly”.  The modules provided the following structure:

1.  Foundational Knowledge: History and Mechanics (Module 1).

2.  Practical Skill sets: Prompting and Tool Usage (Modules 2 & 3).

3.  Critical Evaluation: Limitations, Ethics, and Privacy (Modules 4 & 5).

4. Synthesized Application: Field-Specific Problem Solving (Module 6).

Each module also challenged students to be able to exhibit specific measurable behaviors, ranging from effective prompting to fact-checking AI, to anonymizing data, to citing AI assisted work, and finally to being able to apply the specific behaviors their specific field. 

INSTRUCTION AND INSTRUCTIONAL ACTIVITIES

As detailed in the Needs Analysis, instruction was determined to produce the necessary change in the student behavior.  After having determined our goals and how the objectives would lead to specific measurable changes in students, we began the exciting part of instructional activities for each of the four classifications above.

Our group decided to divide the instructional modules, with each of us focusing on the specific areas that we found most interesting.  For my portion of the project, I chose to focus on the Critical Evaluation, Module 4 and 5, highlighting the limitations, biases and ethics of AI, along with how students could effectively use AI while at the same time protecting their privacy.

To help with the creation of the course, various generative AI models were utilized to create specific, measurable course content, which were then utilized for brainstorming the different quizzes, and assignments that would be used to verify students were actually understanding and implementing the information.  In so doing we wanted to make sure our activities were a balance of symbolic, iconic, and enactive experience as posited by Dale’s Cone of Experience.  

Our initial attempt at creating curriculum was a bit of a mess, before we settled on the scaffolded method of learning that allowed us to be able to implement learning activities that would teach, but also move students from being passive users to active participants with AI.  Once we had determined the scaffolded approach, we determined that each module would consist of one content page, one discussion post, and one major quiz or assignment.  This helped to reduce redundancy and make the modules flow more effectively and also build on each other.  Additionally, our module’s structure allowed us to address Bloom’s Taxonomy of Learning, moving learners from remembering and understanding, to applying and analyzing, and finally evaluating and creating their own learning experiences.

EVALUATION AND DESIRED OUTCOMES

One of the benefits of simplifying the Instructional Activities was a reduction in assignments and variation in metrics of course consistency.  By moving away from having 22-25 various assignments to 12 assignments that scaffolded onto each other, we were able to embrace Kirkpatrick’s Four Levels of Learning. 

Kirkpatrick’s Four levels of Evaluation allowed us to move from Formative Evaluation (Knowledge and Comprehension) with modules 1 and 2 focusing on measuring learner’s reaction to AI and ability to recall facts.  In module’s 3, 4, and 5, we moved to Performance-Based Evaluation (Behavior and Application) where students were challenged to not only know what hallucinations were but also to be able to identify them.  Additionally they were able to learn the structural biases inherent in AI and how to anonymize their data.  The final summative evaluation experience focused on a synthesis of the specific measurable behaviors and skills (prompting, ethical considerations, field specific knowledge, and creating a professional solution).

To be able to measure the desired outcomes, each of the modules when paired together provided students with the necessary skills to move through Kirkpatrick’s Four Levels of Evaluation (Reaction, Learning, Behavior, and Results).  In so doing, each module provided the metrics that highlighted how students were not only learning but applying the skills sets that were determined in the learner Goals and Objectives.

Concluding Thoughts

This case study highlights a means of bridging the digital divide in AI literacy among first-year college students.  To accomplish this task, we began by conducting preliminary research through a Needs, Task, and Learner Analysis, which helped us address the needs of our stakeholders (Higher Education institutions).  This in turn lead to a comprehensive understanding of who are learners were, which in turn helped us to be able to create user personas that painted out the specific needs and challenges that our learners were facing.  With the students personas in mind, we were able to determine the specific measurable goals and objectives for the project, this in turn led us to create a Canvas course consisting of 6 modules that took students from being AI illiterate to being able successfully utilize AI for a case-study.  The modules consisted of scaffolded learning activities including quizzes, assignments, discussions, and a conclusive research paper.  Lastly the project was able to make use of Kirkpatrick’s 4 Levels of Evaluation to evaluate the specific measurable changes that had been implemented through the process.

Although there were many challenges to being able to create a cohesive Canvas course, we were able to create a meaningful learning experience that not only helped teach how to create curriculum but also highlighted how to be able to build out courses that were based in learning and foundational Instructional Design steps.  By utilizing the course both our stakeholders (Higher Education Institutions) and our learners (first-generation students) are able to learn the benefits and the challenges that come with the rise generative AI and be able to utilize their learning in their academic career.