Student Progress Scraper for Code.org Courses

Code.org is a non-profit organization led by Hadi and Ali Partovi that aims to encourage people, particularly school students in the United States and beyond, to learn computer science. Read more in Wikipedia. It features lots of tutorials that teach the basics of computing through puzzles and fun activities.

Code.org also has a function which lets teachers create sections for students to register for taking particular tutorial, and monitor their progress through a dashboard. However the dashboard does not provide a functionality for easily downloading the student progress. There was a long overdue request to Code.org team to resolve this deficiency.

See one discussion at the Code.org Professional Learning Community Forums How to download student progress? that started in 2016. The last response in the thread was in October 2021 from josh.schulte (Senior Program Manager @ Code.org):

Hello @ayusubov and others! You’re far from alone in this request for a print-friendly / exportable or otherwise easily-sharable progress report, and our team has work tracked to address this in future updates. Reports and comments like those in this thread make our site and offerings better for everyone, and allow us to better prioritize our work, so thanks again for chiming in here!

Our Solution

A quick research revealed that the student progress dashboard pages in HTML have a strange behavior:

• student data and their progress data is stored separately in the page
• progress data is downloaded portion-by-portion while you scroll down the list
• if you do not save the page as a complete web page, you do not get this data included in the .html file.

You will have to prepare several .html files of saved from Teacher Dashboard section progress page (view by Levels) for different positions of scrolling in the list. A small Python script by @nsadili will scan these .html files, extract student records data, then match it with progress data and save it as a .csv file.

User Manual

1. Place the downloaded .html files under progress_files folder. Delete the .keep file in the folder.
2. Run solver with the command line arguments:

   python solver.py "SECTION_NAME"

   Example:

   python solver.py "SITE 1101: Homework 1 (Fall 2022)"

3. The user_progress_report.csv file will hold the processed results of each student.

Python Guide

If you have never installed Python on your machine, you will need to download and install Python (preferrably 3.x version).

1. Visit Python Downloads.
2. Download installer for your OS.
3. Run the setup to install. (Check the "Add Python 3.x to PATH", especially for Windows).
4. After installation is complete, check if it went well:
   • Open Terminal (Command Line or PowerShell for Windows)
   • Execute the following command:

     python --version

     This should show the current installed version of Python.
   • If it shows the version proceed with running the application shown in the User Manual.
   • If it cannot find python check the installation.
5. This script does not work without Beautiful Soup, which is a library that makes it easy to scrape information from web pages.
   • To instal Beautiful Soup execute the following command:

     pip install beautifulsoup4