Commit graph

7 commits

Author SHA1 Message Date
rebuilt
ce76c979a4 Add scraper for 3B-i student/#courses ratio 2023-06-05 11:39:08 -07:00
rebuilt
0dfc9726d0 Add automated data cleaning. Modify SurveyItemValues class to use regex
instead of hard coded values.  Produce a clean csv and a csv with all
the removed values and columns with reason for removal. Add script for
running cleaning for each project
2023-05-24 11:59:53 -07:00
rebuilt
f1022728fa Fix problem with dese scraper lumping in 2021-22 data as 2022-23 data.
Deleted unused csvs.  Turned off puts statements in admin loader.
Remove old, now unused admin data loader class.
2023-04-30 12:04:20 -07:00
rebuilt
07ed8dd259 Update logic for calculating student response rate. Remove references
to survey table.  We no longer check or keep track of the survey type.
Instead we look in the database to see if a survey item has at least 10
responses.  If it does, that survey item was presented to the respondent
and we count it, and all responses when calculating the response rate.

Remove response rate timestamp from caching logic because we no longer
add the response rate to the database. All response rates are calculated
on the fly

Update three_b_two scraper to use teacher only numbers

swap over to using https://profiles.doe.mass.edu/statereport/gradesubjectstaffing.aspx as the source of staffing information
2023-04-22 14:00:20 -07:00
rebuilt
1be6e79fd8 Add scrapers for admin data and their corresponding csv files of data.
Make sure specs are off so they don't run in github CI
2022-09-19 19:37:39 -07:00
rebuilt
3e28be2d0d Remove 'scraper' from file/class names add 3B-1 2022-09-12 11:24:57 -07:00
rebuilt
83ef9310a4 Add scraper for 3A-1 and 2C-1. Finishes #183105787 and #183105716 2022-09-07 11:34:42 -07:00