rebuilt
fbaeb2769f
scrape admin, enrollments, and staffing data
2026-03-05 13:20:21 -08:00
rebuilt
ddd20b2053
scrape admin, enrollments, and staffing data
2026-03-05 12:17:44 -08:00
rebuilt
1bf35012d0
update admin data
2025-12-04 20:41:41 -08:00
rebuilt
044a77cd3d
feat: pull down new admin data. Fix problems with some scrapers. For 5D-ii make sure the likert score is 1 if the nurse count is 0. Prevent devision by zero returning infinity. For 3A-ii make sure not to perform division by zero. Likert score will be 1 if the number of guidance couselors, psychologists, or paraprofessionals is zero.
2025-03-12 13:46:01 -07:00
nelson
ef9e5b7ebb
feat: Add scraper for a-curv-i7. Scrape new admin data for that item
2025-02-13 11:45:30 -08:00
nelson
83b426fbc9
feat: add scraper for 4D-ii
2025-02-06 21:09:56 -08:00
nelson
4c28b10bd4
feat: finish scraper for students of color in advanced courses
2025-02-06 13:08:04 -08:00
nelson
685bafd52c
Fix hs only admin data item. Use alternate method to check if a school is a high school
2025-02-06 13:07:05 -08:00
rebuilt
1de5d4d456
chore: fix problem with enrollment scraper
2025-01-27 15:13:25 -08:00
Nelson Jovel
28c6eb7fc0
chore: scrape new admin data; a-ovpe-i1
2025-01-17 18:32:00 -08:00
Nelson Jovel
2333e9c8c7
feat: add admin data item a-curv-i5 with temporary benchmark of 2
2025-01-15 16:32:22 -08:00
Nelson Jovel
33da0859b9
Split academic year into seasons if the academic year's range is
...
initialized with a season, i.e. "2024-25 Fall". Update scapers for
admin data, enrollment and staffing to use the new range standard
correctly. Update the loaders for admin data, enrollment and staffing
so that it populates all seasons in a given year. So admin data for
2024-25 gets loaded into "2024-25 Fall" and "2024-25 Spring". Add tests
for the new range format. Set the default cutoff for the start of Spring season will be the last Sunday in February
2024-04-25 09:21:04 -07:00
5789ebf564
Faster admin data loader + rename School.school_hash
2024-04-22 14:46:37 -04:00
04d61c0b1a
Memoize admin data loader
2024-04-18 17:13:03 -04:00
rebuilt
5c7729beeb
feat: if admin data value is above 5, round down to 5
2023-07-21 12:14:46 -07:00
rebuilt
cbd5687ff0
feat: Add out of state admin data
2023-07-20 17:06:07 -07:00
rebuilt
4509c157fa
Add automated data cleaning. Modify SurveyItemValues class to use regex
...
instead of hard coded values. Produce a clean csv and a csv with all
the removed values and columns with reason for removal. Add script for
running cleaning for each project
2023-05-16 13:38:29 -07:00
rebuilt
359e266a6c
Remove unused TODOs
2023-04-27 15:47:45 -07:00
rebuilt
3f2a7dff50
Fix problem with dese scraper lumping in 2021-22 data as 2022-23 data.
...
Deleted unused csvs. Turned off puts statements in admin loader.
Remove old, now unused admin data loader class.
2023-04-27 15:43:17 -07:00
rebuilt
10cedd2310
Remove unused filename
2023-04-20 17:42:27 -07:00
rebuilt
5b2bae0d92
Fix ThreeATwo scraper
2023-04-19 19:20:54 -07:00
rebuilt
128748addd
Update logic for calculating student response rate. Remove references
...
to survey table. We no longer check or keep track of the survey type.
Instead we look in the database to see if a survey item has at least 10
responses. If it does, that survey item was presented to the respondent
and we count it, and all responses when calculating the response rate.
Remove response rate timestamp from caching logic because we no longer
add the response rate to the database. All response rates are calculated
on the fly
Update three_b_two scraper to use teacher only numbers
swap over to using https://profiles.doe.mass.edu/statereport/gradesubjectstaffing.aspx as the source of staffing information
2023-04-18 13:59:29 -07:00
rebuilt
4c4ccc01cc
Merge branch 'rpp-response-rate' to bring in changes to test files
2023-03-15 15:00:25 -07:00
rebuilt
06f9d2f0e9
Scrape enrollment and staffing information. Seed enrollment and staffing information. Update DatabaseCleaner so it cleans up leftover information in the database. Remove old admin csvs from codebase.
2023-03-05 16:15:35 -08:00
rebuilt
c0332955f3
move csv require statement to application.rb
2023-02-27 15:47:20 -08:00
rebuilt
69c089d9bb
Remove occurances of MCIEA
2023-02-13 17:58:27 -08:00
rebuilt
ed421296c7
Fix bug that caused an incorrect score of 1 to be assigned for admin
...
data item related to parity index of teachers of color to students of
color. Now if there's not enough data to calculate a score, it produces
'NA' instead of 1. Also added admin data for a-reso-i1 and a-curv-i1
2023-01-06 17:43:48 -08:00
rebuilt
7b9abe3793
Scrape latest data from DESE
2022-12-20 15:26:11 -08:00
rebuilt
f4e2e6ca83
Correct headers for 4D so the data can be correctly imported
2022-10-21 17:29:19 -07:00
rebuilt
dd9ed86dcd
Merge branch 'scrapers'
2022-09-21 19:54:24 -07:00
rebuilt
f556844b73
Update a-sust-i1 so it's only applicable to high schools and lower
...
benchmark for a-cure-i1 from 6.7 to 5 percent.
2022-09-21 19:50:33 -07:00
rebuilt
9e82023e48
Make sure we don't divide by 0
2022-09-20 16:44:11 -07:00
rebuilt
1a984d2cf0
Make sure we don't divide by 0
2022-09-20 16:43:19 -07:00
rebuilt
3ee909bbea
Update 3B-2 with an absolute floor
2022-09-20 14:45:20 -07:00
rebuilt
1be6e79fd8
Add scrapers for admin data and their corresponding csv files of data.
...
Make sure specs are off so they don't run in github CI
2022-09-19 19:37:39 -07:00
rebuilt
9e18bf2151
Turn off specs for scrapers. Generate csvs for scraped data
2022-09-13 22:59:04 -07:00
rebuilt
75cea539c5
Change scraped csv file names from word to number format 'one_a_one.csv'
...
-> '1A_1.csv''
2022-09-13 10:46:18 -07:00
rebuilt
3e28be2d0d
Remove 'scraper' from file/class names add 3B-1
2022-09-12 11:24:57 -07:00
rebuilt
83ef9310a4
Add scraper for 3A-1 and 2C-1. Finishes #183105787 and #183105716
2022-09-07 11:34:42 -07:00
rebuilt
b5bc9c9de4
Add 2A-i admin data. Finishes #183105627
2022-09-05 22:37:53 -07:00
rebuilt
459393381b
Load admin data for 1A-iii. Finishes #183105516
2022-09-05 21:01:32 -07:00
rebuilt
f783ce9724
add one-a
2022-09-01 20:00:20 -07:00
rebuilt
048aacd641
Add admin data for 4D. Finishes #182818489 and #182793402
2022-08-30 21:46:23 -07:00
