31e9779deb
chore: ECP-152 Reduce batch size of survey uploads to 100 at a time from 500 to reduce memory usage and also explicitly run garbage collection so we stay under the 500 mb limit of heroku
7 months ago
b0df611f4d
feat: create a parents by language graph
...
Update demographics table with lanugage options
Create a lanugage table to hold the new languages
Update the demographic loader to input languages into the database
Update the cleaner to read the language column
Update the parent table to hold a reference to a language
Update the data uploader script to read the language from the csv and update the language information for any parent items that already exist (or create database entries if none already exist)
update the analyze interface to add controls for selecting ‘parents by group’ and a dropdown for ‘parent by language’
Update the analyze controller to read the parent-by-group parameter
Create a graph for the parent-by-group view
Bubble up averages for language calculations.
Make sure n-size only counts responses for a given measure.
7 months ago
1845d1ac7b
feat: Add language filter for parent analysis
8 months ago
8e5e2a030a
feat: Add language filter for parent analysis
8 months ago
e90e6a4f0f
WIP: add multiple languages to parent
8 months ago
446b3b1096
WIP: Update the parent table to hold a reference to a language
...
Update the data uploader script to read the language from the csv and update the language information for any parent items that already exist (or create database entries if none already exist)
8 months ago
6503d6444f
chore: correctly detect missing grade header
9 months ago
204e9c1b3a
feat: pull down new admin data. Fix problems with some scrapers. For 5D-ii make sure the likert score is 1 if the nurse count is 0. Prevent devision by zero returning infinity. For 3A-ii make sure not to perform division by zero. Likert score will be 1 if the number of guidance couselors, psychologists, or paraprofessionals is zero.
9 months ago
9470e55701
feat: Add scraper for a-curv-i7. Scrape new admin data for that item
10 months ago
5235ad8c6a
chore: rename sftp environment variable
10 months ago
29caf0904f
feat: add scraper for 4D-ii
10 months ago
7c38d3faee
feat: finish scraper for students of color in advanced courses
10 months ago
966e25c995
Fix hs only admin data item. Use alternate method to check if a school is a high school
10 months ago
38cf38fdb5
chore: fix problem with enrollment scraper
10 months ago
b9a0016f14
chore: scrape new admin data; a-ovpe-i1
11 months ago
8ab944acca
feat: add admin data item a-curv-i5 with temporary benchmark of 2
11 months ago
8303d3bfdc
feat: add esp counts when calculating teacher participation rates on overview page
11 months ago
67a9a96433
fix: make sure all likert scores get counted even when the survey item id has different capitalization. Add tests for uploading parent data. Change the parent response rate calcuation to count all students in the school instead of just for the grades that were given the student survey
1 year ago
d338003886
Show parent response rate
1 year ago
c85ddddd8d
feat: add parent survey gauges
1 year ago
22b9b44773
chore: add another variant for column name
1 year ago
66dda0d30d
chore: During cleaning, stop execution if grade column isn't found. Also stop execution if a duplicate header is found. Turn off spec for duplicate header check
2 years ago
8d31faf012
chore: speed up survey data loader
2 years ago
62a7fbb4fc
chore: Add 'English Learner' and 'DirectCert' as valid column headers
2 years ago
5a1db34a18
chore: add a way to print out errors in dese id when cleaning files
2 years ago
d4db4e0fa2
add disaggregation glossary
2 years ago
5851ab1cbf
chore: add parsing rules for disaggregation columns; ell, income, sped. Add tests for new rules
2 years ago
6a9d04f7ff
fix: make sure enrollment loader works
2 years ago
ac5c1edfbf
feat: Split academic year into seasons if the academic year's range is
...
initialized with a season, i.e. "2024-25 Fall". Update scapers for
admin data, enrollment and staffing to use the new range standard
correctly. Update the loaders for admin data, enrollment and staffing
so that it populates all seasons in a given year. So admin data for
2024-25 gets loaded into "2024-25 Fall" and "2024-25 Spring". Add tests
for the new range format. Set the default cutoff for the start of Spring season will be the last Sunday in February
2 years ago
Gabe Farrell
b3e6efdb2e
Faster admin data loader + rename School.school_hash
2 years ago
Gabe Farrell
b5f90376f7
Memoize admin data loader
2 years ago
3c3b0d2cf5
fix: loosen restrictions on date format
2 years ago
bd1a7b3877
match an additional format for Dates. Supported dates are now '1/10/2022 14:21:45' '2022-1-10T14:21:45' '2022-1-10 14:21:45'
2 years ago
c4d8b927af
feat: Support two date formats: ISO 8601 and the standard US date format
...
used in google sheets
2 years ago
7d527b5639
chore: support alternate format for recorded date
2 years ago
8d596faeeb
chore: alphabetize districts when writing filename for cleaned files
2 years ago
c6c9fd38de
fix: batch in smaller size to stay within memory limits
2 years ago
e97e7047b1
fix: no longer support 'form' in filename when cleaning. Only look for 'part X' and add that to the filename if it exists
2 years ago
e1fc5dbbaa
feat: add 2023-24 academic year and make sure previous year enrollment
...
and staffing data get loaded when missing
2 years ago
2a125aa058
chore: bring in some refactoring changes that didn't get copied over from the main branch
2 years ago
75451648e5
chore: move code that normalizes race data to the Race class
2 years ago
e7f06ef4e9
chore: remove lowell seeding rules
2 years ago
bbefdcb5bb
chore: remove student loader since we load race information in data loader
2 years ago
063810a9d7
chore: make sure to load races in data loader instead of student loader
2 years ago
381625f58b
chore: reenable test spec that tests data loader for races
2 years ago
317fe3239a
chore: refactor Race out of survey_item_values
2 years ago
a295b8afb9
chore: refactor Gender out out of survey_item_values row
2 years ago
0359dae88a
chore: rename counts_by_grade to enrollment_by_grade for clarity
2 years ago
ee5e53f992
chore: Make sure 'hispanic' column only gets applied when using SIS race information
2 years ago
6f265302ea
feat: if the filename includes the words 'form' or 'part' add that to the resulting cleaned filename
2 years ago
rebuilt
Gabe Farrell