Commit graph

30 commits

Author SHA1 Message Date
Nelson Jovel
b4c4522956 fix: tests 2024-11-26 10:18:17 -08:00
Nelson Jovel
e741b60bec fix: make sure all likert scores get counted even when the survey item id has different capitalization. Add tests for uploading parent data. Change the parent response rate calcuation to count all students in the school instead of just for the grades that were given the student survey 2024-11-19 19:06:13 -08:00
Nelson Jovel
ee41751f4e chore: correct parsing for 'not sped' and 'lep not first year' 2024-06-26 12:03:31 -07:00
Nelson Jovel
c4d4c35766 update parsing rules from glossary 2024-05-15 10:54:12 -07:00
Nelson Jovel
8e7fbdfb2c add disaggregation glossary 2024-05-15 10:53:59 -07:00
Nelson Jovel
0a27538747 chore: add a test for categorizing sped values of 1 and 0 as 'Special Education' and 'Not Special Education' 2024-05-07 19:38:26 -07:00
Nelson Jovel
65c25fc3c7 Add ell income and sped parsing rules for SIS data. Add tests for the
new inputs.
2024-05-07 17:18:59 -07:00
Nelson Jovel
2561fa28fc feat: Split academic year into seasons if the academic year's range is
initialized with a season, i.e. "2024-25 Fall".  Update scapers for
admin data, enrollment and staffing to use the new range standard
correctly.   Update the loaders for admin data, enrollment and staffing
so that it populates all seasons in a given year.  So admin data for
2024-25 gets loaded into "2024-25 Fall" and "2024-25 Spring".  Add tests
for the new range format.  Set the default cutoff for the start of Spring season will be the last Sunday in February
2024-04-27 14:05:02 -07:00
Nelson Jovel
67ffc996a8 Revert "Split academic year into seasons if the academic year's range is"
This reverts commit a5d4cccb37.
2024-04-26 13:48:30 -07:00
Nelson Jovel
a5d4cccb37 Split academic year into seasons if the academic year's range is
initialized with a season, i.e. "2024-25 Fall".  Update scapers for
admin data, enrollment and staffing to use the new range standard
correctly.   Update the loaders for admin data, enrollment and staffing
so that it populates all seasons in a given year.  So admin data for
2024-25 gets loaded into "2024-25 Fall" and "2024-25 Spring".  Add tests
for the new range format.  Set the default cutoff for the start of Spring season will be the last Sunday in February
2024-04-26 13:31:50 -07:00
6fac759ec2 Faster admin data loader + rename School.school_hash 2024-04-22 15:43:54 -04:00
Nelson Jovel
9bfb76db5a match an additional format for Dates. Supported dates are now '1/10/2022 14:21:45' '2022-1-10T14:21:45' '2022-1-10 14:21:45' 2024-03-01 09:30:43 -08:00
Nelson Jovel
c3cb05701f feat: Support two date formats: ISO 8601 and the standard US date format
used in google sheets
2024-02-27 11:57:20 -08:00
Nelson Jovel
d907e2742e fix: fix failing tests 2024-02-22 12:02:15 -08:00
Nelson Jovel
d4109fda6f chore: refactor Race out of survey_item_values 2023-12-20 12:27:53 -08:00
Nelson Jovel
3e4ef9cb08 chore: refactor Gender out out of survey_item_values row 2023-12-20 12:27:44 -08:00
Nelson Jovel
6a24d4fa23 chore: Make sure 'hispanic' column only gets applied when using SIS race information 2023-12-18 15:26:21 -08:00
Nelson Jovel
a15b01a3e1 fix: instead of looking for 'asian' at the start of a word, look for it
after a word boundary.  This means it still doesn't get confused with
caucasian and it's more flexible whan asian appears inside other text
such as 'Caucasian and Asian and Black'
2023-12-08 14:22:43 -08:00
Nelson Jovel
8a0ba0dbea chore: various fixes for race and gender categorization during cleaning.
Also add tests for race and gender categorization
2023-12-08 14:22:33 -08:00
Nelson Jovel
2ef24caf70 Lower threshold for the number of valid student responses from 17 to 11 2023-12-06 13:56:14 -08:00
Nelson Jovel
f27a590c5a Convert gender and race text into qualtrics codes during cleaning. Abide by 'prefer not to disclose' for self reported race. Give priority to self reported data but use SIS information as backup 2023-11-30 20:57:04 -08:00
Nelson Jovel
a3f9e46414 chore: when searching for dese id, split up pattern so that to be more explicit about the order in which to search out the columns that might have the dese ID we're looking for. 2023-11-06 13:13:37 -08:00
rebuilt
b2fdbe5756 feat: We no longer trust the progress number that gets exported from qualtrics. Instead during the cleaning progress, perform a manual count of the number of responses to filter out rows that don't meet the minimum threshold. 2023-10-27 15:12:58 -07:00
rebuilt
ef44c41965 feat: add special education disaggregation 2023-10-24 13:05:57 -07:00
rebuilt
2fd56047d4 Add disaggregation by ELL 2023-10-24 12:51:12 -07:00
rebuilt
2ac30bb107 feat: Add income table to the database. Add seeder for income. Add a reference to income from survey item response. Update the loader to import income data from the survey response csv. Refactor analyze controller to extract presenter. Add corresponding specs. Add income graph to analyze page 2023-10-24 09:05:27 -07:00
rebuilt
0f23053294 It's possible for admin data likert score values to be above 5. If that happens, we
cap the likert score at 5.   This was happening already at the scraper
level but it's also now being done by the admin data loader for safety.
Also make sure to just update admin data instead of deleting and
reloading all values each load. Add tests to confirm this behavior
2023-06-03 16:47:03 -07:00
rebuilt
e058c523b6 Missing progress or duration information does not result in a row removed in the cleaning process 2023-06-02 15:18:03 -07:00
rebuilt
a71ebbc4e4 Add Overall Response Rate 2023-05-22 16:03:34 +00:00
rebuilt
dbfc9d1d3a Add automated data cleaning. Modify SurveyItemValues class to use regex
instead of hard coded values.  Produce a clean csv and a csv with all
the removed values and columns with reason for removal. Add script for
running cleaning for each project
2023-05-16 13:39:12 -07:00