Commit Graph

91 Commits (305ddf2b1a8f250703d14d5b970fe831ef1159f8)

Author SHA1 Message Date
Nelson Jovel 305ddf2b1a chore: add test for checking duplicate headers during cleaning process
2 years ago
Nelson Jovel b63c327d33 chore: when searching for dese id, split up pattern so that to be more explicit about the order in which to search out the columns that might have the dese ID we're looking for.
2 years ago
rebuilt 1a707eb6bc feat: load student responses in the same pass as loading the survey responses
2 years ago
rebuilt e3fbbabce5 feat: We no longer trust the progress number that gets exported from qualtrics. Instead during the cleaning progress, perform a manual count of the number of responses to filter out rows that don't meet the minimum threshold.
2 years ago
rebuilt 83661540b7 chore: upgrade to rails 7.1.
2 years ago
rebuilt 48e795fcfb feat: add special education disaggregation
2 years ago
rebuilt 060d7aa55a Add disaggregation by ELL
2 years ago
rebuilt abea2cb8fa feat: support multiple columns for race and gender information
2 years ago
rebuilt 714b90b3eb fix: ensure cleaner outputs columns for all survey items. Before the fix, if a survey item varient (ending in -1, ie s-tint-q1-1) did not have a matching survey item s-tint-q1, the resulting csv would not include that column
2 years ago
rebuilt a785c69c44 Add Overall Response Rate
2 years ago
rebuilt 4afa030141 chore: remove precalculated race scores. Calculate race scores on every reload
2 years ago
rebuilt cec48e55d3 chore: remove outdated admin data loader file. We now use Dese::Loader to load school level data
2 years ago
rebuilt 5c7729beeb feat: if admin data value is above 5, round down to 5
2 years ago
rebuilt 4f035f6a63 feat: Add income table to the database. Add seeder for income. Add a reference to income from survey item response. Update the loader to import income data from the survey response csv. Refactor analyze controller to extract presenter. Add corresponding specs. Add income graph to analyze page
2 years ago
rebuilt d72f8d31e0 fix: There was an n+1 problem where we looked up the list of schools for
2 years ago
rebuilt e8aa75bf66 feat: update survey_item_response table to indlude recorded date and import recorded date when loading responses
2 years ago
Nelson Jovel 0a2c5e02c5 feat: add ability to merge disaggregation data with raw survey data to
3 years ago
rebuilt 411c632c25 chore: remove errant comment
3 years ago
rebuilt 30285efd69 It's possible for admin data likert score values to be above 5. If that happens, we
3 years ago
rebuilt 9aeb5f92af Missing progress or duration information does not result in a row removed in the cleaning process
3 years ago
rebuilt e3ae12b425 update response_date to recorded_date
3 years ago
rebuilt 8ef8cfce58 Adjust valid duration threshold of short form items
3 years ago
rebuilt 4509c157fa Add automated data cleaning. Modify SurveyItemValues class to use regex
3 years ago
rebuilt 3f2a7dff50 Fix problem with dese scraper lumping in 2021-22 data as 2022-23 data.
3 years ago
rebuilt 128748addd Update logic for calculating student response rate. Remove references
3 years ago
rebuilt b250ebe415 Memoize schools in SurveyItemValues and academic_years in AcademicYear
3 years ago
rebuilt c15cb7b483 Change survey data loader spec to use factorybot objects instead of loading seeds. Change databasecleaner to use transaction. Add back babel-preset dependency to fix failing javascript test in production.
3 years ago
rebuilt fbfaafa996 Update school/framework files for ecp schools. Fix tests. Update fixture files with dese ids of schools in the consortium.
3 years ago
rebuilt 4c4ccc01cc Merge branch 'rpp-response-rate' to bring in changes to test files
3 years ago
rebuilt 6b31fa9115 Batch imports for staffing data
3 years ago
rebuilt d059177f0c load total students and batch importing records
3 years ago
Nelson Jovel bfa5f28d7b Convert dese::loader from using seeder to factories
3 years ago
Nelson Jovel 7a7b78a9e0 convert student loader from seeding to factories
3 years ago
rebuilt 2cf2b7d7c1 turn off some slow tests that don't add value
3 years ago
rebuilt 2362d884eb Convert admin data loader from using seeder to using factory
3 years ago
rebuilt d0219217de Convert response rate loader spec from using the seeder to using the factory
3 years ago
rebuilt 06f9d2f0e9 Scrape enrollment and staffing information. Seed enrollment and staffing information. Update DatabaseCleaner so it cleans up leftover information in the database. Remove old admin csvs from codebase.
3 years ago
rebuilt c0332955f3 move csv require statement to application.rb
3 years ago
rebuilt 7915a1bc7c rename respondents to repondent
3 years ago
rebuilt 52d2676f80 Populate db with grade level data
3 years ago
rebuilt 01851d8527 Make sure blank grades in csv load as nils instead of zeros
3 years ago
rebuilt 03c6bff7bb Load survey responses from sftp folder
3 years ago
rebuilt d5533bb756 Support survey item variants. Survey item id's in the format s-sbel-q1-1 will load into s-sbel-q1
3 years ago
rebuilt 53a08d41bf Change label for unknown race
3 years ago
rebuilt a1b7db9b2e Add ability to load student demographic information for only lowell
3 years ago
rebuilt fc9e73f656 make sure survey responses for Butler get loaded correctly
3 years ago
rebuilt 9bbba64e9e Add rules to the survey data loader to allow skipping non-lowell schools
3 years ago
rebuilt 7d7bee6498 Add gender disagreggation
3 years ago
rebuilt a1f3dfaa33 Load gender information to survey item responses
3 years ago
rebuilt 8cdd1b6285 Create gender class. Load genders to database from demographics file
3 years ago