Commit Graph

54 Commits (aa7af11a4e2aa0f5a372ba2005ca5670af16d4f2)

Author SHA1 Message Date
Nelson Jovel c565dbe46f chore: remove lowell seeding rules
2 years ago
Nelson Jovel a0c0b1d01d chore: reenable test spec that tests data loader for races
2 years ago
Nelson Jovel d4109fda6f chore: refactor Race out of survey_item_values
2 years ago
Nelson Jovel 3e4ef9cb08 chore: refactor Gender out out of survey_item_values row
2 years ago
rebuilt 02b335ef63 feat: reduce number of rows to process at one time to reduce memory use
2 years ago
rebuilt 019b954ffa feat: load student responses in the same pass as loading the survey responses
2 years ago
rebuilt ef44c41965 feat: add special education disaggregation
2 years ago
rebuilt 2fd56047d4 Add disaggregation by ELL
2 years ago
rebuilt 2ac30bb107 feat: Add income table to the database. Add seeder for income. Add a reference to income from survey item response. Update the loader to import income data from the survey response csv. Refactor analyze controller to extract presenter. Add corresponding specs. Add income graph to analyze page
2 years ago
rebuilt 878ba08a22 fix: There was an n+1 problem where we looked up the list of schools for
2 years ago
rebuilt 3f4aa8e7af Fix: Parse headers when they are surrounded by quotes. This helps load recent csv files correctly
3 years ago
rebuilt a71ebbc4e4 Add Overall Response Rate
3 years ago
rebuilt 435bc4a5be Fix regression in student loader
3 years ago
rebuilt dbfc9d1d3a Add automated data cleaning. Modify SurveyItemValues class to use regex
3 years ago
rebuilt 282a671531 Change survey data loader spec to use factorybot objects instead of loading seeds. Change databasecleaner to use transaction. Add back babel-preset dependency to fix failing javascript test in production.
3 years ago
rebuilt c0332955f3 move csv require statement to application.rb
3 years ago
rebuilt c6b8518cb1 Fix bug with not all survey responses loading when using sftp loader
3 years ago
rebuilt 7a159f7ddf Process 1000 rows at a time to limit memory usage in production
3 years ago
rebuilt 03c6bff7bb Load survey responses from sftp folder
3 years ago
rebuilt a1b7db9b2e Add ability to load student demographic information for only lowell
3 years ago
rebuilt fc9e73f656 make sure survey responses for Butler get loaded correctly
3 years ago
rebuilt 9bbba64e9e Add rules to the survey data loader to allow skipping non-lowell schools
3 years ago
Nelson Jovel 43bc55ed32 Query for survey items once instead of on each row
3 years ago
rebuilt 83fa255e5d Make utility method private
3 years ago
rebuilt 3ca270e3b1 Fix more malformed csv errors
3 years ago
rebuilt 360579852c Send fewer sql queries when loading survey responses
3 years ago
rebuilt 9ae94a2dae further reduce batch size to avoid exceeding memory quota
3 years ago
rebuilt bb472ce6c3 Reduce number of rows to process to avoid exceeding memory limit on heroku
3 years ago
rebuilt 7d7bee6498 Add gender disagreggation
3 years ago
rebuilt a1f3dfaa33 Load gender information to survey item responses
3 years ago
rebuilt f9c4ed8880 make sure grade gets updated when loading survey responses
3 years ago
rebuilt 89a7f27b88 Finishes #183092367. Adds ui and graphs to show grade level disaggregations
3 years ago
rebuilt bb5f668497 Refactor code based on rubocop and reek suggestions
3 years ago
rebuilt 18b479b8b2 Refactor based on rubocop and reek suggestions
3 years ago
Nelson Jovel 2d945fb696 Print a message when a survey item response is rejected from database import due to an invalid likert score
4 years ago
Nelson Jovel 6f629e2feb Add survey responses from 2016 through 2018. Finishes #181255948
4 years ago
Nelson Jovel 101c252896 Import survey item responses based on DESE ID instead of qualtrics school code. Finishes #181704978
4 years ago
Nelson Jovel 36da912253 Modify survey item response csv files for 2019-20 to correct incorrect likert scores
4 years ago
Nelson Jovel ed7a3b8a3d Autoformat code with rubocop
4 years ago
Nelson Jovel ad03606d66 Add benchmarks to survey and admin data items. Remove them from measures. Modify seeder
4 years ago
Liam Morley 65ee27742a Perform single lookup for survey items rather than n lookups
4 years ago
Liam Morley 6477792f1d Improve efficiency when checking whether survey item responses already exist
4 years ago
rebuilt 666eec9a20 Removed print statements from data loader output
4 years ago
Alex Basson b4f6daa406 Process CSV file rows in batches
4 years ago
Alex Basson a269bb0ec1 Try reporting on batch inserts
4 years ago
Alex Basson a41d5bb4c9 Update survey data loader for new student survey responses csv
4 years ago
Alex Basson acfc2fde63 Make SurveyResponseDataLoader.process_row a private class method
4 years ago
Alex Basson 98e7f3ce57 Update survey response data load to conform to survey response CSV standard
4 years ago
Alex Basson 16734f9ce7 Refactor survey response loader to use functional style
4 years ago
Alex Basson fd55c7c82a Optimize School lookup by qualtrics code query
4 years ago