4509c157fa
Add automated data cleaning. Modify SurveyItemValues class to use regex
...
instead of hard coded values. Produce a clean csv and a csv with all
the removed values and columns with reason for removal. Add script for
running cleaning for each project
3 years ago
c15cb7b483
Change survey data loader spec to use factorybot objects instead of loading seeds. Change databasecleaner to use transaction. Add back babel-preset dependency to fix failing javascript test in production.
3 years ago
c0332955f3
move csv require statement to application.rb
3 years ago
c6b8518cb1
Fix bug with not all survey responses loading when using sftp loader
3 years ago
7a159f7ddf
Process 1000 rows at a time to limit memory usage in production
3 years ago
03c6bff7bb
Load survey responses from sftp folder
3 years ago
a1b7db9b2e
Add ability to load student demographic information for only lowell
3 years ago
fc9e73f656
make sure survey responses for Butler get loaded correctly
3 years ago
9bbba64e9e
Add rules to the survey data loader to allow skipping non-lowell schools
3 years ago
43bc55ed32
Query for survey items once instead of on each row
3 years ago
83fa255e5d
Make utility method private
3 years ago
3ca270e3b1
Fix more malformed csv errors
3 years ago
360579852c
Send fewer sql queries when loading survey responses
3 years ago
9ae94a2dae
further reduce batch size to avoid exceeding memory quota
3 years ago
bb472ce6c3
Reduce number of rows to process to avoid exceeding memory limit on heroku
3 years ago
7d7bee6498
Add gender disagreggation
3 years ago
a1f3dfaa33
Load gender information to survey item responses
3 years ago
f9c4ed8880
make sure grade gets updated when loading survey responses
3 years ago
89a7f27b88
Finishes #183092367 . Adds ui and graphs to show grade level disaggregations
3 years ago
bb5f668497
Refactor code based on rubocop and reek suggestions
3 years ago
18b479b8b2
Refactor based on rubocop and reek suggestions
3 years ago
2d945fb696
Print a message when a survey item response is rejected from database import due to an invalid likert score
4 years ago
6f629e2feb
Add survey responses from 2016 through 2018. Finishes #181255948
4 years ago
101c252896
Import survey item responses based on DESE ID instead of qualtrics school code. Finishes #181704978
4 years ago
36da912253
Modify survey item response csv files for 2019-20 to correct incorrect likert scores
4 years ago
ed7a3b8a3d
Autoformat code with rubocop
4 years ago
ad03606d66
Add benchmarks to survey and admin data items. Remove them from measures. Modify seeder
...
Calculate benchmarks for measures based on a weighted average of survey
and admin data items
Added architectural records
4 years ago
65ee27742a
Perform single lookup for survey items rather than n lookups
4 years ago
6477792f1d
Improve efficiency when checking whether survey item responses already exist
...
- see https://semaphoreci.com/blog/2017/03/14/faster-rails-how-to-check-if-a-record-exists.html for more info if curious
4 years ago
666eec9a20
Removed print statements from data loader output
4 years ago
b4f6daa406
Process CSV file rows in batches
4 years ago
a269bb0ec1
Try reporting on batch inserts
4 years ago
a41d5bb4c9
Update survey data loader for new student survey responses csv
4 years ago
acfc2fde63
Make SurveyResponseDataLoader.process_row a private class method
4 years ago
98e7f3ce57
Update survey response data load to conform to survey response CSV standard
4 years ago
16734f9ce7
Refactor survey response loader to use functional style
4 years ago
fd55c7c82a
Optimize School lookup by qualtrics code query
4 years ago
c66134e25e
Move School and AcademicYear lookup methods into models
4 years ago
d6fddb9f7b
use activerecord-import gem to speed up csv uploads
4 years ago
fd091cfa4f
Load 2020-21 student survey responses
4 years ago
f5724407f2
Load teacher survey responses
4 years ago
rebuilt
Liam Morley
Alex Basson