784e23982e
feat: update survey_item_response table to indlude recorded date and import recorded date when loading responses
2 years ago
6b2bceceb6
Fix: Parse headers when they are surrounded by quotes. This helps load recent csv files correctly
3 years ago
37e932e078
Fix regression in student loader
3 years ago
0dfc9726d0
Add automated data cleaning. Modify SurveyItemValues class to use regex
...
instead of hard coded values. Produce a clean csv and a csv with all
the removed values and columns with reason for removal. Add script for
running cleaning for each project
3 years ago
a066f464c7
fix failing tests
3 years ago
a5da0fb0c6
Fix bug with not all survey responses loading when using sftp loader
3 years ago
47c1856281
Process 1000 rows at a time to limit memory usage in production
3 years ago
380048f5cd
Load survey responses from sftp folder
3 years ago
a1b7db9b2e
Add ability to load student demographic information for only lowell
3 years ago
fc9e73f656
make sure survey responses for Butler get loaded correctly
3 years ago
9bbba64e9e
Add rules to the survey data loader to allow skipping non-lowell schools
3 years ago
43bc55ed32
Query for survey items once instead of on each row
3 years ago
83fa255e5d
Make utility method private
3 years ago
3ca270e3b1
Fix more malformed csv errors
3 years ago
360579852c
Send fewer sql queries when loading survey responses
3 years ago
9ae94a2dae
further reduce batch size to avoid exceeding memory quota
3 years ago
bb472ce6c3
Reduce number of rows to process to avoid exceeding memory limit on heroku
3 years ago
7d7bee6498
Add gender disagreggation
3 years ago
a1f3dfaa33
Load gender information to survey item responses
3 years ago
f9c4ed8880
make sure grade gets updated when loading survey responses
3 years ago
89a7f27b88
Finishes #183092367 . Adds ui and graphs to show grade level disaggregations
3 years ago
bb5f668497
Refactor code based on rubocop and reek suggestions
3 years ago
18b479b8b2
Refactor based on rubocop and reek suggestions
3 years ago
2d945fb696
Print a message when a survey item response is rejected from database import due to an invalid likert score
4 years ago
6f629e2feb
Add survey responses from 2016 through 2018. Finishes #181255948
4 years ago
101c252896
Import survey item responses based on DESE ID instead of qualtrics school code. Finishes #181704978
4 years ago
36da912253
Modify survey item response csv files for 2019-20 to correct incorrect likert scores
4 years ago
ed7a3b8a3d
Autoformat code with rubocop
4 years ago
ad03606d66
Add benchmarks to survey and admin data items. Remove them from measures. Modify seeder
...
Calculate benchmarks for measures based on a weighted average of survey
and admin data items
Added architectural records
4 years ago
65ee27742a
Perform single lookup for survey items rather than n lookups
4 years ago
6477792f1d
Improve efficiency when checking whether survey item responses already exist
...
- see https://semaphoreci.com/blog/2017/03/14/faster-rails-how-to-check-if-a-record-exists.html for more info if curious
4 years ago
666eec9a20
Removed print statements from data loader output
4 years ago
b4f6daa406
Process CSV file rows in batches
4 years ago
a269bb0ec1
Try reporting on batch inserts
4 years ago
a41d5bb4c9
Update survey data loader for new student survey responses csv
4 years ago
acfc2fde63
Make SurveyResponseDataLoader.process_row a private class method
4 years ago
98e7f3ce57
Update survey response data load to conform to survey response CSV standard
4 years ago
16734f9ce7
Refactor survey response loader to use functional style
4 years ago
fd55c7c82a
Optimize School lookup by qualtrics code query
4 years ago
c66134e25e
Move School and AcademicYear lookup methods into models
4 years ago
d6fddb9f7b
use activerecord-import gem to speed up csv uploads
4 years ago
fd091cfa4f
Load 2020-21 student survey responses
4 years ago
f5724407f2
Load teacher survey responses
4 years ago
rebuilt
Liam Morley
Alex Basson