Update student survey responses for the 2018-19 school year

README.md

@@ -141,6 +141,11 @@ $ ./scripts/load_survey_responses_on_heroku beta
 $ ./scripts/load_survey_responses_on_heroku dashboard
 ```
 
+There is also an example one-off task to load a single csv at a time.
+```bash
+bundle exec rake one_off:load_2018_19_student_responses
+```
+
 ### Loading Admin Data Values
 Loading admin data is similar to loading survey item responses.  Run the one of the following scripts to load admin data to a selected environment.
 

data/survey_responses/2018-19_student_survey_responses.csv

@@ -1,4 +1,4 @@
-StartDate,Response.Type,IPAddress,Progress,Duration..in.seconds.,Finished,RecordedDate,ResponseID,Location.Latitude,DistributionChannel,UserLanguage,district,student_id,grade,s.emsa.q1,s.emsa.q2,s.emsa.q3,s.tint.q1,s.tint.q2,s.tint.q3,s.tint.q4,s.tint.q5,s-acpr-q1,s-acpr-q2,s-acpr-q3,s-acpr-q4,s-cure-q1,s-cure-q2,s-cure-q3,s-cure-q4,s-sten-q1,s-sten-q2,s-sten-q3,s-sper-q1,s-sper-q2,s-sper-q3,s-sper-q4,s-civp-q1,s-civp-q2,s-civp-q3,s-civp-q4,s-grmi-q1,s-grmi-q2,s-grmi-q3,s-grmi-q4,s-appa-q1,s-appa-q2,s-appa-q3,s-peff-q1,s-peff-q2,s-peff-q3,s-peff-q4,s-peff-q5,s-peff-q6,s-sbel-q1,s-sbel-q2,s-sbel-q3,s-sbel-q4,s-sbel-q5,s-phys-q1,s-phys-q2,s-phys-q3,s-phys-q4,s-vale-q1,s-vale-q2,s-vale-q3,s-vale-q4,s-acst-q1,s-acst-q2,s-acst-q3,s-sust-q1,s-sust-q2,s-grit-q1,s-grit-q2,s-grit-q3,s-grit-q4,s-expa-q1,s-poaf-q1,s-poaf-q2,s-poaf-q3,s-poaf-q4,gender,Q90_3_TEXT,race,Q91_7_TEXT,language,MathTeacher,ScienceTeacher,SocialTeacher,EnglishTeacher,DESE ID
+StartDate,Response.Type,IPAddress,Progress,Duration..in.seconds.,Finished,RecordedDate,ResponseID,Location.Latitude,DistributionChannel,UserLanguage,district,student_id,grade,s-emsa-q1,s-emsa-q2,s-emsa-q3,s-tint-q1,s-tint-q2,s-tint-q3,s-tint-q4,s-tint-q5,s-acpr-q1,s-acpr-q2,s-acpr-q3,s-acpr-q4,s-cure-q1,s-cure-q2,s-cure-q3,s-cure-q4,s-sten-q1,s-sten-q2,s-sten-q3,s-sper-q1,s-sper-q2,s-sper-q3,s-sper-q4,s-civp-q1,s-civp-q2,s-civp-q3,s-civp-q4,s-grmi-q1,s-grmi-q2,s-grmi-q3,s-grmi-q4,s-appa-q1,s-appa-q2,s-appa-q3,s-peff-q1,s-peff-q2,s-peff-q3,s-peff-q4,s-peff-q5,s-peff-q6,s-sbel-q1,s-sbel-q2,s-sbel-q3,s-sbel-q4,s-sbel-q5,s-phys-q1,s-phys-q2,s-phys-q3,s-phys-q4,s-vale-q1,s-vale-q2,s-vale-q3,s-vale-q4,s-acst-q1,s-acst-q2,s-acst-q3,s-sust-q1,s-sust-q2,s-grit-q1,s-grit-q2,s-grit-q3,s-grit-q4,s-expa-q1,s-poaf-q1,s-poaf-q2,s-poaf-q3,s-poaf-q4,gender,Q90_3_TEXT,race,Q91_7_TEXT,language,MathTeacher,ScienceTeacher,SocialTeacher,EnglishTeacher,DESE ID
 3/12/19 8:47,IP Address,209.80.140.129,100,323,TRUE,2019-03-12T8:52:0,R_24IvMAxf0EV3C69,42.4125061,anonymous,EN,Revere,36434,4,NA,NA,NA,NA,NA,NA,NA,NA,3,2,3,2,NA,NA,NA,NA,NA,NA,NA,2,1,3,4,NA,NA,NA,NA,NA,NA,NA,NA,1,4,5,2,5,3,4,2,2,3,4,3,4,4,3,5,1,1,4,5,5,5,3,4,1,3,NA,4,1,5,1,5,1,4,2,4,2,,7,,EN,,,,,02480003
 3/12/19 8:47,IP Address,209.80.140.129,100,530,TRUE,2019-03-12T8:55:0,R_1mWRko0fMVAqNRE,42.4125061,anonymous,EN,Revere,37788,4,NA,NA,NA,NA,NA,NA,NA,NA,5,4,5,5,NA,NA,NA,NA,NA,NA,NA,5,5,3,5,NA,NA,NA,NA,NA,NA,NA,NA,1,5,5,4,4,5,3,3,5,2,3,3,3,3,5,5,4,5,3,1,3,4,1,2,3,5,3,2,2,2,3,4,3,2,3,4,1,,6,,EN,,,,,02480003
 3/12/19 8:47,IP Address,209.80.140.129,100,558,TRUE,2019-03-12T8:56:0,R_3NVwGDvFnTiR2rf,42.4125061,anonymous,EN,Revere,37810,4,5,4,2,5,4,5,2,2,NA,NA,NA,NA,NA,NA,NA,NA,NA,NA,NA,NA,NA,NA,NA,NA,NA,NA,NA,1,1,3,3,NA,NA,NA,4,4,2,3,4,5,4,1,1,4,4,5,5,3,2,1,4,2,2,NA,NA,2,1,3,5,4,4,5,3,4,4,3,5,2,,7,,EN,,,,,02480003

doc/architectural_decision_records/6.md

@@ -0,0 +1,17 @@
+# Decision record 6
+
+# Upload a single csv at a time
+
+## Status
+
+Completed
+
+## Context
+
+Uploading survey results takes a long time.  It's on the order of 20-30 minutes per csv file.
+## Decision
+
+Instead of uploading all records each time, upload only a single csv file
+## Consequences
+
+There is a lot less time spent performing work that has already been done. This script, like the script to import all csvs, does not remove any responses.  It only adds responses that do not already exist and modifies the likert value to the one found in the csv if the value differs from the database.

lib/tasks/one_off.rake

@@ -45,9 +45,9 @@ namespace :one_off do
     measure_4aii.update! measure_id: '4A-i'
   end
 
-  desc 'load boston 2018-19 responses'
-  task load_boston_2018_19_responses: :environment do
-    filepath = Rails.root.join('data', 'survey_responses', '2018-19_boston_student_survey_responses.csv')
+  desc 'load 2018-19 student responses'
+  task load_2018_19_student_responses: :environment do
+    filepath = Rails.root.join('data', 'survey_responses', '2018-19_student_survey_responses.csv')
     puts "=====================> Loading data from csv at path: #{filepath}"
     SurveyResponsesDataLoader.load_data filepath: filepath
     puts "=====================> Completed loading #{SurveyItemResponse.count} survey responses"