diff --git a/README.md b/README.md index 8403ae80..5589e8f6 100644 --- a/README.md +++ b/README.md @@ -141,6 +141,11 @@ $ ./scripts/load_survey_responses_on_heroku beta $ ./scripts/load_survey_responses_on_heroku dashboard ``` +There is also an example one-off task to load a single csv at a time. +```bash +bundle exec rake one_off:load_2018_19_student_responses +``` + ### Loading Admin Data Values Loading admin data is similar to loading survey item responses. Run the one of the following scripts to load admin data to a selected environment. diff --git a/data/survey_responses/2018-19_student_survey_responses.csv b/data/survey_responses/2018-19_student_survey_responses.csv index 3e818432..5ad5da5a 100644 --- a/data/survey_responses/2018-19_student_survey_responses.csv +++ b/data/survey_responses/2018-19_student_survey_responses.csv @@ -1,4 +1,4 @@ -StartDate,Response.Type,IPAddress,Progress,Duration..in.seconds.,Finished,RecordedDate,ResponseID,Location.Latitude,DistributionChannel,UserLanguage,district,student_id,grade,s.emsa.q1,s.emsa.q2,s.emsa.q3,s.tint.q1,s.tint.q2,s.tint.q3,s.tint.q4,s.tint.q5,s-acpr-q1,s-acpr-q2,s-acpr-q3,s-acpr-q4,s-cure-q1,s-cure-q2,s-cure-q3,s-cure-q4,s-sten-q1,s-sten-q2,s-sten-q3,s-sper-q1,s-sper-q2,s-sper-q3,s-sper-q4,s-civp-q1,s-civp-q2,s-civp-q3,s-civp-q4,s-grmi-q1,s-grmi-q2,s-grmi-q3,s-grmi-q4,s-appa-q1,s-appa-q2,s-appa-q3,s-peff-q1,s-peff-q2,s-peff-q3,s-peff-q4,s-peff-q5,s-peff-q6,s-sbel-q1,s-sbel-q2,s-sbel-q3,s-sbel-q4,s-sbel-q5,s-phys-q1,s-phys-q2,s-phys-q3,s-phys-q4,s-vale-q1,s-vale-q2,s-vale-q3,s-vale-q4,s-acst-q1,s-acst-q2,s-acst-q3,s-sust-q1,s-sust-q2,s-grit-q1,s-grit-q2,s-grit-q3,s-grit-q4,s-expa-q1,s-poaf-q1,s-poaf-q2,s-poaf-q3,s-poaf-q4,gender,Q90_3_TEXT,race,Q91_7_TEXT,language,MathTeacher,ScienceTeacher,SocialTeacher,EnglishTeacher,DESE ID +StartDate,Response.Type,IPAddress,Progress,Duration..in.seconds.,Finished,RecordedDate,ResponseID,Location.Latitude,DistributionChannel,UserLanguage,district,student_id,grade,s-emsa-q1,s-emsa-q2,s-emsa-q3,s-tint-q1,s-tint-q2,s-tint-q3,s-tint-q4,s-tint-q5,s-acpr-q1,s-acpr-q2,s-acpr-q3,s-acpr-q4,s-cure-q1,s-cure-q2,s-cure-q3,s-cure-q4,s-sten-q1,s-sten-q2,s-sten-q3,s-sper-q1,s-sper-q2,s-sper-q3,s-sper-q4,s-civp-q1,s-civp-q2,s-civp-q3,s-civp-q4,s-grmi-q1,s-grmi-q2,s-grmi-q3,s-grmi-q4,s-appa-q1,s-appa-q2,s-appa-q3,s-peff-q1,s-peff-q2,s-peff-q3,s-peff-q4,s-peff-q5,s-peff-q6,s-sbel-q1,s-sbel-q2,s-sbel-q3,s-sbel-q4,s-sbel-q5,s-phys-q1,s-phys-q2,s-phys-q3,s-phys-q4,s-vale-q1,s-vale-q2,s-vale-q3,s-vale-q4,s-acst-q1,s-acst-q2,s-acst-q3,s-sust-q1,s-sust-q2,s-grit-q1,s-grit-q2,s-grit-q3,s-grit-q4,s-expa-q1,s-poaf-q1,s-poaf-q2,s-poaf-q3,s-poaf-q4,gender,Q90_3_TEXT,race,Q91_7_TEXT,language,MathTeacher,ScienceTeacher,SocialTeacher,EnglishTeacher,DESE ID 3/12/19 8:47,IP Address,209.80.140.129,100,323,TRUE,2019-03-12T8:52:0,R_24IvMAxf0EV3C69,42.4125061,anonymous,EN,Revere,36434,4,NA,NA,NA,NA,NA,NA,NA,NA,3,2,3,2,NA,NA,NA,NA,NA,NA,NA,2,1,3,4,NA,NA,NA,NA,NA,NA,NA,NA,1,4,5,2,5,3,4,2,2,3,4,3,4,4,3,5,1,1,4,5,5,5,3,4,1,3,NA,4,1,5,1,5,1,4,2,4,2,,7,,EN,,,,,02480003 3/12/19 8:47,IP Address,209.80.140.129,100,530,TRUE,2019-03-12T8:55:0,R_1mWRko0fMVAqNRE,42.4125061,anonymous,EN,Revere,37788,4,NA,NA,NA,NA,NA,NA,NA,NA,5,4,5,5,NA,NA,NA,NA,NA,NA,NA,5,5,3,5,NA,NA,NA,NA,NA,NA,NA,NA,1,5,5,4,4,5,3,3,5,2,3,3,3,3,5,5,4,5,3,1,3,4,1,2,3,5,3,2,2,2,3,4,3,2,3,4,1,,6,,EN,,,,,02480003 3/12/19 8:47,IP Address,209.80.140.129,100,558,TRUE,2019-03-12T8:56:0,R_3NVwGDvFnTiR2rf,42.4125061,anonymous,EN,Revere,37810,4,5,4,2,5,4,5,2,2,NA,NA,NA,NA,NA,NA,NA,NA,NA,NA,NA,NA,NA,NA,NA,NA,NA,NA,NA,1,1,3,3,NA,NA,NA,4,4,2,3,4,5,4,1,1,4,4,5,5,3,2,1,4,2,2,NA,NA,2,1,3,5,4,4,5,3,4,4,3,5,2,,7,,EN,,,,,02480003 diff --git a/doc/architectural_decision_records/6.md b/doc/architectural_decision_records/6.md new file mode 100644 index 00000000..2a3cb520 --- /dev/null +++ b/doc/architectural_decision_records/6.md @@ -0,0 +1,17 @@ +# Decision record 6 + +# Upload a single csv at a time + +## Status + +Completed + +## Context + +Uploading survey results takes a long time. It's on the order of 20-30 minutes per csv file. +## Decision + +Instead of uploading all records each time, upload only a single csv file +## Consequences + +There is a lot less time spent performing work that has already been done. This script, like the script to import all csvs, does not remove any responses. It only adds responses that do not already exist and modifies the likert value to the one found in the csv if the value differs from the database. diff --git a/lib/tasks/one_off.rake b/lib/tasks/one_off.rake index a893a88b..aa73b3c7 100644 --- a/lib/tasks/one_off.rake +++ b/lib/tasks/one_off.rake @@ -45,9 +45,9 @@ namespace :one_off do measure_4aii.update! measure_id: '4A-i' end - desc 'load boston 2018-19 responses' - task load_boston_2018_19_responses: :environment do - filepath = Rails.root.join('data', 'survey_responses', '2018-19_boston_student_survey_responses.csv') + desc 'load 2018-19 student responses' + task load_2018_19_student_responses: :environment do + filepath = Rails.root.join('data', 'survey_responses', '2018-19_student_survey_responses.csv') puts "=====================> Loading data from csv at path: #{filepath}" SurveyResponsesDataLoader.load_data filepath: filepath puts "=====================> Completed loading #{SurveyItemResponse.count} survey responses"