41 commits

Author	SHA1	Message	Date
rebuilt	37e932e078	Fix regression in student loader	2023-05-24 12:04:39 -07:00
rebuilt	0dfc9726d0	Add automated data cleaning. Modify SurveyItemValues class to use regex ... instead of hard coded values. Produce a clean csv and a csv with all the removed values and columns with reason for removal. Add script for running cleaning for each project	2023-05-24 11:59:53 -07:00
rebuilt	a066f464c7	fix failing tests	2023-04-30 16:35:08 -07:00
rebuilt	a5da0fb0c6	Fix bug with not all survey responses loading when using sftp loader	2023-02-19 19:36:14 -08:00
rebuilt	47c1856281	Process 1000 rows at a time to limit memory usage in production	2023-02-19 19:36:14 -08:00
rebuilt	380048f5cd	Load survey responses from sftp folder	2023-02-19 19:36:14 -08:00
rebuilt	a1b7db9b2e	Add ability to load student demographic information for only lowell	2022-12-03 11:33:11 -08:00
rebuilt	fc9e73f656	make sure survey responses for Butler get loaded correctly	2022-12-02 14:11:54 -08:00
rebuilt	9bbba64e9e	Add rules to the survey data loader to allow skipping non-lowell schools	2022-12-01 16:15:45 -08:00
Nelson Jovel	43bc55ed32	Query for survey items once instead of on each row	2022-11-28 11:58:58 -08:00
rebuilt	83fa255e5d	Make utility method private	2022-11-25 03:50:17 -08:00
rebuilt	3ca270e3b1	Fix more malformed csv errors	2022-11-25 03:09:00 -08:00
rebuilt	360579852c	Send fewer sql queries when loading survey responses	2022-11-24 03:14:20 -08:00
rebuilt	9ae94a2dae	further reduce batch size to avoid exceeding memory quota	2022-11-23 13:54:11 -08:00
rebuilt	bb472ce6c3	Reduce number of rows to process to avoid exceeding memory limit on heroku	2022-11-23 12:32:31 -08:00
rebuilt	7d7bee6498	Add gender disagreggation	2022-10-27 15:56:50 -07:00
rebuilt	a1f3dfaa33	Load gender information to survey item responses	2022-10-21 21:38:33 -07:00
rebuilt	f9c4ed8880	make sure grade gets updated when loading survey responses	2022-10-16 22:31:10 -07:00
rebuilt	89a7f27b88	Finishes #183092367. Adds ui and graphs to show grade level disaggregations	2022-10-16 22:03:25 -07:00
rebuilt	bb5f668497	Refactor code based on rubocop and reek suggestions	2022-07-11 16:46:30 -07:00
rebuilt	18b479b8b2	Refactor based on rubocop and reek suggestions	2022-07-08 19:07:23 -07:00
Nelson Jovel	2d945fb696	Print a message when a survey item response is rejected from database import due to an invalid likert score	2022-04-15 12:49:03 -07:00
Nelson Jovel	6f629e2feb	Add survey responses from 2016 through 2018. Finishes #181255948	2022-03-31 17:21:27 -07:00
Nelson Jovel	101c252896	Import survey item responses based on DESE ID instead of qualtrics school code. Finishes #181704978	2022-03-30 17:19:06 -07:00
Nelson Jovel	36da912253	Modify survey item response csv files for 2019-20 to correct incorrect likert scores	2022-03-15 21:31:11 -07:00
Nelson Jovel	ed7a3b8a3d	Autoformat code with rubocop	2022-03-04 15:29:54 +01:00
Nelson Jovel	ad03606d66	Add benchmarks to survey and admin data items. Remove them from measures. Modify seeder ... Calculate benchmarks for measures based on a weighted average of survey and admin data items Added architectural records	2021-12-28 14:10:34 +01:00
Liam Morley	65ee27742a	Perform single lookup for survey items rather than n lookups	2021-12-02 08:51:36 -05:00
Liam Morley	6477792f1d	Improve efficiency when checking whether survey item responses already exist ... - see https://semaphoreci.com/blog/2017/03/14/faster-rails-how-to-check-if-a-record-exists.html for more info if curious	2021-12-01 14:24:53 -05:00
rebuilt	666eec9a20	Removed print statements from data loader output	2021-09-28 18:03:14 +02:00
Alex Basson	b4f6daa406	Process CSV file rows in batches	2021-09-27 17:04:16 -04:00
Alex Basson	a269bb0ec1	Try reporting on batch inserts	2021-09-27 16:30:23 -04:00
Alex Basson	a41d5bb4c9	Update survey data loader for new student survey responses csv	2021-09-27 13:54:25 -04:00
Alex Basson	acfc2fde63	Make SurveyResponseDataLoader.process_row a private class method	2021-09-24 11:52:17 -04:00
Alex Basson	98e7f3ce57	Update survey response data load to conform to survey response CSV standard	2021-09-24 10:52:29 -04:00
Alex Basson	16734f9ce7	Refactor survey response loader to use functional style	2021-09-23 14:29:25 -04:00
Alex Basson	fd55c7c82a	Optimize School lookup by qualtrics code query	2021-09-23 07:30:27 -04:00
Alex Basson	c66134e25e	Move School and AcademicYear lookup methods into models	2021-09-22 21:20:17 -04:00
rebuilt	d6fddb9f7b	use activerecord-import gem to speed up csv uploads	2021-09-22 21:03:12 -04:00
Alex Basson	fd091cfa4f	Load 2020-21 student survey responses	2021-09-22 15:15:59 -04:00
Alex Basson	f5724407f2	Load teacher survey responses	2021-09-22 08:44:54 -04:00