108 commits

Author	SHA1	Message	Date
rebuilt	282a671531	Change survey data loader spec to use factorybot objects instead of loading seeds. Change databasecleaner to use transaction. Add back babel-preset dependency to fix failing javascript test in production.	2023-03-29 15:45:48 -07:00
rebuilt	825259bdd8	Merge branch 'rpp-response-rate' into rpp-main to bring in improvements ... to how we get enrollment and staffing information. Also speed up tests	2023-03-22 16:52:55 -07:00
rebuilt	6b31fa9115	Batch imports for staffing data	2023-03-08 04:51:15 -08:00
rebuilt	d059177f0c	load total students and batch importing records	2023-03-08 04:48:25 -08:00
rebuilt	2362d884eb	Convert admin data loader from using seeder to using factory	2023-03-07 11:12:52 -08:00
rebuilt	d0219217de	Convert response rate loader spec from using the seeder to using the factory	2023-03-07 10:35:38 -08:00
rebuilt	06f9d2f0e9	Scrape enrollment and staffing information. Seed enrollment and staffing information. Update DatabaseCleaner so it cleans up leftover information in the database. Remove old admin csvs from codebase.	2023-03-05 16:15:35 -08:00
rebuilt	c0332955f3	move csv require statement to application.rb	2023-02-27 15:47:20 -08:00
rebuilt	ae6d7151b3	Load student respondent information from enrollment loader	2023-02-27 12:29:16 -08:00
rebuilt	984fedff02	Add academic year column to enrollment file. Add enrollment loader to seeder file	2023-02-22 01:12:09 -08:00
rebuilt	52d2676f80	Populate db with grade level data	2023-02-21 16:58:08 -08:00
rebuilt	69c089d9bb	Remove occurances of MCIEA	2023-02-13 17:58:27 -08:00
rebuilt	01851d8527	Make sure blank grades in csv load as nils instead of zeros	2023-02-10 13:44:48 -08:00
rebuilt	3a744db975	Start adding kindergarten	2023-01-17 15:27:15 -08:00
rebuilt	c6b8518cb1	Fix bug with not all survey responses loading when using sftp loader	2023-01-12 17:05:00 -08:00
rebuilt	b1200dca33	update default folder for survey responses	2023-01-12 17:05:00 -08:00
rebuilt	7a159f7ddf	Process 1000 rows at a time to limit memory usage in production	2023-01-12 17:05:00 -08:00
rebuilt	79adff5faa	Don't print sftptogo_url	2023-01-12 17:05:00 -08:00
rebuilt	03c6bff7bb	Load survey responses from sftp folder	2023-01-12 17:04:57 -08:00
rebuilt	1426e7cc63	add proof of concept sftp downloader	2023-01-12 17:02:35 -08:00
rebuilt	eeeb4f5d14	Add Stoklosa survey responses	2023-01-12 17:02:35 -08:00
rebuilt	d5533bb756	Support survey item variants. Survey item id's in the format s-sbel-q1-1 will load into s-sbel-q1	2023-01-12 17:02:05 -08:00
rebuilt	ed421296c7	Fix bug that caused an incorrect score of 1 to be assigned for admin ... data item related to parity index of teachers of color to students of color. Now if there's not enough data to calculate a score, it produces 'NA' instead of 1. Also added admin data for a-reso-i1 and a-curv-i1	2023-01-06 17:43:48 -08:00
rebuilt	7b9abe3793	Scrape latest data from DESE	2022-12-20 15:26:11 -08:00
rebuilt	850ef9d1cd	Make sure demographic loader doesn't break when changing the designation of a Race	2022-12-12 12:03:53 -08:00
rebuilt	53a08d41bf	Change label for unknown race	2022-12-10 15:44:56 -08:00
rebuilt	a1b7db9b2e	Add ability to load student demographic information for only lowell	2022-12-03 11:33:11 -08:00
rebuilt	d49cf918ac	Add a way to seed only lowell schools	2022-12-03 10:23:02 -08:00
rebuilt	fc9e73f656	make sure survey responses for Butler get loaded correctly	2022-12-02 14:11:54 -08:00
rebuilt	9bbba64e9e	Add rules to the survey data loader to allow skipping non-lowell schools	2022-12-01 16:15:45 -08:00
Nelson Jovel	43bc55ed32	Query for survey items once instead of on each row	2022-11-28 11:58:58 -08:00
rebuilt	83fa255e5d	Make utility method private	2022-11-25 03:50:17 -08:00
rebuilt	3ca270e3b1	Fix more malformed csv errors	2022-11-25 03:09:00 -08:00
rebuilt	360579852c	Send fewer sql queries when loading survey responses	2022-11-24 03:14:20 -08:00
rebuilt	9ae94a2dae	further reduce batch size to avoid exceeding memory quota	2022-11-23 13:54:11 -08:00
rebuilt	bb472ce6c3	Reduce number of rows to process to avoid exceeding memory limit on heroku	2022-11-23 12:32:31 -08:00
rebuilt	4626d43e3e	Add changes to demographic loader so it the tests don't fail when run in parallel	2022-11-03 14:46:45 -07:00
rebuilt	7d7bee6498	Add gender disagreggation	2022-10-27 15:56:50 -07:00
rebuilt	a1f3dfaa33	Load gender information to survey item responses	2022-10-21 21:38:33 -07:00
rebuilt	8cdd1b6285	Create gender class. Load genders to database from demographics file	2022-10-21 17:47:52 -07:00
rebuilt	f4e2e6ca83	Correct headers for 4D so the data can be correctly imported	2022-10-21 17:29:19 -07:00
rebuilt	f9c4ed8880	make sure grade gets updated when loading survey responses	2022-10-16 22:31:10 -07:00
rebuilt	89a7f27b88	Finishes #183092367. Adds ui and graphs to show grade level disaggregations	2022-10-16 22:03:25 -07:00
Nelson Jovel	6ed405d16d	Update student csvs	2022-09-26 15:57:56 -07:00
rebuilt	dd9ed86dcd	Merge branch 'scrapers'	2022-09-21 19:54:24 -07:00
rebuilt	f556844b73	Update a-sust-i1 so it's only applicable to high schools and lower ... benchmark for a-cure-i1 from 6.7 to 5 percent.	2022-09-21 19:50:33 -07:00
rebuilt	9e82023e48	Make sure we don't divide by 0	2022-09-20 16:44:11 -07:00
rebuilt	1a984d2cf0	Make sure we don't divide by 0	2022-09-20 16:43:19 -07:00
rebuilt	3ee909bbea	Update 3B-2 with an absolute floor	2022-09-20 14:45:20 -07:00
rebuilt	1be6e79fd8	Add scrapers for admin data and their corresponding csv files of data. ... Make sure specs are off so they don't run in github CI	2022-09-19 19:37:39 -07:00