98 commits

Author	SHA1	Message	Date
rebuilt	d3a28f7635	Fix ThreeATwo scraper	2023-04-22 14:01:01 -07:00
rebuilt	07ed8dd259	Update logic for calculating student response rate. Remove references ... to survey table. We no longer check or keep track of the survey type. Instead we look in the database to see if a survey item has at least 10 responses. If it does, that survey item was presented to the respondent and we count it, and all responses when calculating the response rate. Remove response rate timestamp from caching logic because we no longer add the response rate to the database. All response rates are calculated on the fly Update three_b_two scraper to use teacher only numbers swap over to using https://profiles.doe.mass.edu/statereport/gradesubjectstaffing.aspx as the source of staffing information	2023-04-22 14:00:20 -07:00
rebuilt	357c7427d1	Batch imports for staffing data	2023-04-22 13:19:34 -07:00
rebuilt	d272e48adc	load total students and batch importing records	2023-04-22 13:18:50 -07:00
rebuilt	cf2b2433e9	Use an sftp uri unique to MCIEA	2023-02-19 19:36:14 -08:00
rebuilt	a5da0fb0c6	Fix bug with not all survey responses loading when using sftp loader	2023-02-19 19:36:14 -08:00
rebuilt	ef087a6cd0	update default folder for survey responses	2023-02-19 19:36:14 -08:00
rebuilt	47c1856281	Process 1000 rows at a time to limit memory usage in production	2023-02-19 19:36:14 -08:00
rebuilt	640de1c8df	Don't print sftptogo_url	2023-02-19 19:36:14 -08:00
rebuilt	380048f5cd	Load survey responses from sftp folder	2023-02-19 19:36:14 -08:00
rebuilt	6f986ff8a1	add proof of concept sftp downloader	2023-02-19 19:36:05 -08:00
rebuilt	d5533bb756	Support survey item variants. Survey item id's in the format s-sbel-q1-1 will load into s-sbel-q1	2023-01-12 17:02:05 -08:00
rebuilt	ed421296c7	Fix bug that caused an incorrect score of 1 to be assigned for admin ... data item related to parity index of teachers of color to students of color. Now if there's not enough data to calculate a score, it produces 'NA' instead of 1. Also added admin data for a-reso-i1 and a-curv-i1	2023-01-06 17:43:48 -08:00
rebuilt	7b9abe3793	Scrape latest data from DESE	2022-12-20 15:26:11 -08:00
rebuilt	850ef9d1cd	Make sure demographic loader doesn't break when changing the designation of a Race	2022-12-12 12:03:53 -08:00
rebuilt	53a08d41bf	Change label for unknown race	2022-12-10 15:44:56 -08:00
rebuilt	a1b7db9b2e	Add ability to load student demographic information for only lowell	2022-12-03 11:33:11 -08:00
rebuilt	d49cf918ac	Add a way to seed only lowell schools	2022-12-03 10:23:02 -08:00
rebuilt	fc9e73f656	make sure survey responses for Butler get loaded correctly	2022-12-02 14:11:54 -08:00
rebuilt	9bbba64e9e	Add rules to the survey data loader to allow skipping non-lowell schools	2022-12-01 16:15:45 -08:00
Nelson Jovel	43bc55ed32	Query for survey items once instead of on each row	2022-11-28 11:58:58 -08:00
rebuilt	83fa255e5d	Make utility method private	2022-11-25 03:50:17 -08:00
rebuilt	3ca270e3b1	Fix more malformed csv errors	2022-11-25 03:09:00 -08:00
rebuilt	360579852c	Send fewer sql queries when loading survey responses	2022-11-24 03:14:20 -08:00
rebuilt	9ae94a2dae	further reduce batch size to avoid exceeding memory quota	2022-11-23 13:54:11 -08:00
rebuilt	bb472ce6c3	Reduce number of rows to process to avoid exceeding memory limit on heroku	2022-11-23 12:32:31 -08:00
rebuilt	4626d43e3e	Add changes to demographic loader so it the tests don't fail when run in parallel	2022-11-03 14:46:45 -07:00
rebuilt	7d7bee6498	Add gender disagreggation	2022-10-27 15:56:50 -07:00
rebuilt	a1f3dfaa33	Load gender information to survey item responses	2022-10-21 21:38:33 -07:00
rebuilt	8cdd1b6285	Create gender class. Load genders to database from demographics file	2022-10-21 17:47:52 -07:00
rebuilt	f4e2e6ca83	Correct headers for 4D so the data can be correctly imported	2022-10-21 17:29:19 -07:00
rebuilt	f9c4ed8880	make sure grade gets updated when loading survey responses	2022-10-16 22:31:10 -07:00
rebuilt	89a7f27b88	Finishes #183092367. Adds ui and graphs to show grade level disaggregations	2022-10-16 22:03:25 -07:00
Nelson Jovel	6ed405d16d	Update student csvs	2022-09-26 15:57:56 -07:00
rebuilt	dd9ed86dcd	Merge branch 'scrapers'	2022-09-21 19:54:24 -07:00
rebuilt	f556844b73	Update a-sust-i1 so it's only applicable to high schools and lower ... benchmark for a-cure-i1 from 6.7 to 5 percent.	2022-09-21 19:50:33 -07:00
rebuilt	9e82023e48	Make sure we don't divide by 0	2022-09-20 16:44:11 -07:00
rebuilt	1a984d2cf0	Make sure we don't divide by 0	2022-09-20 16:43:19 -07:00
rebuilt	3ee909bbea	Update 3B-2 with an absolute floor	2022-09-20 14:45:20 -07:00
rebuilt	1be6e79fd8	Add scrapers for admin data and their corresponding csv files of data. ... Make sure specs are off so they don't run in github CI	2022-09-19 19:37:39 -07:00
rebuilt	9e18bf2151	Turn off specs for scrapers. Generate csvs for scraped data	2022-09-13 22:59:04 -07:00
rebuilt	75cea539c5	Change scraped csv file names from word to number format 'one_a_one.csv' ... -> '1A_1.csv''	2022-09-13 10:46:18 -07:00
rebuilt	3e28be2d0d	Remove 'scraper' from file/class names add 3B-1	2022-09-12 11:24:57 -07:00
rebuilt	83ef9310a4	Add scraper for 3A-1 and 2C-1. Finishes #183105787 and #183105716	2022-09-07 11:34:42 -07:00
rebuilt	b5bc9c9de4	Add 2A-i admin data. Finishes #183105627	2022-09-05 22:37:53 -07:00
rebuilt	459393381b	Load admin data for 1A-iii. Finishes #183105516	2022-09-05 21:01:32 -07:00
rebuilt	f783ce9724	add one-a	2022-09-01 20:00:20 -07:00
rebuilt	048aacd641	Add admin data for 4D. Finishes #182818489 and #182793402	2022-08-30 21:46:23 -07:00
rebuilt	043f6de8e7	Prefer using over uniq so the filtering happens in sql	2022-08-24 19:17:54 -07:00
rebuilt	30eae881f3	Load correct column for 2021-22 year	2022-08-22 22:09:07 -07:00