149 Commits (85ba1218e364c3f87c7614fe6f6a33600324a4e5)

Author	SHA1	Message	Date
Nelson Jovel	2ef24caf70	Lower threshold for the number of valid student responses from 17 to 11	2 years ago
Nelson Jovel	6365cad71b	chore: refactor code to put logic into models	2 years ago
Nelson Jovel	f27a590c5a	Convert gender and race text into qualtrics codes during cleaning. Abide by 'prefer not to disclose' for self reported race. Give priority to self reported data but use SIS information as backup	2 years ago
Nelson Jovel	81b683e95c	chore: Add comment about StaffingLoader also cloning enrollment data when it clones staffing data	2 years ago
Nelson Jovel	11617abd89	fix: make sure to grab the 'Gender- Qcode' column	2 years ago
Nelson Jovel	3d9bbb52eb	fix: make sure to populate total_students when loading enrollement	2 years ago
Nelson Jovel	123b570bb9	Detect and print message when there are duplicate headers during ... cleaning process	2 years ago
Nelson Jovel	6d84204f83	Add race and gender columns to cleaned cvs files when those headers are ... missing	2 years ago
Nelson Jovel	a11a134805	fix: also load enrollment data when loading staffing information for the latest year	2 years ago
Nelson Jovel	1b82ff9413	chore: since it's now possible for there to be multiple district and dese id columns separated by a dash and a number, be more explicit when we only want to filter out survey item ids that end in a -1	2 years ago
Nelson Jovel	a3f9e46414	chore: when searching for dese id, split up pattern so that to be more explicit about the order in which to search out the columns that might have the dese ID we're looking for.	2 years ago
rebuilt	02b335ef63	feat: reduce number of rows to process at one time to reduce memory use	2 years ago
rebuilt	019b954ffa	feat: load student responses in the same pass as loading the survey responses ... chore: remove student loader since loading students is now done with the survey response loader	2 years ago
rebuilt	b2fdbe5756	feat: We no longer trust the progress number that gets exported from qualtrics. Instead during the cleaning progress, perform a manual count of the number of responses to filter out rows that don't meet the minimum threshold.	2 years ago
rebuilt	ef44c41965	feat: add special education disaggregation	2 years ago
rebuilt	18ab51c860	chore: upgrade to rails 7.1. ... upgrade rspec fix failing tests upgrade devise	2 years ago
rebuilt	2fd56047d4	Add disaggregation by ELL	2 years ago
rebuilt	490522eb1e	feat: support multiple columns for race and gender information	2 years ago
rebuilt	ce5340648f	fix: hide scores on analyze page for scores that don't meet the student threshold of 25%	2 years ago
rebuilt	7bd7923d41	fix: ensure cleaner outputs columns for all survey items. Before the fix, if a survey item varient (ending in -1, ie s-tint-q1-1) did not have a matching survey item s-tint-q1, the resulting csv would not include that column	2 years ago
rebuilt	2c9df34fac	fix: start fixing problem with variants not getting added to the cleaned csv	2 years ago
rebuilt	a0807295ed	feat: add popover to analyze graphs that displays the n-size of the different columns. Make sure to only calculate a score for a race if there are more than 10 respondents to a question.	2 years ago
rebuilt	45e324e7d2	chore: remove outdated admin data loader file. We now use Dese::Loader to load school level data	2 years ago
rebuilt	2ac30bb107	feat: Add income table to the database. Add seeder for income. Add a reference to income from survey item response. Update the loader to import income data from the survey response csv. Refactor analyze controller to extract presenter. Add corresponding specs. Add income graph to analyze page	2 years ago
rebuilt	23ddaed2ce	feat: if admin data value is above 5, round down to 5	2 years ago
rebuilt	878ba08a22	fix: There was an n+1 problem where we looked up the list of schools for ... every row. Now we query the list of schools just once per file	2 years ago
rebuilt	c4b90a711c	Fix: make sure values don't get reordered after copying over row values from survey item variants. This fixes a problem where cleaner would produce a row with likert scores that got shifted to align with the wrong column	3 years ago
rebuilt	5f66d3d99a	Fix: enable correct detection of student surveys types by rejecting any ... headers ending with '-1' (the variants of standard questions)	3 years ago
rebuilt	02a2c6cd97	Don't write a file if there's an empty dataset	3 years ago
rebuilt	3f4aa8e7af	Fix: Parse headers when they are surrounded by quotes. This helps load recent csv files correctly	3 years ago
rebuilt	0f23053294	It's possible for admin data likert score values to be above 5. If that happens, we ... cap the likert score at 5. This was happening already at the scraper level but it's also now being done by the admin data loader for safety. Also make sure to just update admin data instead of deleting and reloading all values each load. Add tests to confirm this behavior	3 years ago
rebuilt	c789c46032	Don't check standard deviation for early education surveys	3 years ago
rebuilt	e058c523b6	Missing progress or duration information does not result in a row removed in the cleaning process	3 years ago
rebuilt	a71ebbc4e4	Add Overall Response Rate	3 years ago
rebuilt	435bc4a5be	Fix regression in student loader	3 years ago
rebuilt	dbfc9d1d3a	Add automated data cleaning. Modify SurveyItemValues class to use regex ... instead of hard coded values. Produce a clean csv and a csv with all the removed values and columns with reason for removal. Add script for running cleaning for each project	3 years ago
rebuilt	2e0d266434	Calculate response rate on the fly instead of looking it up from the db ... when calculating response rates.	3 years ago
rebuilt	654bb7df08	Remove unused filename	3 years ago
rebuilt	91bacfeb13	Fix ThreeATwo scraper	3 years ago
rebuilt	65b8599c6e	Update logic for calculating student response rate. Remove references ... to survey table. We no longer check or keep track of the survey type. Instead we look in the database to see if a survey item has at least 10 responses. If it does, that survey item was presented to the respondent and we count it, and all responses when calculating the response rate. Remove response rate timestamp from caching logic because we no longer add the response rate to the database. All response rates are calculated on the fly Update three_b_two scraper to use teacher only numbers swap over to using https://profiles.doe.mass.edu/statereport/gradesubjectstaffing.aspx as the source of staffing information	3 years ago
rebuilt	05bff3ee89	Memoize schools in SurveyItemValues and academic_years in AcademicYear ... for performace improvement	3 years ago
rebuilt	282a671531	Change survey data loader spec to use factorybot objects instead of loading seeds. Change databasecleaner to use transaction. Add back babel-preset dependency to fix failing javascript test in production.	3 years ago
rebuilt	825259bdd8	Merge branch 'rpp-response-rate' into rpp-main to bring in improvements ... to how we get enrollment and staffing information. Also speed up tests	3 years ago
rebuilt	6b31fa9115	Batch imports for staffing data	3 years ago
rebuilt	d059177f0c	load total students and batch importing records	3 years ago
rebuilt	2362d884eb	Convert admin data loader from using seeder to using factory	3 years ago
rebuilt	d0219217de	Convert response rate loader spec from using the seeder to using the factory	3 years ago
rebuilt	06f9d2f0e9	Scrape enrollment and staffing information. Seed enrollment and staffing information. Update DatabaseCleaner so it cleans up leftover information in the database. Remove old admin csvs from codebase.	3 years ago
rebuilt	c0332955f3	move csv require statement to application.rb	3 years ago
rebuilt	ae6d7151b3	Load student respondent information from enrollment loader	3 years ago