690f2db863
ECP-162 fix benefits so it gets scored correctly
5 months ago
a3082037fa
fix: ensure loaded survey item responses are uniq by id to avoid PG::CardinalityViolation: ERROR: ON CONFLICT DO UPDATE command cannot affect row a second time (PG::CardinalityViolation)
5 months ago
1da1b238e0
fix: PG::CardinalityViolation: ERROR: ON CONFLICT DO UPDATE command cannot affect row a second time (PG::CardinalityViolation) HINT: Ensure that no rows proposed for insertion within the same command have duplicate constrained values.
5 months ago
cab0a3955e
finish up adding socio economic disag filter
6 months ago
a258b32b39
WIP: create backend for socio-economic-status
6 months ago
9d8543afae
feat: Update cleaner and uploader to load gender data from parent surveys
...
Implement graph views for parent gender data
7 months ago
5b00454a1b
ECP-125 feat:
...
Update cleaner to read parent races.
Update uploader to set races for parents.
Add race graphs to analyze page. Show both measure and scale level graphs
7 months ago
76f2467b97
chore: ECP-152 Reduce batch size of survey uploads to 100 at a time from 500 to reduce memory usage and also explicitly run garbage collection so we stay under the 500 mb limit of heroku
7 months ago
0f457becf0
feat: create a parents by language graph
...
Update demographics table with lanugage options
Create a lanugage table to hold the new languages
Update the demographic loader to input languages into the database
Update the cleaner to read the language column
Update the parent table to hold a reference to a language
Update the data uploader script to read the language from the csv and update the language information for any parent items that already exist (or create database entries if none already exist)
update the analyze interface to add controls for selecting ‘parents by group’ and a dropdown for ‘parent by language’
Update the analyze controller to read the parent-by-group parameter
Create a graph for the parent-by-group view
Bubble up averages for language calculations.
Make sure n-size only counts responses for a given measure.
7 months ago
beb93aa8e8
fix: make sure all likert scores get counted even when the survey item id has different capitalization. Add tests for uploading parent data. Change the parent response rate calcuation to count all students in the school instead of just for the grades that were given the student survey
1 year ago
0c661930e9
Show parent response rate
1 year ago
e5e969b968
feat: add parent survey gauges
1 year ago
532ca39e84
chore: speed up survey data loader
2 years ago
33da0859b9
Split academic year into seasons if the academic year's range is
...
initialized with a season, i.e. "2024-25 Fall". Update scapers for
admin data, enrollment and staffing to use the new range standard
correctly. Update the loaders for admin data, enrollment and staffing
so that it populates all seasons in a given year. So admin data for
2024-25 gets loaded into "2024-25 Fall" and "2024-25 Spring". Add tests
for the new range format. Set the default cutoff for the start of Spring season will be the last Sunday in February
2 years ago
Gabe Farrell
5789ebf564
Faster admin data loader + rename School.school_hash
2 years ago
56b512810c
chore: remove lowell seeding rules
2 years ago
880b438eb4
chore: reenable test spec that tests data loader for races
2 years ago
36e21515c3
chore: refactor Race out of survey_item_values
2 years ago
e7fb009425
chore: refactor Gender out out of survey_item_values row
2 years ago
cddea60c8b
feat: reduce number of rows to process at one time to reduce memory use
2 years ago
1a707eb6bc
feat: load student responses in the same pass as loading the survey responses
...
chore: remove student loader since loading students is now done with the survey response loader
2 years ago
48e795fcfb
feat: add special education disaggregation
2 years ago
060d7aa55a
Add disaggregation by ELL
2 years ago
a785c69c44
Add Overall Response Rate
2 years ago
4f035f6a63
feat: Add income table to the database. Add seeder for income. Add a reference to income from survey item response. Update the loader to import income data from the survey response csv. Refactor analyze controller to extract presenter. Add corresponding specs. Add income graph to analyze page
2 years ago
d72f8d31e0
fix: There was an n+1 problem where we looked up the list of schools for
...
every row. Now we query the list of schools just once per file
2 years ago
e8aa75bf66
feat: update survey_item_response table to indlude recorded date and import recorded date when loading responses
2 years ago
76b79b99c2
Fix: Parse headers when they are surrounded by quotes. This helps load recent csv files correctly
3 years ago
d6b2521883
Fix regression in student loader
3 years ago
4509c157fa
Add automated data cleaning. Modify SurveyItemValues class to use regex
...
instead of hard coded values. Produce a clean csv and a csv with all
the removed values and columns with reason for removal. Add script for
running cleaning for each project
3 years ago
c15cb7b483
Change survey data loader spec to use factorybot objects instead of loading seeds. Change databasecleaner to use transaction. Add back babel-preset dependency to fix failing javascript test in production.
3 years ago
c0332955f3
move csv require statement to application.rb
3 years ago
c6b8518cb1
Fix bug with not all survey responses loading when using sftp loader
3 years ago
7a159f7ddf
Process 1000 rows at a time to limit memory usage in production
3 years ago
03c6bff7bb
Load survey responses from sftp folder
3 years ago
a1b7db9b2e
Add ability to load student demographic information for only lowell
3 years ago
fc9e73f656
make sure survey responses for Butler get loaded correctly
3 years ago
9bbba64e9e
Add rules to the survey data loader to allow skipping non-lowell schools
3 years ago
43bc55ed32
Query for survey items once instead of on each row
3 years ago
83fa255e5d
Make utility method private
3 years ago
3ca270e3b1
Fix more malformed csv errors
3 years ago
360579852c
Send fewer sql queries when loading survey responses
3 years ago
9ae94a2dae
further reduce batch size to avoid exceeding memory quota
3 years ago
bb472ce6c3
Reduce number of rows to process to avoid exceeding memory limit on heroku
3 years ago
7d7bee6498
Add gender disagreggation
3 years ago
a1f3dfaa33
Load gender information to survey item responses
3 years ago
f9c4ed8880
make sure grade gets updated when loading survey responses
3 years ago
89a7f27b88
Finishes #183092367 . Adds ui and graphs to show grade level disaggregations
3 years ago
bb5f668497
Refactor code based on rubocop and reek suggestions
3 years ago
18b479b8b2
Refactor based on rubocop and reek suggestions
3 years ago
rebuilt
Gabe Farrell