Update logic for calculating student response rate. Remove references

to survey table.  We no longer check or keep track of the survey type.
Instead we look in the database to see if a survey item has at least 10
responses.  If it does, that survey item was presented to the respondent
and we count it, and all responses when calculating the response rate.

Remove response rate timestamp from caching logic because we no longer
add the response rate to the database. All response rates are calculated
on the fly

Update three_b_two scraper to use teacher only numbers

swap over to using https://profiles.doe.mass.edu/statereport/gradesubjectstaffing.aspx as the source of staffing information
This commit is contained in:
rebuilt 2023-04-02 21:07:58 -07:00
parent 0bfde2805a
commit 07ed8dd259
38 changed files with 26376 additions and 425 deletions

21
lib/tasks/scrape.rake Normal file
View file

@ -0,0 +1,21 @@
namespace :scrape do
desc 'scrape dese site for admin data'
task admin: :environment do
puts 'scraping data from dese'
scrapers = [Dese::OneAOne, Dese::OneAThree, Dese::TwoAOne, Dese::TwoCOne, Dese::ThreeAOne, Dese::ThreeATwo,
Dese::ThreeBOne, Dese::ThreeBTwo, Dese::FourAOne, Dese::FourBTwo, Dese::FourDOne, Dese::FiveCOne, Dese::FiveDTwo]
scrapers.each do |scraper|
scraper.new.run_all
end
end
desc 'scrape dese site for teacher staffing information'
task enrollment: :environment do
Dese::ThreeATwo.new.scrape_enrollments(filepath: Rails.root.join('data', 'enrollment', 'enrollment.csv'))
end
desc 'scrape dese site for student staffing information'
task staffing: :environment do
Dese::Staffing.new.run_all
end
end