Here’s what I saw: New Mastery Records: |= | 0.00% (15) ETA: > 4 daysĭid you catch it? Here it is again: ETA: > 4 days. With round trip in a datacenter at 0.5ms, one approach might be to have each query run in 1ms, and 18 processes working in parallel. That means I would need 18 queries to run in 1ms. I should have started by doing a little math: what would it take to get this done in ~8 hours? There are about 8 hours every evening when usage is low enough to run it. Student_topic.update_attribute(:new_mastery_score, score)Ī script like this produces noticeable load against the production server, so we can’t run it during peek hours. Score = student_topic.calculate_mastery_score_the_old_way StudentTopic.find_each do |student_topic|
Task :sync_mastery_scores => :environment do We have a Rails app, and I wrote the script as a rake task. I built the simplest possible script, told the team we could start doing QA on our staging machine tomorrow, and went home to run the script on staging. For each record I needed two queries: calculate the current score, then create a new record for the new engine with the same score. This post is a deep dive into what I learned, and mistakes I made, when writing that backfill. At the time, I glossed over the problem of backfilling what was, at the time, about 250,000,000 rows. Not so long ago, I wrote about how we swapped out our old mastery engine for a shiny, new one without any downtime.
0 kommentar(er)
