mirror of
https://github.com/gabehf/Koito.git
synced 2026-04-22 12:01:52 -07:00
8ce6ec494d
Adopts ListenBrainz-inspired patterns to speed up imports from ~24h to under 30 minutes for 49k scrobbles. Phase 1 - track_lookup cache table: - New migration (000006) adds persistent entity lookup cache - Maps normalized (artist, track, album) → (artist_id, album_id, track_id) - SubmitListen fast path: cache hit skips 18 DB queries → 2 queries - Cache populated after entity resolution, invalidated on merge/delete - Benefits both live scrobbles and imports Phase 2 - SaveListensBatch: - New batch listen insert using pgx CopyFrom → temp table → INSERT ON CONFLICT - Thousands of inserts per second vs one-at-a-time Phase 3 - BulkSubmitter: - Reusable import accelerator for all importers - Pre-deduplicates scrobbles by (artist, track, album) in memory - Worker pool (4 goroutines) for parallel entity creation on cache miss - Batch listen insertion via SaveListensBatch Phase 4 - Migrate importers: - Maloja, Spotify, LastFM, ListenBrainz importers use BulkSubmitter - Koito importer left as-is (already fast with pre-resolved IDs) Phase 5 - Skip image lookups during import: - GetArtistImage/GetAlbumImage calls fully skipped when SkipCacheImage=true - Background tasks (FetchMissingArtistImages/FetchMissingAlbumImages) backfill Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> |
||
|---|---|---|
| .. | ||
| album.go | ||
| album_test.go | ||
| artist.go | ||
| artist_test.go | ||
| counts.go | ||
| counts_test.go | ||
| exports.go | ||
| images.go | ||
| images_test.go | ||
| interest.go | ||
| interest_test.go | ||
| listen.go | ||
| listen_activity.go | ||
| listen_activity_test.go | ||
| listen_test.go | ||
| merge.go | ||
| merge_test.go | ||
| psql.go | ||
| psql_test.go | ||
| search.go | ||
| search_test.go | ||
| sessions.go | ||
| sessions_test.go | ||
| top_albums.go | ||
| top_albums_test.go | ||
| top_artists.go | ||
| top_artists_test.go | ||
| top_tracks.go | ||
| top_tracks_test.go | ||
| track.go | ||
| track_lookup.go | ||
| track_test.go | ||
| user.go | ||
| user_test.go | ||