feat: new modpack permissions system (#6005)

* Begin external projects moderator database frontend

* add copy link button

* begin project page permissions settings

* MEL database backend routes

* include filename in external files

* wip: when uploading a version file, fetch its overrides as a list

* wip: override license checks

* improve FileHost ref counting

* file host read capability

* scan files when inserting version file

* add dependency sha1 field

* clean up version files

* wip: attributions

* update s3 file host

* attribution scanning basic works

* works

* insert attribution info after resolving

* add routes

* remove dep sha1 stuff

* prepr

* wip: override file sources

* add files_missing_attributions to versions

* return extended version info + attributed at/by

* hook up frontend to backend (mostly)

* expose version date published

* withholding version visibility

* frontend work

* prepr

* use api-client for img upload

* moar frontend

* prepr

* Add schema to attribution resolution and Flame project results

* sqlx prepare

* changes

* remove feature flag, fix optional proof images

* fix schema

* fmt

* fix deletion and file fetch

* prepare

* fix admonition

* update frontend stuff to new schema

* prepr

* attribution on dependencies

* fixes

* sqlx prepare

* fixes

* routes

* fix routes

* Version grandfathering

* prepare

* wip: bulk routes

* pushing what i've got rn

* include link in NoPermission

* change hash insert to bulk route

* query flame even if entry in MEL

* delete file with weird name

* Prioritise putting override files in existing groups even with ExternalLicense

* fix how hex bytes are handled in route

* feat: coolbot moderation changes (#6215)

* Update moderator checklist

* move permissions stage order

* Updated nagContext.versions to v3, added nag for permissions

* Update permissions.vue default messages

* prepr

---------

Co-authored-by: coolbot100s <76798835+coolbot100s@users.noreply.github.com>

* QA

* prepr

* should group by project

* return attribution resolution correctly

* updated by moderator info

* Track what moderator reviewed an attribution moderation status

* default deser FMA field

* new version page

* clean up fetching + add a couple missing features

* qa items

* prepr

* provide moderation package stuff with DI

* format?

* don't redact moderated_at

* move supplementary resources

* Reorganize moderation messages.

* Quick replies for external content permissions.

* prepare

* QA

* allow exempting projects

* Ignore Flame projects which 404

* fix ci

* fix cross project attribution stuff

* Fix permission error

* change what files get cscanned

* add more logging

* QA Jun 22

* fix

* idempotency

* Expose route for rescanning

* update blog link

---------

Co-authored-by: aecsocket <aecsocket@tutanota.com>
Co-authored-by: coolbot100s <76798835+coolbot100s@users.noreply.github.com>
Co-authored-by: aecsocket <43144841+aecsocket@users.noreply.github.com>

apps/labrinth/src/background_task.rs

@@ -3,10 +3,12 @@ use crate::database::PgPool;
 use crate::database::models::ids::DBUserId;
 use crate::database::models::notification_item::NotificationBuilder;
 use crate::database::redis::RedisPool;
+use crate::file_hosting::FileHost;
 use crate::models::notifications::NotificationBody;
 use crate::queue::analytics::cache::cache_analytics;
 use crate::queue::billing::{index_billing, index_subscriptions};
 use crate::queue::email::EmailQueue;
+use crate::queue::file_scan::scan_all_files;
 use crate::queue::payouts::{
     PayoutsQueue, index_payouts_notifications,
     insert_bank_balances_and_webhook, process_affiliate_payouts,
@@ -38,6 +40,10 @@ pub enum BackgroundTask {
     /// Attempts to ping Minecraft Java servers as if we were a client, to
     /// collect info on if they're online, game version, description, etc.
     PingMinecraftJavaServers,
+    /// Finds files of versions which have not been scanned for attributions
+    /// yet, extracts them to find file overrides, and finds any overrides which
+    /// require attribution from the creator.
+    ScanFiles,
     /// Queues Discord Creator Club role claim emails for newly eligible users.
     DiscordRoleEmailCampaign,
 }
@@ -50,6 +56,7 @@ impl BackgroundTask {
         ro_pool: PgPool,
         redis_pool: RedisPool,
         search_backend: web::Data<dyn SearchBackend>,
+        file_host: web::Data<dyn FileHost>,
         kafka_client: web::Data<crate::util::kafka::KafkaClientState>,
         clickhouse: clickhouse::Client,
         stripe_client: stripe::Client,
@@ -112,6 +119,7 @@ impl BackgroundTask {
                 )
                 .await
             }
+            ScanFiles => scan_all_files(&pool, &redis_pool, &**file_host).await,
             DiscordRoleEmailCampaign => {
                 discord_role_email_campaign(pool, redis_pool).await
             }