Commit Graph
16 Commits
Author SHA1 Message Date
aecsocket b3b0b85691 fix: change incremental search indexing operation set (#7179)
* use ro_pool for search indexing again

* adjust incremental indexing operations
2026-08-17 17:00:12 +00:00
Sychic 549b333eb2 feat(labrinth): project disclosures (#6945)
* feat(labrinth): project disclosures model

* feat(labrinth): project disclosures database model

* feat(labrinth): project disclosures get endpoint

* feat(labrinth): censor user ids if set by moderator

* feat(labrinth): wrap disclosures in struct

* feat(labrinth): edit project disclosures endpoint

* style(labrinth): cargo fmt

* style(labrinth): fix typo

* feat(labrinth): add fields for ai content disclosure

* fix(labrinth): field typo

* chore(labrinth): update query cache

* feat(labrinth): index disclosures in search

* refactor(labrinth): use enum instead of bools

* feat(labrinth): trigger incremental index on disclosure change

* feat(labrinth): change archived status to disclosure

* feat(labrinth): use disclosure for archival status

* fix(labrinth): error type for deserialization

* fix(labrinth): migration timestamp

* fix(labrinth): add disclosures to elasticsearch schema

* fix(labrinth): don't remove archival disclosure when changing status via v3 api

* feat(labrinth): derive str for ai usages and telemtry consent

* feat(labrinth): include ai usages and telemetry consent in disclosure types

* refactor(labrinth): move disclosure parsing to document creation

* fix(labrinth): prevent removing moderator added archival disclosures

* fix(labrinth): dedupe ai usages

* fix(labrinth): auth logic on archived disclosure removal

* feat(labrinth): add interactions field

* style(labrinth): cargo fmt

* feat(labrinth): soft delete disclosures

* feat(labrinth): return soft-deleted disclosures

* fix(labrinth): update disclosure on removal

---------

Co-authored-by: Prospector <6166773+Prospector@users.noreply.github.com>
2026-08-12 17:26:14 -07:00
aecsocket c34bd26f87 fix: search indexing and schema changes (#7090)
* fix: incremental indexing uses latest db data

* dependent search fields
2026-08-12 11:54:35 -04:00
aecsocket 5148e8ec35 feat: use postcard for redis serde (#6956)
* redo error handling in xredis

* give proper types to metadata fields

* add round-trip tests

* inline loader enum metadata fields

* postcard roundtrips

* prepare

* bump redis key version

* serde-binhum

* clippy

* fix

* fix frontend checking existence of component fields rather than non-null-ness

* prepare
2026-08-05 10:36:05 +00:00
aecsocket c3249ee51d feat: ElasticSearch backend (#6903)
* (do not merge) search test branch

* perf

* fix up search pagination

* more parity and perf work

* expand what parity does

* fix author query

* approach parity even without explicit parity enabled

* remove old parity code

* fix shear

* fix

* fmt
2026-08-03 17:14:07 +00:00
aecsocket 158de019a6 fix: search indexing performance and batching (#6521)
* Add consume batching delay

* maybe fix

* max batch size

* more logging

* parallelize remove tasks

* delete/upsert project/version messages

* prepare

* more logging

* log number of docs

* try more targeted, homogenous version change ops

* ensure only necessary fields are serialized into typesense

* disable index background task

* wip: script changes

* don't facet by project id

* fix

* fix clippy

* batch by document and dedup loaders

* fix test

* wip: projects/versions collections

* clean up SearchBackend interface

* cleanup pass

* cleanup pass 2

* standardise fn names

* cleanup pass

* fix compile

* factor out filter rewriting

* query perf

* put categories into the project doc

* wip: search filter AST

* doc comment

* more AST normalization

* (temp) convert search request errors into internal errors

* implement unary NOT

* tombi fmt

* fix tests

* revert request error

* try increase stack size
2026-07-26 14:55:00 +00:00
François-Xavier Talbot b4d681e713 feat(labrinth): Redis Cluster (#6771)
* chore(labrinth): bump to redis 1.4.1

* feat(labrinth): redis cluster

* chore: cleanup

* feat(labrinth): cache locking

* fix(labrinth): clippy

* chore(labrinth): cleanup env, remove postcard support

* chore(ci): fix test env for labrinth

* chore(labrinth): bump all key versions

* chore(labrinth): improve redis key identities handling

* chore(labrinth): simplify deadline handling

* chore(labrinth): remove unused lease tracking

* chore(labrinth): remove distributed cache locking for now

* chore(labrinth): improve redis backend init error

* feat(labrinth): expose redis read replica strategy

* chore(ci): remove other connection mode tests

* chore: split xredis crate

* feat(xredis): primaries routing

* chore: tombi fmt

* chore: clippy

* chore: update query cache
2026-07-23 11:35:02 +02:00
Prospector 96ab120cd0 feat: add all_project_types to search results (#6654)
* feat: add all_project_types to search results

* update docs
2026-07-07 21:30:42 +00:00
aecsocket c6ef10e798 Incremental search indexing (#6453)
* wip: initial kafka support

* wip: kafka search queue

* wip: kafka consumer

* wip: create/delete docs

* reindexing works

* reindex in more places

* also incrementally update server pings

* cleanup

* tombi

* fix ci

* update tombi

* prepare

* fix ci

* fix

* fix PLEASE

* 🙏

* add route to force reindex project
2026-06-21 10:03:20 +00:00
aecsocket 64e17c7c1b Include compatible dependencies field in project search (#6366)
* include compatible dependency projects field

* fix
2026-06-11 20:18:44 +00:00
aecsocket d2a66bb2b0 Allow searching by project dependencies (#6350)
* Allow searching by project dependencies

* change field name

* use query macro
2026-06-10 17:30:03 +00:00
aecsocket 9403462915 Update to rustc 1.95.0 (#5962)
* Update to rustc 1.95.0

* fmt
2026-05-01 19:39:14 +00:00
Sychic 6862cf5ab2 Show orgs in project card when a project is owned by an org (#5892)
* fix: link to user using id instead of username

* feat: show org in project card

* fix: account for outdated documents

* refactor: use struct to store owner information

* fix: default new fields

* fix lint
2026-04-23 17:32:19 +00:00
aecsocket 3c5bd0756d Index search by original and split title (#5589)
* Index search by original and split title

* better normalization of title/author names for indexing

* replace println with warn

* fix test
2026-03-20 04:01:19 +00:00
aecsocket 991b4d8c13 Fix Typesense tests (#5541)
* Fix Typesense tests

* fix

* add back author

* Split project title and authors by words and index on that

* clippy
2026-03-13 13:34:51 +00:00
aecsocket f0224dfff7 Search backend refactor with typesense impl (#5528)
* initial elasticsearch impl

* working elastic cluster

* replace SearchError with ApiError for preparation of search backend

* start factoring meili out to trait

* move meili to backend

* update routes to use search backend trait

* wip

* Update projects.rs

* search backend is only init'd once in config

* wip

* wip: backend agnostic

* change search internal routes to delegate to backend

* initial elasticsearch impl

* fix filtering

* elastic impl

* refactor indexing into its own module

* clean up elastic code

* fix ci

* fix tests

* fix elastic health check

* fix up env rebase

* fix compile

* dummy commit to update github pr

* Fix rebase

* Elastic basic https auth

* Fix duplicate projects showing up

* Fix up tests

* Replace search `ApiErrors` with `eyre::Reports`, propagate background task errors

* clean up agents files

* make index chunk size configurable

* make `match_phrase` in elastic case-insensitive

* use current/next indices and swap between them

* test case for error body

* Fix failing case

* da merge

* factor out common stuff from search backends

* allow fetching hit metadata from search results

* allow customising elasticsearch search config

* bit of docs

* add mappings to indices for elastic

* Implement Typesense

* wip

* fix up some sort fields stuff

* use different approach to filterable field sets

* remove a bunch of search fields which weren't used for filtering

* bucket text matches

* Bucketing by text_match for typesense

* fix tombi lint

* fix some sentry errors and dont prioritise 2+ term matches

* tweak ts query settings

* expose some more search settings

* query sort changes

* small fixes

* should fix pagination stuff

* fix healthcheck maybe

* ragebait ci

* tests

* tests

* revert environment
2026-03-12 18:58:55 +01:00