Commit graph

641 commits

Author SHA1 Message Date
Oneric
1a4238bf98 cosmetic: fix concurrent index creation warnings
Since those old migrations will now most likely only run during db init,
there’s not much point in running them in the background concurrently
anyway, so just drop the cncurrent setting rather than disabling
migration locks.
2024-06-19 02:25:23 +02:00
Mark Felder
5da9cbd8a5 RichMedia refactor
Rich Media parsing was previously handled on-demand with a 2 second HTTP request timeout and retained only in Cachex. Every time a Pleroma instance is restarted it will have to request and parse the data for each status with a URL detected. When fetching a batch of statuses they were processed in parallel to attempt to keep the maximum latency at 2 seconds, but often resulted in a timeline appearing to hang during loading due to a URL that could not be successfully reached. URLs which had images links that expire (Amazon AWS) were parsed and inserted with a TTL to ensure the image link would not break.

Rich Media data is now cached in the database and fetched asynchronously. Cachex is used as a read-through cache. When the data becomes available we stream an update to the clients. If the result is returned quickly the experience is almost seamless. Activities were already processed for their Rich Media data during ingestion to warm the cache, so users should not normally encounter the asynchronous loading of the Rich Media data.

Implementation notes:

- The async worker is a Task with a globally unique process name to prevent duplicate processing of the same URL
- The Task will attempt to fetch the data 3 times with increasing sleep time between attempts
- The HTTP request obeys the default HTTP request timeout value instead of 2 seconds
- URLs that cannot be successfully parsed due to an unexpected error receives a negative cache entry for 15 minutes
- URLs that fail with an expected error will receive a negative cache with no TTL
- Activities that have no detected URLs insert a nil value in the Cachex :scrubber_cache so we do not repeat parsing the object content with Floki every time the activity is rendered
- Expiring image URLs are handled with an Oban job
- There is no automatic cleanup of the Rich Media data in the database, but it is safe to delete at any time
- The post draft/preview feature makes the URL processing synchronous so the rendered post preview will have an accurate rendering

Overall performance of timelines and creating new posts which contain URLs is greatly improved.
2024-06-09 17:33:48 +01:00
floatingghost
72871d4514 Merge pull request 'Drop unused indices' (#767) from Oneric/akkoma:purge-unused-indices into develop
Reviewed-on: https://akkoma.dev/AkkomaGang/akkoma/pulls/767
2024-05-28 01:35:18 +00:00
Oneric
5256678901 Fix Exiftool migration id
Applying works fine with a 20220220135625 version, but it won’t be
rolled back in the right order. Fortunately this action is idempotent
so we can just rename and reapply it with a new id.

To also not break large-scale rollbacks past 2022 for anyone
who already applied it with the old id, keep a stub migration.
2024-05-07 00:16:21 +02:00
Oneric
b7e3d44756 Drop unused indices
This promotes and expands our existing optional migration.
Based on usage statistics from several instances, see:
https://akkoma.dev/AkkomaGang/akkoma/issues/764

activities_hosts is now retained after all since it’s essential
for the "instance" query parameter of *oma’s public timeline to
reliably work in a reasonable amount of time. (Although akkoma-fe has
no support for this feature and apparently barely anyone uses it.)

activities_actor_index was already dropped before in
20221211234352_remove_unused_indices; no need to drop it again.

Birthday indices were introduced in pleroma starting with
20220116183110_add_birthday_to_users which is past the
last common migration 20210416051708.
2024-05-02 00:08:33 +02:00
timorl
cd7af81896
Rename StripLocation to StripMetadata for temporal-proofing reasons 2024-04-16 20:37:00 +02:00
timorl
b144218dce
Merge branch 'develop' into elseinspe 2024-04-14 20:31:33 +02:00
Oneric
376f6b15ca Add ability to auto-approve followbacks
Resolves: https://akkoma.dev/AkkomaGang/akkoma/issues/148
2024-02-13 15:42:37 +01:00
Oneric
8cf183cb42 Drop Chat tables
Chats were removed in 0f132b802d
2024-02-11 05:15:08 +01:00
FloatingGhost
5c164028cf ensure ap_enabled true if coming back pleroma 2023-08-16 23:11:36 +01:00
FloatingGhost
6139c3346d Add extra rollbacks to pleroma develop 2023-08-16 22:49:23 +01:00
FloatingGhost
98cb255d12 Support elixir1.15
OTP builds to 1.15

Changelog entry

Ensure policies are fully loaded

Fix :warn

use main branch for linkify

Fix warn in tests

Migrations for phoenix 1.17

Revert "Migrations for phoenix 1.17"

This reverts commit 6a3b2f15b7.

Oban upgrade

Add default empty whitelist

mix format

limit test to amd64

OTP 26 tests for 1.15

use OTP_VERSION tag

baka

just 1.15

Massive deps update

Update locale, deps

Mix format

shell????

multiline???

?

max cases 1

use assert_recieve

don't put_env in async tests

don't async conn/fs tests

mix format

FIx some uploader issues

Fix tests
2023-08-03 17:44:09 +01:00
Weblate
eba3cce77b Update translation files
Updated by "Squash Git commits" hook in Weblate.

Translation: Pleroma fe/Akkoma Backend (Config Descriptions)
Translate-URL: http://translate.akkoma.dev/projects/akkoma/akkoma-backend-config-descriptions/
2023-07-27 13:14:05 +00:00
ilja
3a13f91fff Adapt some migrations so they can be rolled back
This is useful for people who want to migrate back to Pleroma.
It's also added in the docs, but also noted that this is barely tested and to be used at their own risk.
2023-06-09 22:02:26 +02:00
FloatingGhost
ab34680554 switch to using an enum system for DM acceptance 2023-05-23 10:29:08 +01:00
FloatingGhost
d310f99d6a Add MRFs for direct message manipulation 2023-05-22 23:53:44 +01:00
ilja
8b2adc4fb4 Rename users_ap_id_COALESCE_follower_address_index for faster db restoration
By default Postgresql first restores the data and then the indexes when dumping and restoring the database.
Restoring index activities_visibility_index took a very long time.
users_ap_id_COALESCE_follower_address_index was later added because having this could speed up the restoration tremendously.
The problem now is that restoration apparently happens in alphabetical order, so this new index wasn't created yet
by the time activities_visibility_index needed it.
There were several work-arounds which included more complex steps during backup/restore.
By renaming this index, it should be restored first and thus activities_visibility_index can make use of it.
This speeds up restoration significantly without requiring more complex or unexpected steps from people.
2023-02-26 14:33:17 +01:00
ilja
b71db2f82d create_service_actor is now type Application
This is used for internal fetch and for relay. Both represent the instance and therefore are an aplication.
2023-02-04 21:00:21 +00:00
FloatingGhost
9c71782861 Test removed HTTP adapter 2022-12-11 23:50:31 +00:00
floatingghost
d55de5debf Remerge of hashtag following (#341)
this time with less idiot

Co-authored-by: FloatingGhost <hannah@coffee-and-dreams.uk>
Reviewed-on: https://akkoma.dev/AkkomaGang/akkoma/pulls/341
2022-12-05 12:58:48 +00:00
floatingghost
ec6bf8c3f7 revert 4a94c9a31e
revert Add ability to follow hashtags (#336)

Co-authored-by: FloatingGhost <hannah@coffee-and-dreams.uk>
Reviewed-on: https://akkoma.dev/AkkomaGang/akkoma/pulls/336
2022-12-04 20:04:09 +00:00
floatingghost
4a94c9a31e Add ability to follow hashtags (#336)
Co-authored-by: FloatingGhost <hannah@coffee-and-dreams.uk>
Reviewed-on: https://akkoma.dev/AkkomaGang/akkoma/pulls/336
2022-12-04 17:36:59 +00:00
floatingghost
db60640c5b Fixing up deletes a bit (#327)
Co-authored-by: FloatingGhost <hannah@coffee-and-dreams.uk>
Reviewed-on: https://akkoma.dev/AkkomaGang/akkoma/pulls/327
2022-12-01 15:00:53 +00:00
floatingghost
0cfd5b4e89 Add ability to set a default post expiry (#321)
Co-authored-by: FloatingGhost <hannah@coffee-and-dreams.uk>
Reviewed-on: https://akkoma.dev/AkkomaGang/akkoma/pulls/321
2022-11-28 13:34:54 +00:00
@luna@f.l4.pm
a90c45b7e9 Add Signed Fetch Statistics (#312)
Close #304.

Notes:
 - This patch was made on top of Pleroma develop, so I created a separate cachex worker for request signature actions, instead of Akkoma's instance cache. If that is a merge blocker, I can attempt to move logic around for that.
 - Regarding the `has_request_signatures: true -> false` state transition: I think that is a higher level thing (resetting instance state on new instance actor key) which is separate from the changes relevant to this one.

Co-authored-by: Luna <git@l4.pm>
Reviewed-on: https://akkoma.dev/AkkomaGang/akkoma/pulls/312
Co-authored-by: @luna@f.l4.pm <akkoma@l4.pm>
Co-committed-by: @luna@f.l4.pm <akkoma@l4.pm>
2022-11-26 19:22:56 +00:00
FloatingGhost
e0032e4799 Add rollbacks for associated_object_id 2022-11-07 00:08:20 +00:00
floatingghost
b7e8ce2350 Scrape instance nodeinfo (#251)
Co-authored-by: FloatingGhost <hannah@coffee-and-dreams.uk>
Reviewed-on: https://akkoma.dev/AkkomaGang/akkoma/pulls/251
2022-11-06 22:49:39 +00:00
Ilja
338612d72b Use EXIF data of image to prefill image description
During attachment upload Pleroma returns a "description" field.

* This MR allows Pleroma to read the EXIF data during upload and return the description to the FE using this field.
    * If a description is already present (e.g. because a previous module added it), it will use that
    * Otherwise it will read from the EXIF data. First it will check -ImageDescription, if that's empty, it will check -iptc:Caption-Abstract
    * If no description is found, it will simply return nil, which is the default value
* When people set up a new instance, they will be asked if they want to read metadata and this module will be activated if so

There was an Exiftool module, which has now been renamed to Exiftool.StripLocation
2022-10-23 14:46:16 +02:00
floatingghost
c6e63aaf6b Backend settings sync (#226)
Co-authored-by: FloatingGhost <hannah@coffee-and-dreams.uk>
Reviewed-on: https://akkoma.dev/AkkomaGang/akkoma/pulls/226
2022-10-06 16:22:15 +00:00
FloatingGhost
ee2eb7752d Ensure rollback succeeds 2022-09-16 13:00:40 +01:00
floatingghost
911f8feb0a Ensure migrations succeed (#216)
Co-authored-by: FloatingGhost <hannah@coffee-and-dreams.uk>
Reviewed-on: https://akkoma.dev/AkkomaGang/akkoma/pulls/216
2022-09-16 11:53:11 +00:00
Hélène
b6891fe190 Migrations: generate unset user keys
User keys are now generated on user creation instead of "when needed",
to prevent race conditions in federation and a few other issues. This
migration will generate keys missing for local users.
2022-09-11 19:53:31 +01:00
floatingghost
2641dcdd15 Post editing (#202)
Rebased from #103

Co-authored-by: Tusooa Zhu <tusooa@kazv.moe>
Co-authored-by: FloatingGhost <hannah@coffee-and-dreams.uk>
Reviewed-on: https://akkoma.dev/AkkomaGang/akkoma/pulls/202
2022-09-06 19:24:02 +00:00
floatingghost
8e4de118c1 Don't persist local undone follow (#194)
same deal but backwards this time

Co-authored-by: FloatingGhost <hannah@coffee-and-dreams.uk>
Reviewed-on: https://akkoma.dev/AkkomaGang/akkoma/pulls/194
2022-08-31 18:00:36 +00:00
floatingghost
62e179f446 make conversation-id deterministic (#154)
Reviewed-on: https://akkoma.dev/AkkomaGang/akkoma/pulls/154
2022-08-06 20:59:15 +00:00
floatingghost
0ec3a11895 don't persist undo of follows (#149)
Reviewed-on: https://akkoma.dev/AkkomaGang/akkoma/pulls/149
2022-08-05 13:28:56 +00:00
floatingghost
c9600dbbbf local-only-fixed (#138)
Reviewed-on: https://akkoma.dev/AkkomaGang/akkoma/pulls/138
2022-08-02 14:46:46 +00:00
floatingghost
54ed8760ff Merge branch 'from/upstream-develop/tusooa/server-announcements' into 'develop' (#85)
Reviewed-on: https://akkoma.dev/AkkomaGang/akkoma/pulls/85
2022-07-18 13:08:36 +00:00
floatingghost
3cbc401fe0 upgrade oban to v11 (#84)
Reviewed-on: https://akkoma.dev/AkkomaGang/akkoma/pulls/84
2022-07-18 10:48:49 +00:00
Pete
56203aae30 Add index hotspots 2022-07-01 13:59:26 +01:00
Tusooa Zhu
7726148472 Send emails i18n'd using backend-stored user language 2022-06-29 20:45:19 +01:00
FloatingGhost
d884087e7c re-add mastofe_settings column 2022-01-08 21:47:52 +00:00
FloatingGhost
0d012ebea1 Revert "Merge branch 'remove/mastofe' into 'develop'"
This reverts commit 6b3842cf50, reversing
changes made to 6b1282a829.
2022-01-08 21:44:37 +00:00
Ivan Tashkinov
a7bdefc208 mix format 2021-12-29 11:44:33 +03:00
Ivan Tashkinov
0c7fb520bf Added index on [:target_id, :relationship_type] to :user_relationships (speeds up Notification.exclude_blockers/_). 2021-12-29 11:41:21 +03:00
Alex Gleason
fa35e24a5e
Apps: add user_id index 2021-12-27 18:05:35 -06:00
Alex Gleason
f5c3d45120
Merge remote-tracking branch 'origin/develop' into apps-api-endpoint 2021-12-27 18:01:25 -06:00
marcin mikołajczak
de006443f0 MastoAPI: Profile directory 2021-12-26 02:35:17 +00:00
Alex Gleason
73609211a4 Merge branch 'account-notes' into 'develop'
MastoAPI: Add user notes on accounts

See merge request pleroma/pleroma!3540
2021-12-25 01:41:12 +00:00
Alex Gleason
bd853199d9 Merge branch 'v2-suggestions' into 'develop'
V2 suggestions

See merge request pleroma/pleroma!3547
2021-12-19 17:31:17 +00:00