darkkirb/akkoma
1
0
Fork 0
forked from mirrors/akkoma
Code Pull requests Activity

Merge remote-tracking branch 'upstream/develop' into email-fix-develop

Browse source
This commit is contained in:
Alex Gleason 2020-09-22 13:37:02 -05:00
commit 3104367931
No known key found for this signature in database
GPG key ID: 7211D1F99744FBB7
267 changed files with 8507 additions and 9189 deletions
Show stats Download patch file Download diff file Expand all files Collapse all files

395
CC-BY-4.0 Normal file
View file

@ -0,0 +1,395 @@
Attribution 4.0 International
=======================================================================
Creative Commons Corporation ("Creative Commons") is not a law firm and
does not provide legal services or legal advice. Distribution of
Creative Commons public licenses does not create a lawyer-client or
other relationship. Creative Commons makes its licenses and related
information available on an "as-is" basis. Creative Commons gives no
warranties regarding its licenses, any material licensed under their
terms and conditions, or any related information. Creative Commons
disclaims all liability for damages resulting from their use to the
fullest extent possible.
Using Creative Commons Public Licenses
Creative Commons public licenses provide a standard set of terms and
conditions that creators and other rights holders may use to share
original works of authorship and other material subject to copyright
and certain other rights specified in the public license below. The
following considerations are for informational purposes only, are not
exhaustive, and do not form part of our licenses.
Considerations for licensors: Our public licenses are
intended for use by those authorized to give the public
permission to use material in ways otherwise restricted by
copyright and certain other rights. Our licenses are
irrevocable. Licensors should read and understand the terms
and conditions of the license they choose before applying it.
Licensors should also secure all rights necessary before
applying our licenses so that the public can reuse the
material as expected. Licensors should clearly mark any
material not subject to the license. This includes other CC-
licensed material, or material used under an exception or
limitation to copyright. More considerations for licensors:
wiki.creativecommons.org/Considerations_for_licensors
Considerations for the public: By using one of our public
licenses, a licensor grants the public permission to use the
licensed material under specified terms and conditions. If
the licensor's permission is not necessary for any reason--for
example, because of any applicable exception or limitation to
copyright--then that use is not regulated by the license. Our
licenses grant only permissions under copyright and certain
other rights that a licensor has authority to grant. Use of
the licensed material may still be restricted for other
reasons, including because others have copyright or other
rights in the material. A licensor may make special requests,
such as asking that all changes be marked or described.
Although not required by our licenses, you are encouraged to
respect those requests where reasonable. More considerations
for the public:
wiki.creativecommons.org/Considerations_for_licensees
=======================================================================
Creative Commons Attribution 4.0 International Public License
By exercising the Licensed Rights (defined below), You accept and agree
to be bound by the terms and conditions of this Creative Commons
Attribution 4.0 International Public License ("Public License"). To the
extent this Public License may be interpreted as a contract, You are
granted the Licensed Rights in consideration of Your acceptance of
these terms and conditions, and the Licensor grants You such rights in
consideration of benefits the Licensor receives from making the
Licensed Material available under these terms and conditions.
Section 1 -- Definitions.
a. Adapted Material means material subject to Copyright and Similar
Rights that is derived from or based upon the Licensed Material
and in which the Licensed Material is translated, altered,
arranged, transformed, or otherwise modified in a manner requiring
permission under the Copyright and Similar Rights held by the
Licensor. For purposes of this Public License, where the Licensed
Material is a musical work, performance, or sound recording,
Adapted Material is always produced where the Licensed Material is
synched in timed relation with a moving image.
b. Adapter's License means the license You apply to Your Copyright
and Similar Rights in Your contributions to Adapted Material in
accordance with the terms and conditions of this Public License.
c. Copyright and Similar Rights means copyright and/or similar rights
closely related to copyright including, without limitation,
performance, broadcast, sound recording, and Sui Generis Database
Rights, without regard to how the rights are labeled or
categorized. For purposes of this Public License, the rights
specified in Section 2(b)(1)-(2) are not Copyright and Similar
Rights.
d. Effective Technological Measures means those measures that, in the
absence of proper authority, may not be circumvented under laws
fulfilling obligations under Article 11 of the WIPO Copyright
Treaty adopted on December 20, 1996, and/or similar international
agreements.
e. Exceptions and Limitations means fair use, fair dealing, and/or
any other exception or limitation to Copyright and Similar Rights
that applies to Your use of the Licensed Material.
f. Licensed Material means the artistic or literary work, database,
or other material to which the Licensor applied this Public
License.
g. Licensed Rights means the rights granted to You subject to the
terms and conditions of this Public License, which are limited to
all Copyright and Similar Rights that apply to Your use of the
Licensed Material and that the Licensor has authority to license.
h. Licensor means the individual(s) or entity(ies) granting rights
under this Public License.
i. Share means to provide material to the public by any means or
process that requires permission under the Licensed Rights, such
as reproduction, public display, public performance, distribution,
dissemination, communication, or importation, and to make material
available to the public including in ways that members of the
public may access the material from a place and at a time
individually chosen by them.
j. Sui Generis Database Rights means rights other than copyright
resulting from Directive 96/9/EC of the European Parliament and of
the Council of 11 March 1996 on the legal protection of databases,
as amended and/or succeeded, as well as other essentially
equivalent rights anywhere in the world.
k. You means the individual or entity exercising the Licensed Rights
under this Public License. Your has a corresponding meaning.
Section 2 -- Scope.
a. License grant.
1. Subject to the terms and conditions of this Public License,
the Licensor hereby grants You a worldwide, royalty-free,
non-sublicensable, non-exclusive, irrevocable license to
exercise the Licensed Rights in the Licensed Material to:
a. reproduce and Share the Licensed Material, in whole or
in part; and
b. produce, reproduce, and Share Adapted Material.
2. Exceptions and Limitations. For the avoidance of doubt, where
Exceptions and Limitations apply to Your use, this Public
License does not apply, and You do not need to comply with
its terms and conditions.
3. Term. The term of this Public License is specified in Section
6(a).
4. Media and formats; technical modifications allowed. The
Licensor authorizes You to exercise the Licensed Rights in
all media and formats whether now known or hereafter created,
and to make technical modifications necessary to do so. The
Licensor waives and/or agrees not to assert any right or
authority to forbid You from making technical modifications
necessary to exercise the Licensed Rights, including
technical modifications necessary to circumvent Effective
Technological Measures. For purposes of this Public License,
simply making modifications authorized by this Section 2(a)
(4) never produces Adapted Material.
5. Downstream recipients.
a. Offer from the Licensor -- Licensed Material. Every
recipient of the Licensed Material automatically
receives an offer from the Licensor to exercise the
Licensed Rights under the terms and conditions of this
Public License.
b. No downstream restrictions. You may not offer or impose
any additional or different terms or conditions on, or
apply any Effective Technological Measures to, the
Licensed Material if doing so restricts exercise of the
Licensed Rights by any recipient of the Licensed
Material.
6. No endorsement. Nothing in this Public License constitutes or
may be construed as permission to assert or imply that You
are, or that Your use of the Licensed Material is, connected
with, or sponsored, endorsed, or granted official status by,
the Licensor or others designated to receive attribution as
provided in Section 3(a)(1)(A)(i).
b. Other rights.
1. Moral rights, such as the right of integrity, are not
licensed under this Public License, nor are publicity,
privacy, and/or other similar personality rights; however, to
the extent possible, the Licensor waives and/or agrees not to
assert any such rights held by the Licensor to the limited
extent necessary to allow You to exercise the Licensed
Rights, but not otherwise.
2. Patent and trademark rights are not licensed under this
Public License.
3. To the extent possible, the Licensor waives any right to
collect royalties from You for the exercise of the Licensed
Rights, whether directly or through a collecting society
under any voluntary or waivable statutory or compulsory
licensing scheme. In all other cases the Licensor expressly
reserves any right to collect such royalties.
Section 3 -- License Conditions.
Your exercise of the Licensed Rights is expressly made subject to the
following conditions.
a. Attribution.
1. If You Share the Licensed Material (including in modified
form), You must:
a. retain the following if it is supplied by the Licensor
with the Licensed Material:
i. identification of the creator(s) of the Licensed
Material and any others designated to receive
attribution, in any reasonable manner requested by
the Licensor (including by pseudonym if
designated);
ii. a copyright notice;
iii. a notice that refers to this Public License;
iv. a notice that refers to the disclaimer of
warranties;
v. a URI or hyperlink to the Licensed Material to the
extent reasonably practicable;
b. indicate if You modified the Licensed Material and
retain an indication of any previous modifications; and
c. indicate the Licensed Material is licensed under this
Public License, and include the text of, or the URI or
hyperlink to, this Public License.
2. You may satisfy the conditions in Section 3(a)(1) in any
reasonable manner based on the medium, means, and context in
which You Share the Licensed Material. For example, it may be
reasonable to satisfy the conditions by providing a URI or
hyperlink to a resource that includes the required
information.
3. If requested by the Licensor, You must remove any of the
information required by Section 3(a)(1)(A) to the extent
reasonably practicable.
4. If You Share Adapted Material You produce, the Adapter's
License You apply must not prevent recipients of the Adapted
Material from complying with this Public License.
Section 4 -- Sui Generis Database Rights.
Where the Licensed Rights include Sui Generis Database Rights that
apply to Your use of the Licensed Material:
a. for the avoidance of doubt, Section 2(a)(1) grants You the right
to extract, reuse, reproduce, and Share all or a substantial
portion of the contents of the database;
b. if You include all or a substantial portion of the database
contents in a database in which You have Sui Generis Database
Rights, then the database in which You have Sui Generis Database
Rights (but not its individual contents) is Adapted Material; and
c. You must comply with the conditions in Section 3(a) if You Share
all or a substantial portion of the contents of the database.
For the avoidance of doubt, this Section 4 supplements and does not
replace Your obligations under this Public License where the Licensed
Rights include other Copyright and Similar Rights.
Section 5 -- Disclaimer of Warranties and Limitation of Liability.
a. UNLESS OTHERWISE SEPARATELY UNDERTAKEN BY THE LICENSOR, TO THE
EXTENT POSSIBLE, THE LICENSOR OFFERS THE LICENSED MATERIAL AS-IS
AND AS-AVAILABLE, AND MAKES NO REPRESENTATIONS OR WARRANTIES OF
ANY KIND CONCERNING THE LICENSED MATERIAL, WHETHER EXPRESS,
IMPLIED, STATUTORY, OR OTHER. THIS INCLUDES, WITHOUT LIMITATION,
WARRANTIES OF TITLE, MERCHANTABILITY, FITNESS FOR A PARTICULAR
PURPOSE, NON-INFRINGEMENT, ABSENCE OF LATENT OR OTHER DEFECTS,
ACCURACY, OR THE PRESENCE OR ABSENCE OF ERRORS, WHETHER OR NOT
KNOWN OR DISCOVERABLE. WHERE DISCLAIMERS OF WARRANTIES ARE NOT
ALLOWED IN FULL OR IN PART, THIS DISCLAIMER MAY NOT APPLY TO YOU.
b. TO THE EXTENT POSSIBLE, IN NO EVENT WILL THE LICENSOR BE LIABLE
TO YOU ON ANY LEGAL THEORY (INCLUDING, WITHOUT LIMITATION,
NEGLIGENCE) OR OTHERWISE FOR ANY DIRECT, SPECIAL, INDIRECT,
INCIDENTAL, CONSEQUENTIAL, PUNITIVE, EXEMPLARY, OR OTHER LOSSES,
COSTS, EXPENSES, OR DAMAGES ARISING OUT OF THIS PUBLIC LICENSE OR
USE OF THE LICENSED MATERIAL, EVEN IF THE LICENSOR HAS BEEN
ADVISED OF THE POSSIBILITY OF SUCH LOSSES, COSTS, EXPENSES, OR
DAMAGES. WHERE A LIMITATION OF LIABILITY IS NOT ALLOWED IN FULL OR
IN PART, THIS LIMITATION MAY NOT APPLY TO YOU.
c. The disclaimer of warranties and limitation of liability provided
above shall be interpreted in a manner that, to the extent
possible, most closely approximates an absolute disclaimer and
waiver of all liability.
Section 6 -- Term and Termination.
a. This Public License applies for the term of the Copyright and
Similar Rights licensed here. However, if You fail to comply with
this Public License, then Your rights under this Public License
terminate automatically.
b. Where Your right to use the Licensed Material has terminated under
Section 6(a), it reinstates:
1. automatically as of the date the violation is cured, provided
it is cured within 30 days of Your discovery of the
violation; or
2. upon express reinstatement by the Licensor.
For the avoidance of doubt, this Section 6(b) does not affect any
right the Licensor may have to seek remedies for Your violations
of this Public License.
c. For the avoidance of doubt, the Licensor may also offer the
Licensed Material under separate terms or conditions or stop
distributing the Licensed Material at any time; however, doing so
will not terminate this Public License.
d. Sections 1, 5, 6, 7, and 8 survive termination of this Public
License.
Section 7 -- Other Terms and Conditions.
a. The Licensor shall not be bound by any additional or different
terms or conditions communicated by You unless expressly agreed.
b. Any arrangements, understandings, or agreements regarding the
Licensed Material not stated herein are separate from and
independent of the terms and conditions of this Public License.
Section 8 -- Interpretation.
a. For the avoidance of doubt, this Public License does not, and
shall not be interpreted to, reduce, limit, restrict, or impose
conditions on any use of the Licensed Material that could lawfully
be made without permission under this Public License.
b. To the extent possible, if any provision of this Public License is
deemed unenforceable, it shall be automatically reformed to the
minimum extent necessary to make it enforceable. If the provision
cannot be reformed, it shall be severed from this Public License
without affecting the enforceability of the remaining terms and
conditions.
c. No term or condition of this Public License will be waived and no
failure to comply consented to unless expressly agreed to by the
Licensor.
d. Nothing in this Public License constitutes or may be interpreted
as a limitation upon, or waiver of, any privileges and immunities
that apply to the Licensor or You, including from the legal
processes of any jurisdiction or authority.
=======================================================================
Creative Commons is not a party to its public
licenses. Notwithstanding, Creative Commons may elect to apply one of
its public licenses to material it publishes and in those instances
will be considered the “Licensor.” The text of the Creative Commons
public licenses is dedicated to the public domain under the CC0 Public
Domain Dedication. Except for the limited purpose of indicating that
material is shared under a Creative Commons public license or as
otherwise permitted by the Creative Commons policies published at
creativecommons.org/policies, Creative Commons does not authorize the
use of the trademark "Creative Commons" or any other trademark or logo
of Creative Commons without its prior written consent including,
without limitation, in connection with any unauthorized modifications
to any of its public licenses or any other arrangements,
understandings, or agreements concerning use of licensed material. For
the avoidance of doubt, this paragraph does not form part of the
public licenses.
Creative Commons may be contacted at creativecommons.org.

32
CHANGELOG.md
View file

@ -3,6 +3,32 @@ All notable changes to this project will be documented in this file.
The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/).
## Unreleased
### Added
- Experimental websocket-based federation between Pleroma instances.
### Changed
- Renamed `:await_up_timeout` in `:connections_pool` namespace to `:connect_timeout`, old name is deprecated.
- Renamed `:timeout` in `pools` namespace to `:recv_timeout`, old name is deprecated.
- The `discoverable` field in the `User` struct will now add a NOINDEX metatag to profile pages when false.
- Users with the `discoverable` field set to false will not show up in searches.
- Minimum lifetime for ephmeral activities changed to 10 minutes and made configurable (`:min_lifetime` option).
### Added
- Media preview proxy (requires media proxy be enabled; see `:media_preview_proxy` config for more details).
- Pleroma API: Importing the mutes users from CSV files.
### Removed
- **Breaking:** `Pleroma.Workers.Cron.StatsWorker` setting from Oban `:crontab` (moved to a simpler implementation).
- **Breaking:** `Pleroma.Workers.Cron.ClearOauthTokenWorker` setting from Oban `:crontab` (moved to scheduled jobs).
- **Breaking:** `Pleroma.Workers.Cron.PurgeExpiredActivitiesWorker` setting from Oban `:crontab` (moved to scheduled jobs).
- Removed `:managed_config` option. In practice, it was accidentally removed with 2.0.0 release when frontends were
switched to a new configuration mechanism, however it was not officially removed until now.
## [2.1.2] - 2020-09-17
### Security
@ -40,6 +66,12 @@ The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/).
### Added
- Rich media failure tracking (along with `:failure_backoff` option).
<details>
<summary>Admin API Changes</summary>
- Add `PATCH /api/pleroma/admin/instance_document/:document_name` to modify the Terms of Service and Instance Panel HTML pages via Admin API
</details>
### Fixed
- Default HTTP adapter not respecting pool setting, leading to possible OOM.
- Fixed uploading webp images when the Exiftool Upload Filter is enabled by skipping them

2
Dockerfile
View file

@ -31,7 +31,7 @@ LABEL maintainer="ops@pleroma.social" \
ARG HOME=/opt/pleroma
ARG DATA=/var/lib/pleroma
RUN echo "http://nl.alpinelinux.org/alpine/latest-stable/community" >> /etc/apk/repositories &&\
RUN echo "https://nl.alpinelinux.org/alpine/latest-stable/community" >> /etc/apk/repositories &&\
apk update &&\
apk add exiftool imagemagick ncurses postgresql-client &&\
adduser --system --shell /bin/false --home ${HOME} pleroma &&\

7
README.md
View file

@ -18,15 +18,16 @@ If you are running Linux (glibc or musl) on x86/arm, the recommended way to inst
### From Source
If your platform is not supported, or you just want to be able to edit the source code easily, you may install Pleroma from source.
- [Debian-based](https://docs-develop.pleroma.social/backend/installation/debian_based_en/)
- [Debian-based (jp)](https://docs-develop.pleroma.social/backend/installation/debian_based_jp/)
- [Alpine Linux](https://docs-develop.pleroma.social/backend/installation/alpine_linux_en/)
- [Arch Linux](https://docs-develop.pleroma.social/backend/installation/arch_linux_en/)
- [CentOS 7](https://docs-develop.pleroma.social/backend/installation/centos7_en/)
- [Debian-based](https://docs-develop.pleroma.social/backend/installation/debian_based_en/)
- [Debian-based (jp)](https://docs-develop.pleroma.social/backend/installation/debian_based_jp/)
- [FreeBSD](https://docs-develop.pleroma.social/backend/installation/freebsd_en/)
- [Gentoo Linux](https://docs-develop.pleroma.social/backend/installation/gentoo_en/)
- [NetBSD](https://docs-develop.pleroma.social/backend/installation/netbsd_en/)
- [OpenBSD](https://docs-develop.pleroma.social/backend/installation/openbsd_en/)
- [OpenBSD (fi)](https://docs-develop.pleroma.social/backend/installation/openbsd_fi/)
- [CentOS 7](https://docs-develop.pleroma.social/backend/installation/centos7_en/)
### OS/Distro packages
Currently Pleroma is not packaged by any OS/Distros, but if you want to package it for one, we can guide you through the process on our [community channels](#community-channels). If you want to change default options in your Pleroma package, please **discuss it with us first**.

2
SECURITY.md
View file

@ -6,7 +6,7 @@ Currently, Pleroma offers bugfixes and security patches only for the latest mino
| Version | Support
|---------| --------
| 2.0 | Bugfixes and security patches
| 2.1 | Bugfixes and security patches
## Reporting a vulnerability

45
config/config.exs
View file

@ -130,6 +130,7 @@ config :pleroma, Pleroma.Web.Endpoint,
dispatch: [
{:_,
[
{"/api/fedsocket/v1", Pleroma.Web.FedSockets.IncomingHandler, []},
{"/api/v1/streaming", Pleroma.Web.MastodonAPI.WebsocketHandler, []},
{"/websocket", Phoenix.Endpoint.CowboyWebSocket,
{Phoenix.Transports.WebSocket,
@ -148,6 +149,16 @@ config :pleroma, Pleroma.Web.Endpoint,
"SameSite=Lax"
]
config :pleroma, :fed_sockets,
enabled: false,
connection_duration: :timer.hours(8),
rejection_duration: :timer.minutes(15),
fed_socket_fetches: [
default: 12_000,
interval: 3_000,
lazy: false
]
# Configures Elixir's Logger
config :logger, :console,
level: :debug,
@ -216,7 +227,6 @@ config :pleroma, :instance,
allow_relay: true,
public: true,
quarantined_instances: [],
managed_config: true,
static_dir: "instance/static/",
allowed_post_formats: [
"text/plain",
@ -424,6 +434,8 @@ config :pleroma, :media_proxy,
proxy_opts: [
redirect_on_failure: false,
max_body_length: 25 * 1_048_576,
# Note: max_read_duration defaults to Pleroma.ReverseProxy.max_read_duration_default/1
max_read_duration: 30_000,
http: [
follow_redirect: true,
pool: :media
@ -438,6 +450,14 @@ config :pleroma, Pleroma.Web.MediaProxy.Invalidation.Http,
config :pleroma, Pleroma.Web.MediaProxy.Invalidation.Script, script_path: nil
# Note: media preview proxy depends on media proxy to be enabled
config :pleroma, :media_preview_proxy,
enabled: false,
thumbnail_max_width: 600,
thumbnail_max_height: 600,
image_quality: 85,
min_content_length: 100 * 1024
config :pleroma, :chat, enabled: true
config :phoenix, :format_encoders, json: Jason
@ -530,8 +550,10 @@ config :pleroma, Oban,
log: false,
queues: [
activity_expiration: 10,
token_expiration: 5,
federator_incoming: 50,
federator_outgoing: 50,
ingestion_queue: 50,
web_push: 50,
mailer: 10,
transmogrifier: 20,
@ -543,9 +565,6 @@ config :pleroma, Oban,
],
plugins: [Oban.Plugins.Pruner],
crontab: [
{"0 0 * * *", Pleroma.Workers.Cron.ClearOauthTokenWorker},
{"0 * * * *", Pleroma.Workers.Cron.StatsWorker},
{"* * * * *", Pleroma.Workers.Cron.PurgeExpiredActivitiesWorker},
{"0 0 * * 0", Pleroma.Workers.Cron.DigestEmailsWorker},
{"0 0 * * *", Pleroma.Workers.Cron.NewUsersDigestWorker}
]
@ -656,7 +675,7 @@ config :pleroma, :rate_limit,
account_confirmation_resend: {8_640_000, 5},
ap_routes: {60_000, 15}
config :pleroma, Pleroma.ActivityExpiration, enabled: true
config :pleroma, Pleroma.Workers.PurgeExpiredActivity, enabled: true, min_lifetime: 600
config :pleroma, Pleroma.Plugs.RemoteIp, enabled: true
@ -670,7 +689,7 @@ config :pleroma, :static_fe, enabled: false
# With no frontend configuration, the bundled files from the `static` directory will
# be used.
#
# config :pleroma, :frontends,
# config :pleroma, :frontends,
# primary: %{"name" => "pleroma-fe", "ref" => "develop"},
# admin: %{"name" => "admin-fe", "ref" => "stable"},
# available: %{...}
@ -734,28 +753,28 @@ config :pleroma, :connections_pool,
max_connections: 250,
max_idle_time: 30_000,
retry: 0,
await_up_timeout: 5_000
connect_timeout: 5_000
config :pleroma, :pools,
federation: [
size: 50,
max_waiting: 10,
timeout: 10_000
recv_timeout: 10_000
],
media: [
size: 50,
max_waiting: 10,
timeout: 10_000
max_waiting: 20,
recv_timeout: 15_000
],
upload: [
size: 25,
max_waiting: 5,
timeout: 15_000
recv_timeout: 15_000
],
default: [
size: 10,
max_waiting: 2,
timeout: 5_000
recv_timeout: 5_000
]
config :pleroma, :hackney_pools,
@ -790,6 +809,8 @@ config :tzdata, :http_client, Pleroma.HTTP.Tzdata
config :ex_aws, http_client: Pleroma.HTTP.ExAws
config :web_push_encryption, http_client: Pleroma.HTTP
config :pleroma, :instances_favicons, enabled: false
config :floki, :html_parser, Floki.HTMLParser.FastHtml

87
config/description.exs
View file

@ -270,6 +270,19 @@ config :pleroma, :config_description, [
}
]
},
%{
group: :pleroma,
key: :fed_sockets,
type: :group,
description: "Websocket based federation",
children: [
%{
key: :enabled,
type: :boolean,
description: "Enable FedSockets"
}
]
},
%{
group: :pleroma,
key: Pleroma.Emails.Mailer,
@ -764,12 +777,6 @@ config :pleroma, :config_description, [
"*.quarantined.com"
]
},
%{
key: :managed_config,
type: :boolean,
description:
"Whenether the config for pleroma-fe is configured in this config or in static/config.json"
},
%{
key: :static_dir,
type: :string,
@ -1880,6 +1887,7 @@ config :pleroma, :config_description, [
suggestions: [
redirect_on_failure: false,
max_body_length: 25 * 1_048_576,
max_read_duration: 30_000,
http: [
follow_redirect: true,
pool: :media
@ -1900,6 +1908,11 @@ config :pleroma, :config_description, [
"Limits the content length to be approximately the " <>
"specified length. It is validated with the `content-length` header and also verified when proxying."
},
%{
key: :max_read_duration,
type: :integer,
description: "Timeout (in milliseconds) of GET request to remote URI."
},
%{
key: :http,
label: "HTTP",
@ -1946,6 +1959,43 @@ config :pleroma, :config_description, [
}
]
},
%{
group: :pleroma,
key: :media_preview_proxy,
type: :group,
description: "Media preview proxy",
children: [
%{
key: :enabled,
type: :boolean,
description:
"Enables proxying of remote media preview to the instance's proxy. Requires enabled media proxy."
},
%{
key: :thumbnail_max_width,
type: :integer,
description:
"Max width of preview thumbnail for images (video preview always has original dimensions)."
},
%{
key: :thumbnail_max_height,
type: :integer,
description:
"Max height of preview thumbnail for images (video preview always has original dimensions)."
},
%{
key: :image_quality,
type: :integer,
description: "Quality of the output. Ranges from 0 (min quality) to 100 (max quality)."
},
%{
key: :min_content_length,
type: :integer,
description:
"Min content length to perform preview, in bytes. If greater than 0, media smaller in size will be served as is, without thumbnailing."
}
]
},
%{
group: :pleroma,
key: Pleroma.Web.MediaProxy.Invalidation.Http,
@ -2290,9 +2340,6 @@ config :pleroma, :config_description, [
type: {:list, :tuple},
description: "Settings for cron background jobs",
suggestions: [
{"0 0 * * *", Pleroma.Workers.Cron.ClearOauthTokenWorker},
{"0 * * * *", Pleroma.Workers.Cron.StatsWorker},
{"* * * * *", Pleroma.Workers.Cron.PurgeExpiredActivitiesWorker},
{"0 0 * * 0", Pleroma.Workers.Cron.DigestEmailsWorker},
{"0 0 * * *", Pleroma.Workers.Cron.NewUsersDigestWorker}
]
@ -2398,7 +2445,7 @@ config :pleroma, :config_description, [
%{
group: :pleroma,
key: Pleroma.Formatter,
label: "Auto Linker",
label: "Linkify",
type: :group,
description:
"Configuration for Pleroma's link formatter which parses mentions, hashtags, and URLs.",
@ -2475,14 +2522,20 @@ config :pleroma, :config_description, [
},
%{
group: :pleroma,
key: Pleroma.ActivityExpiration,
key: Pleroma.Workers.PurgeExpiredActivity,
type: :group,
description: "Expired activity settings",
description: "Expired activities settings",
children: [
%{
key: :enabled,
type: :boolean,
description: "Whether expired activities will be sent to the job queue to be deleted"
description: "Enables expired activities addition & deletion"
},
%{
key: :min_lifetime,
type: :integer,
description: "Minimum lifetime for ephemeral activity (in seconds)",
suggestions: [600]
}
]
},
@ -3378,7 +3431,7 @@ config :pleroma, :config_description, [
suggestions: [250]
},
%{
key: :await_up_timeout,
key: :connect_timeout,
type: :integer,
description: "Timeout while `gun` will wait until connection is up. Default: 5000ms.",
suggestions: [5000]
@ -3416,6 +3469,12 @@ config :pleroma, :config_description, [
description:
"Maximum number of requests waiting for other requests to finish. After this number is reached, the pool will start returning errrors when a new request is made",
suggestions: [10]
},
%{
key: :recv_timeout,
type: :integer,
description: "Timeout for the pool while gun will wait for response",
suggestions: [10_000]
}
]
}

7
config/test.exs
View file

@ -19,6 +19,11 @@ config :logger, :console,
level: :warn,
format: "\n[$level] $message\n"
config :pleroma, :fed_sockets,
enabled: false,
connection_duration: 5,
rejection_duration: 5
config :pleroma, :auth, oauth_consumer_strategies: []
config :pleroma, Pleroma.Upload,
@ -114,8 +119,6 @@ config :pleroma, Pleroma.Plugs.RemoteIp, enabled: false
config :pleroma, Pleroma.Web.ApiSpec.CastAndValidate, strict: true
config :pleroma, :instances_favicons, enabled: false
config :pleroma, Pleroma.Uploaders.S3,
bucket: nil,
streaming_enabled: true,

163
docs/API/admin_api.md
View file

@ -1334,3 +1334,166 @@ Loads json generated from `config/descriptions.exs`.
{ }
```
## GET /api/pleroma/admin/users/:nickname/chats
### List a user's chats
- Params: None
- Response:
```json
[
{
"sender": {
"id": "someflakeid",
"username": "somenick",
...
},
"receiver": {
"id": "someflakeid",
"username": "somenick",
...
},
"id" : "1",
"unread" : 2,
"last_message" : {...}, // The last message in that chat
"updated_at": "2020-04-21T15:11:46.000Z"
}
]
```
## GET /api/pleroma/admin/chats/:chat_id
### View a single chat
- Params: None
- Response:
```json
{
"sender": {
"id": "someflakeid",
"username": "somenick",
...
},
"receiver": {
"id": "someflakeid",
"username": "somenick",
...
},
"id" : "1",
"unread" : 2,
"last_message" : {...}, // The last message in that chat
"updated_at": "2020-04-21T15:11:46.000Z"
}
```
## GET /api/pleroma/admin/chats/:chat_id/messages
### List the messages in a chat
- Params: `max_id`, `min_id`
- Response:
```json
[
{
"account_id": "someflakeid",
"chat_id": "1",
"content": "Check this out :firefox:",
"created_at": "2020-04-21T15:11:46.000Z",
"emojis": [
{
"shortcode": "firefox",
"static_url": "https://dontbulling.me/emoji/Firefox.gif",
"url": "https://dontbulling.me/emoji/Firefox.gif",
"visible_in_picker": false
}
],
"id": "13",
"unread": true
},
{
"account_id": "someflakeid",
"chat_id": "1",
"content": "Whats' up?",
"created_at": "2020-04-21T15:06:45.000Z",
"emojis": [],
"id": "12",
"unread": false
}
]
```
## DELETE /api/pleroma/admin/chats/:chat_id/messages/:message_id
### Delete a single message
- Params: None
- Response:
```json
{
"account_id": "someflakeid",
"chat_id": "1",
"content": "Check this out :firefox:",
"created_at": "2020-04-21T15:11:46.000Z",
"emojis": [
{
"shortcode": "firefox",
"static_url": "https://dontbulling.me/emoji/Firefox.gif",
"url": "https://dontbulling.me/emoji/Firefox.gif",
"visible_in_picker": false
}
],
"id": "13",
"unread": false
}
```
## `GET /api/pleroma/admin/instance_document/:document_name`
### Get an instance document
- Authentication: required
- Response:
Returns the content of the document
```html
<h1>Instance panel</h1>
```
## `PATCH /api/pleroma/admin/instance_document/:document_name`
- Params:
- `file` (the file to be uploaded, using multipart form data.)
### Update an instance document
- Authentication: required
- Response:
``` json
{
"url": "https://example.com/instance/panel.html"
}
```
## `DELETE /api/pleroma/admin/instance_document/:document_name`
### Delete an instance document
- Response:
``` json
{
"url": "https://example.com/instance/panel.html"
}
```

16
docs/API/pleroma_api.md
View file

@ -44,6 +44,22 @@ Request parameters can be passed via [query strings](https://en.wikipedia.org/wi
* Response: HTTP 200 on success, 500 on error
* Note: Users that can't be followed are silently skipped.
## `/api/pleroma/blocks_import`
### Imports your blocks.
* Method: `POST`
* Authentication: required
* Params:
* `list`: STRING or FILE containing a whitespace-separated list of accounts to block
* Response: HTTP 200 on success, 500 on error
## `/api/pleroma/mutes_import`
### Imports your mutes.
* Method: `POST`
* Authentication: required
* Params:
* `list`: STRING or FILE containing a whitespace-separated list of accounts to mute
* Response: HTTP 200 on success, 500 on error
## `/api/pleroma/captcha`
### Get a new captcha
* Method: `GET`

7
docs/administration/updating.md
View file

@ -18,9 +18,10 @@ su pleroma -s $SHELL -lc "./bin/pleroma_ctl migrate"
1. Go to the working directory of Pleroma (default is `/opt/pleroma`)
2. Run `git pull`. This pulls the latest changes from upstream.
3. Run `mix deps.get`. This pulls in any new dependencies.
3. Run `mix deps.get` [^1]. This pulls in any new dependencies.
4. Stop the Pleroma service.
5. Run `mix ecto.migrate`[^1]. This task performs database migrations, if there were any.
5. Run `mix ecto.migrate` [^1] [^2]. This task performs database migrations, if there were any.
6. Start the Pleroma service.
[^1]: Prefix with `MIX_ENV=prod` to run it using the production config file.
[^1]: Depending on which install guide you followed (for example on Debian/Ubuntu), you want to run `mix` tasks as `pleroma` user by adding `sudo -Hu pleroma` before the command.
[^2]: Prefix with `MIX_ENV=prod` to run it using the production config file.

43
docs/configuration/cheatsheet.md
View file

@ -18,7 +18,7 @@ To add configuration to your config file, you can copy it from the base config.
* `notify_email`: Email used for notifications.
* `description`: The instance’s description, can be seen in nodeinfo and ``/api/v1/instance``.
* `limit`: Posts character limit (CW/Subject included in the counter).
* `discription_limit`: The character limit for image descriptions.
* `description_limit`: The character limit for image descriptions.
* `chat_limit`: Character limit of the instance chat messages.
* `remote_limit`: Hard character limit beyond which remote posts will be dropped.
* `upload_limit`: File size limit of uploads (except for avatar, background, banner).
@ -40,7 +40,6 @@ To add configuration to your config file, you can copy it from the base config.
* `allow_relay`: Enable Pleroma’s Relay, which makes it possible to follow a whole instance.
* `public`: Makes the client API in authenticated mode-only except for user-profiles. Useful for disabling the Local Timeline and The Whole Known Network. Note that there is a dependent setting restricting or allowing unauthenticated access to specific resources, see `restrict_unauthenticated` for more details.
* `quarantined_instances`: List of ActivityPub instances where private (DMs, followers-only) activities will not be send.
* `managed_config`: Whenether the config for pleroma-fe is configured in [:frontend_configurations](#frontend_configurations) or in ``static/config.json``.
* `allowed_post_formats`: MIME-type list of formats allowed to be posted (transformed into HTML).
* `extended_nickname_format`: Set to `true` to use extended local nicknames format (allows underscores/dashes). This will break federation with
older software for theses nicknames.
@ -115,6 +114,7 @@ To add configuration to your config file, you can copy it from the base config.
* `Pleroma.Web.ActivityPub.MRF.VocabularyPolicy`: Restricts activities to a configured set of vocabulary. (See [`:mrf_vocabulary`](#mrf_vocabulary)).
* `Pleroma.Web.ActivityPub.MRF.ObjectAgePolicy`: Rejects or delists posts based on their age when received. (See [`:mrf_object_age`](#mrf_object_age)).
* `Pleroma.Web.ActivityPub.MRF.ActivityExpirationPolicy`: Sets a default expiration on all posts made by users of the local instance. Requires `Pleroma.ActivityExpiration` to be enabled for processing the scheduled delections.
* `Pleroma.Web.ActivityPub.MRF.ForceBotUnlistedPolicy`: Makes all bot posts to disappear from public timelines.
* `transparency`: Make the content of your Message Rewrite Facility settings public (via nodeinfo).
* `transparency_exclusions`: Exclude specific instance names from MRF transparency. The use of the exclusions feature will be disclosed in nodeinfo as a boolean value.
@ -225,6 +225,16 @@ Enables the worker which processes posts scheduled for deletion. Pinned posts ar
* `enabled`: whether expired activities will be sent to the job queue to be deleted
## FedSockets
FedSockets is an experimental feature allowing for Pleroma backends to federate using a persistant websocket connection as opposed to making each federation a seperate http connection. This feature is currently off by default. It is configurable throught he following options.
### :fedsockets
* `enabled`: Enables FedSockets for this instance. `false` by default.
* `connection_duration`: Time an idle websocket is kept open.
* `rejection_duration`: Failures to connect via FedSockets will not be retried for this period of time.
* `fed_socket_fetches` and `fed_socket_rejections`: Settings passed to `cachex` for the fetch registry, and rejection stacks. See `Pleroma.Web.FedSockets` for more details.
## Frontends
### :frontend_configurations
@ -314,6 +324,14 @@ This section describe PWA manifest instance-specific values. Currently this opti
* `enabled`: Enables purge cache
* `provider`: Which one of the [purge cache strategy](#purge-cache-strategy) to use.
## :media_preview_proxy
* `enabled`: Enables proxying of remote media preview to the instance’s proxy. Requires enabled media proxy (`media_proxy/enabled`).
* `thumbnail_max_width`: Max width of preview thumbnail for images (video preview always has original dimensions).
* `thumbnail_max_height`: Max height of preview thumbnail for images (video preview always has original dimensions).
* `image_quality`: Quality of the output. Ranges from 0 (min quality) to 100 (max quality).
* `min_content_length`: Min content length to perform preview, in bytes. If greater than 0, media smaller in size will be served as is, without thumbnailing.
### Purge cache strategy
#### Pleroma.Web.MediaProxy.Invalidation.Script
@ -496,7 +514,7 @@ Settings for HTTP connection pool.
* `:connection_acquisition_wait` - Timeout to acquire a connection from pool.The total max time is this value multiplied by the number of retries.
* `connection_acquisition_retries` - Number of attempts to acquire the connection from the pool if it is overloaded. Each attempt is timed `:connection_acquisition_wait` apart.
* `:max_connections` - Maximum number of connections in the pool.
* `:await_up_timeout` - Timeout to connect to the host.
* `:connect_timeout` - Timeout to connect to the host.
* `:reclaim_multiplier` - Multiplied by `:max_connections` this will be the maximum number of idle connections that will be reclaimed in case the pool is overloaded.
### :pools
@ -515,7 +533,7 @@ There are four pools used:
For each pool, the options are:
* `:size` - limit to how much requests can be concurrently executed.
* `:timeout` - timeout while `gun` will wait for response
* `:recv_timeout` - timeout while `gun` will wait for response
* `:max_waiting` - limit to how much requests can be waiting for others to finish, after this is reached, subsequent requests will be dropped.
## Captcha
@ -690,9 +708,8 @@ Pleroma has the following queues:
Pleroma has these periodic job workers:
`Pleroma.Workers.Cron.ClearOauthTokenWorker` - a job worker to cleanup expired oauth tokens.
Example:
* `Pleroma.Workers.Cron.DigestEmailsWorker` - digest emails for users with new mentions and follows
* `Pleroma.Workers.Cron.NewUsersDigestWorker` - digest emails for admins with new registrations
```elixir
config :pleroma, Oban,
@ -704,7 +721,8 @@ config :pleroma, Oban,
federator_outgoing: 50
],
crontab: [
{"0 0 * * *", Pleroma.Workers.Cron.ClearOauthTokenWorker}
{"0 0 * * 0", Pleroma.Workers.Cron.DigestEmailsWorker},
{"0 0 * * *", Pleroma.Workers.Cron.NewUsersDigestWorker}
]
```
@ -971,7 +989,7 @@ Configure OAuth 2 provider capabilities:
* `token_expires_in` - The lifetime in seconds of the access token.
* `issue_new_refresh_token` - Keeps old refresh token or generate new refresh token when to obtain an access token.
* `clean_expired_tokens` - Enable a background job to clean expired oauth tokens. Defaults to `false`. Interval settings sets in configuration periodic jobs [`Oban.Cron`](#obancron)
* `clean_expired_tokens` - Enable a background job to clean expired oauth tokens. Defaults to `false`.
## Link parsing
@ -1090,3 +1108,10 @@ config :pleroma, :frontends,
```
This would serve the frontend from the the folder at `$instance_static/frontends/pleroma/stable`. You have to copy the frontend into this folder yourself. You can choose the name and ref any way you like, but they will be used by mix tasks to automate installation in the future, the name referring to the project and the ref referring to a commit.
## Ephemeral activities (Pleroma.Workers.PurgeExpiredActivity)
Settings to enable and configure expiration for ephemeral activities
* `:enabled` - enables ephemeral activities creation
* `:min_lifetime` - minimum lifetime for ephemeral activities (in seconds). Default: 10 minutes.

33
installation/pleroma.nginx
View file

@ -9,6 +9,12 @@
proxy_cache_path /tmp/pleroma-media-cache levels=1:2 keys_zone=pleroma_media_cache:10m max_size=10g
inactive=720m use_temp_path=off;
# this is explicitly IPv4 since Pleroma.Web.Endpoint binds on IPv4 only
# and `localhost.` resolves to [::0] on some systems: see issue #930
upstream phoenix {
server 127.0.0.1:4000 max_fails=5 fail_timeout=60s;
}
server {
server_name example.tld;
@ -63,19 +69,16 @@ server {
# the nginx default is 1m, not enough for large media uploads
client_max_body_size 16m;
ignore_invalid_headers off;
proxy_http_version 1.1;
proxy_set_header Upgrade $http_upgrade;
proxy_set_header Connection "upgrade";
proxy_set_header Host $http_host;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
location / {
proxy_http_version 1.1;
proxy_set_header Upgrade $http_upgrade;
proxy_set_header Connection "upgrade";
proxy_set_header Host $http_host;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
# this is explicitly IPv4 since Pleroma.Web.Endpoint binds on IPv4 only
# and `localhost.` resolves to [::0] on some systems: see issue #930
proxy_pass http://127.0.0.1:4000;
client_max_body_size 16m;
proxy_pass http://phoenix;
}
location ~ ^/(media|proxy) {
@ -83,12 +86,16 @@ server {
slice 1m;
proxy_cache_key $host$uri$is_args$args$slice_range;
proxy_set_header Range $slice_range;
proxy_http_version 1.1;
proxy_cache_valid 200 206 301 304 1h;
proxy_cache_lock on;
proxy_ignore_client_abort on;
proxy_buffering on;
chunked_transfer_encoding on;
proxy_pass http://127.0.0.1:4000;
proxy_pass http://phoenix;
}
location /api/fedsocket/v1 {
proxy_request_buffering off;
proxy_pass http://phoenix/api/fedsocket/v1;
}
}

1
lib/mix/pleroma.ex
View file

@ -18,6 +18,7 @@ defmodule Mix.Pleroma do
@doc "Common functions to be reused in mix tasks"
def start_pleroma do
Pleroma.Config.Holder.save_default()
Pleroma.Config.Oban.warn()
Application.put_env(:phoenix, :serve_endpoints, false, persistent: true)
if Pleroma.Config.get(:env) != :test do

11
lib/mix/tasks/pleroma/benchmark.ex
View file

@ -91,20 +91,17 @@ defmodule Mix.Tasks.Pleroma.Benchmark do
"Without conn and without pool" => fn ->
{:ok, %Tesla.Env{}} =
Pleroma.HTTP.get("https://httpbin.org/stream-bytes/1500", [],
adapter: [pool: :no_pool, receive_conn: false]
pool: :no_pool,
receive_conn: false
)
end,
"Without conn and with pool" => fn ->
{:ok, %Tesla.Env{}} =
Pleroma.HTTP.get("https://httpbin.org/stream-bytes/1500", [],
adapter: [receive_conn: false]
)
Pleroma.HTTP.get("https://httpbin.org/stream-bytes/1500", [], receive_conn: false)
end,
"With reused conn and without pool" => fn ->
{:ok, %Tesla.Env{}} =
Pleroma.HTTP.get("https://httpbin.org/stream-bytes/1500", [],
adapter: [pool: :no_pool]
)
Pleroma.HTTP.get("https://httpbin.org/stream-bytes/1500", [], pool: :no_pool)
end,
"With reused conn and with pool" => fn ->
{:ok, %Tesla.Env{}} = Pleroma.HTTP.get("https://httpbin.org/stream-bytes/1500")

10
lib/mix/tasks/pleroma/config.ex
View file

@ -32,7 +32,8 @@ defmodule Mix.Tasks.Pleroma.Config do
@spec migrate_to_db(Path.t() | nil) :: any()
def migrate_to_db(file_path \\ nil) do
if Pleroma.Config.get([:configurable_from_database]) do
with true <- Pleroma.Config.get([:configurable_from_database]),
:ok <- Pleroma.Config.DeprecationWarnings.warn() do
config_file =
if file_path do
file_path
@ -46,7 +47,8 @@ defmodule Mix.Tasks.Pleroma.Config do
do_migrate_to_db(config_file)
else
migration_error()
:error -> deprecation_error()
_ -> migration_error()
end
end
@ -120,6 +122,10 @@ defmodule Mix.Tasks.Pleroma.Config do
)
end
defp deprecation_error do
shell_error("Migration is not allowed until all deprecation warnings have been resolved.")
end
if Code.ensure_loaded?(Config.Reader) do
defp config_header, do: "import Config\r\n\r\n"
defp read_file(config_file), do: Config.Reader.read_imports!(config_file)

21
lib/mix/tasks/pleroma/database.ex
View file

@ -99,7 +99,7 @@ defmodule Mix.Tasks.Pleroma.Database do
where: fragment("(?)->>'likes' is not null", object.data),
select: %{id: object.id, likes: fragment("(?)->>'likes'", object.data)}
)
|> Pleroma.RepoStreamer.chunk_stream(100)
|> Pleroma.Repo.chunk_stream(100, :batches)
|> Stream.each(fn objects ->
ids =
objects
@ -133,8 +133,7 @@ defmodule Mix.Tasks.Pleroma.Database do
days = Pleroma.Config.get([:mrf_activity_expiration, :days], 365)
Pleroma.Activity
|> join(:left, [a], u in assoc(a, :expiration))
|> join(:inner, [a, _u], o in Object,
|> join(:inner, [a], o in Object,
on:
fragment(
"(?->>'id') = COALESCE((?)->'object'->> 'id', (?)->>'object')",
@ -144,14 +143,20 @@ defmodule Mix.Tasks.Pleroma.Database do
)
)
|> where(local: true)
|> where([a, u], is_nil(u))
|> where([a], fragment("(? ->> 'type'::text) = 'Create'", a.data))
|> where([_a, _u, o], fragment("?->>'type' = 'Note'", o.data))
|> Pleroma.RepoStreamer.chunk_stream(100)
|> where([_a, o], fragment("?->>'type' = 'Note'", o.data))
|> Pleroma.Repo.chunk_stream(100, :batches)
|> Stream.each(fn activities ->
Enum.each(activities, fn activity ->
expires_at = Timex.shift(activity.inserted_at, days: days)
Pleroma.ActivityExpiration.create(activity, expires_at, false)
expires_at =
activity.inserted_at
|> DateTime.from_naive!("Etc/UTC")
|> Timex.shift(days: days)
Pleroma.Workers.PurgeExpiredActivity.enqueue(%{
activity_id: activity.id,
expires_at: expires_at
})
end)
end)
|> Stream.run()

12
lib/mix/tasks/pleroma/emoji.ex
View file

@ -183,7 +183,7 @@ defmodule Mix.Tasks.Pleroma.Emoji do
IO.puts("Downloading the pack and generating SHA256")
binary_archive = Tesla.get!(client(), src).body
{:ok, %{body: binary_archive}} = Pleroma.HTTP.get(src)
archive_sha = :crypto.hash(:sha256, binary_archive) |> Base.encode16()
IO.puts("SHA256 is #{archive_sha}")
@ -252,7 +252,7 @@ defmodule Mix.Tasks.Pleroma.Emoji do
end
defp fetch("http" <> _ = from) do
with {:ok, %{body: body}} <- Tesla.get(client(), from) do
with {:ok, %{body: body}} <- Pleroma.HTTP.get(from) do
{:ok, body}
end
end
@ -271,13 +271,5 @@ defmodule Mix.Tasks.Pleroma.Emoji do
)
end
defp client do
middleware = [
{Tesla.Middleware.FollowRedirects, [max_redirects: 3]}
]
Tesla.client(middleware)
end
defp default_manifest, do: Pleroma.Config.get!([:emoji, :default_manifest])
end

7
lib/mix/tasks/pleroma/frontend.ex
View file

@ -69,7 +69,7 @@ defmodule Mix.Tasks.Pleroma.Frontend do
fe_label = "#{frontend} (#{ref})"
tmp_dir = Path.join(dest, "tmp")
tmp_dir = Path.join([instance_static_dir, "frontends", "tmp"])
with {_, :ok} <-
{:download_or_unzip, download_or_unzip(frontend_info, tmp_dir, options[:file])},
@ -124,9 +124,7 @@ defmodule Mix.Tasks.Pleroma.Frontend do
url = String.replace(frontend_info["build_url"], "${ref}", frontend_info["ref"])
with {:ok, %{status: 200, body: zip_body}} <-
Pleroma.HTTP.get(url, [],
adapter: [pool: :media, timeout: 120_000, recv_timeout: 120_000]
) do
Pleroma.HTTP.get(url, [], pool: :media, recv_timeout: 120_000) do
unzip(zip_body, dest)
else
e -> {:error, e}
@ -135,6 +133,7 @@ defmodule Mix.Tasks.Pleroma.Frontend do
defp install_frontend(frontend_info, source, dest) do
from = frontend_info["build_dir"] || "dist"
File.rm_rf!(dest)
File.mkdir_p!(dest)
File.cp_r!(Path.join([source, from]), dest)
:ok

4
lib/mix/tasks/pleroma/user.ex
View file

@ -179,7 +179,7 @@ defmodule Mix.Tasks.Pleroma.User do
start_pleroma()
Pleroma.User.Query.build(%{nickname: "@#{instance}"})
|> Pleroma.RepoStreamer.chunk_stream(500)
|> Pleroma.Repo.chunk_stream(500, :batches)
|> Stream.each(fn users ->
users
|> Enum.each(fn user ->
@ -370,7 +370,7 @@ defmodule Mix.Tasks.Pleroma.User do
start_pleroma()
Pleroma.User.Query.build(%{local: true})
|> Pleroma.RepoStreamer.chunk_stream(500)
|> Pleroma.Repo.chunk_stream(500, :batches)
|> Stream.each(fn users ->
users
|> Enum.each(fn user ->

7
lib/pleroma/activity.ex
View file

@ -7,7 +7,6 @@ defmodule Pleroma.Activity do
alias Pleroma.Activity
alias Pleroma.Activity.Queries
alias Pleroma.ActivityExpiration
alias Pleroma.Bookmark
alias Pleroma.Notification
alias Pleroma.Object
@ -60,8 +59,6 @@ defmodule Pleroma.Activity do
# typical case.
has_one(:object, Object, on_delete: :nothing, foreign_key: :id)
has_one(:expiration, ActivityExpiration, on_delete: :delete_all)
timestamps()
end
@ -304,14 +301,14 @@ defmodule Pleroma.Activity do
|> Repo.all()
end
def follow_requests_for_actor(%Pleroma.User{ap_id: ap_id}) do
def follow_requests_for_actor(%User{ap_id: ap_id}) do
ap_id
|> Queries.by_object_id()
|> Queries.by_type("Follow")
|> where([a], fragment("? ->> 'state' = 'pending'", a.data))
end
def following_requests_for_actor(%Pleroma.User{ap_id: ap_id}) do
def following_requests_for_actor(%User{ap_id: ap_id}) do
Queries.by_type("Follow")
|> where([a], fragment("?->>'state' = 'pending'", a.data))
|> where([a], a.actor == ^ap_id)

74
lib/pleroma/activity_expiration.ex
View file

@ -1,74 +0,0 @@
# Pleroma: A lightweight social networking server
# Copyright © 2017-2020 Pleroma Authors <https://pleroma.social/>
# SPDX-License-Identifier: AGPL-3.0-only
defmodule Pleroma.ActivityExpiration do
use Ecto.Schema
alias Pleroma.Activity
alias Pleroma.ActivityExpiration
alias Pleroma.Repo
import Ecto.Changeset
import Ecto.Query
@type t :: %__MODULE__{}
@min_activity_lifetime :timer.hours(1)
schema "activity_expirations" do
belongs_to(:activity, Activity, type: FlakeId.Ecto.CompatType)
field(:scheduled_at, :naive_datetime)
end
def changeset(%ActivityExpiration{} = expiration, attrs, validate_scheduled_at) do
expiration
|> cast(attrs, [:scheduled_at])
|> validate_required([:scheduled_at])
|> validate_scheduled_at(validate_scheduled_at)
end
def get_by_activity_id(activity_id) do
ActivityExpiration
|> where([exp], exp.activity_id == ^activity_id)
|> Repo.one()
end
def create(%Activity{} = activity, scheduled_at, validate_scheduled_at \\ true) do
%ActivityExpiration{activity_id: activity.id}
|> changeset(%{scheduled_at: scheduled_at}, validate_scheduled_at)
|> Repo.insert()
end
def due_expirations(offset \\ 0) do
naive_datetime =
NaiveDateTime.utc_now()
|> NaiveDateTime.add(offset, :millisecond)
ActivityExpiration
|> where([exp], exp.scheduled_at < ^naive_datetime)
|> limit(50)
|> preload(:activity)
|> Repo.all()
|> Enum.reject(fn %{activity: activity} ->
Activity.pinned_by_actor?(activity)
end)
end
def validate_scheduled_at(changeset, false), do: changeset
def validate_scheduled_at(changeset, true) do
validate_change(changeset, :scheduled_at, fn _, scheduled_at ->
if not expires_late_enough?(scheduled_at) do
[scheduled_at: "an ephemeral activity must live for at least one hour"]
else
[]
end
end)
end
def expires_late_enough?(scheduled_at) do
now = NaiveDateTime.utc_now()
diff = NaiveDateTime.diff(scheduled_at, now, :millisecond)
diff > @min_activity_lifetime
end
end

30
lib/pleroma/application.ex
View file

@ -22,13 +22,18 @@ defmodule Pleroma.Application do
def repository, do: @repository
def user_agent do
case Config.get([:http, :user_agent], :default) do
:default ->
info = "#{Pleroma.Web.base_url()} <#{Config.get([:instance, :email], "")}>"
named_version() <> "; " <> info
if Process.whereis(Pleroma.Web.Endpoint) do
case Config.get([:http, :user_agent], :default) do
:default ->
info = "#{Pleroma.Web.base_url()} <#{Config.get([:instance, :email], "")}>"
named_version() <> "; " <> info
custom ->
custom
custom ->
custom
end
else
# fallback, if endpoint is not started yet
"Pleroma Data Loader"
end
end
@ -39,9 +44,13 @@ defmodule Pleroma.Application do
# every time the application is restarted, so we disable module
# conflicts at runtime
Code.compiler_options(ignore_module_conflict: true)
# Disable warnings_as_errors at runtime, it breaks Phoenix live reload
# due to protocol consolidation warnings
Code.compiler_options(warnings_as_errors: false)
Pleroma.Telemetry.Logger.attach()
Config.Holder.save_default()
Pleroma.HTML.compile_scrubbers()
Pleroma.Config.Oban.warn()
Config.DeprecationWarnings.warn()
Pleroma.Plugs.HTTPSecurityPlug.warn_if_disabled()
Pleroma.ApplicationRequirements.verify!()
@ -90,7 +99,7 @@ defmodule Pleroma.Application do
{Oban, Config.get(Oban)}
] ++
task_children(@env) ++
streamer_child(@env) ++
dont_run_in_test(@env) ++
chat_child(@env, chat_enabled?()) ++
[
Pleroma.Web.Endpoint,
@ -179,16 +188,17 @@ defmodule Pleroma.Application do
defp chat_enabled?, do: Config.get([:chat, :enabled])
defp streamer_child(env) when env in [:test, :benchmark], do: []
defp dont_run_in_test(env) when env in [:test, :benchmark], do: []
defp streamer_child(_) do
defp dont_run_in_test(_) do
[
{Registry,
[
name: Pleroma.Web.Streamer.registry(),
keys: :duplicate,
partitions: System.schedulers_online()
]}
]},
Pleroma.Web.FedSockets.Supervisor
]
end

10
lib/pleroma/chat.ex
View file

@ -6,7 +6,9 @@ defmodule Pleroma.Chat do
use Ecto.Schema
import Ecto.Changeset
import Ecto.Query
alias Pleroma.Chat
alias Pleroma.Repo
alias Pleroma.User
@ -69,4 +71,12 @@ defmodule Pleroma.Chat do
conflict_target: [:user_id, :recipient]
)
end
@spec for_user_query(FlakeId.Ecto.CompatType.t()) :: Ecto.Query.t()
def for_user_query(user_id) do
from(c in Chat,
where: c.user_id == ^user_id,
order_by: [desc: c.updated_at]
)
end
end

101
lib/pleroma/config/deprecation_warnings.ex
View file

@ -8,7 +8,7 @@ defmodule Pleroma.Config.DeprecationWarnings do
require Logger
alias Pleroma.Config
@type config_namespace() :: [atom()]
@type config_namespace() :: atom() | [atom()]
@type config_map() :: {config_namespace(), config_namespace(), String.t()}
@mrf_config_map [
@ -26,6 +26,10 @@ defmodule Pleroma.Config.DeprecationWarnings do
!!!DEPRECATION WARNING!!!
You are using the old configuration mechanism for the hellthread filter. Please check config.md.
""")
:error
else
:ok
end
end
@ -47,15 +51,26 @@ defmodule Pleroma.Config.DeprecationWarnings do
config :pleroma, :mrf_user_allowlist, #{inspect(rewritten, pretty: true)}
""")
:error
else
:ok
end
end
def warn do
check_hellthread_threshold()
mrf_user_allowlist()
check_old_mrf_config()
check_media_proxy_whitelist_config()
check_welcome_message_config()
with :ok <- check_hellthread_threshold(),
:ok <- mrf_user_allowlist(),
:ok <- check_old_mrf_config(),
:ok <- check_media_proxy_whitelist_config(),
:ok <- check_welcome_message_config(),
:ok <- check_gun_pool_options(),
:ok <- check_activity_expiration_config() do
:ok
else
_ ->
:error
end
end
def check_welcome_message_config do
@ -72,6 +87,10 @@ defmodule Pleroma.Config.DeprecationWarnings do
\n* `config :pleroma, :instance, welcome_user_nickname` is now `config :pleroma, :welcome, :direct_message, :sender_nickname`
\n* `config :pleroma, :instance, welcome_message` is now `config :pleroma, :welcome, :direct_message, :message`
""")
:error
else
:ok
end
end
@ -99,8 +118,11 @@ defmodule Pleroma.Config.DeprecationWarnings do
end
end)
if warning != "" do
if warning == "" do
:ok
else
Logger.warn(warning_preface <> warning)
:error
end
end
@ -113,6 +135,71 @@ defmodule Pleroma.Config.DeprecationWarnings do
!!!DEPRECATION WARNING!!!
Your config is using old format (only domain) for MediaProxy whitelist option. Setting should work for now, but you are advised to change format to scheme with port to prevent possible issues later.
""")
:error
else
:ok
end
end
def check_gun_pool_options do
pool_config = Config.get(:connections_pool)
if timeout = pool_config[:await_up_timeout] do
Logger.warn("""
!!!DEPRECATION WARNING!!!
Your config is using old setting name `await_up_timeout` instead of `connect_timeout`. Setting should work for now, but you are advised to change format to scheme with port to prevent possible issues later.
""")
Config.put(:connections_pool, Keyword.put_new(pool_config, :connect_timeout, timeout))
end
pools_configs = Config.get(:pools)
warning_preface = """
!!!DEPRECATION WARNING!!!
Your config is using old setting name `timeout` instead of `recv_timeout` in pool settings. Setting should work for now, but you are advised to change format to scheme with port to prevent possible issues later.
"""
updated_config =
Enum.reduce(pools_configs, [], fn {pool_name, config}, acc ->
if timeout = config[:timeout] do
Keyword.put(acc, pool_name, Keyword.put_new(config, :recv_timeout, timeout))
else
acc
end
end)
if updated_config != [] do
pool_warnings =
updated_config
|> Keyword.keys()
|> Enum.map(fn pool_name ->
"\n* `:timeout` options in #{pool_name} pool is now `:recv_timeout`"
end)
Logger.warn(Enum.join([warning_preface | pool_warnings]))
Config.put(:pools, updated_config)
:error
else
:ok
end
end
@spec check_activity_expiration_config() :: :ok | nil
def check_activity_expiration_config do
warning_preface = """
!!!DEPRECATION WARNING!!!
Your config is using old namespace for activity expiration configuration. Setting should work for now, but you are advised to change to new namespace to prevent possible issues later:
"""
move_namespace_and_warn(
[
{Pleroma.ActivityExpiration, Pleroma.Workers.PurgeExpiredActivity,
"\n* `config :pleroma, Pleroma.ActivityExpiration` is now `config :pleroma, Pleroma.Workers.PurgeExpiredActivity`"}
],
warning_preface
)
end
end

34
lib/pleroma/config/oban.ex Normal file
View file

@ -0,0 +1,34 @@
defmodule Pleroma.Config.Oban do
require Logger
def warn do
oban_config = Pleroma.Config.get(Oban)
crontab =
[
Pleroma.Workers.Cron.StatsWorker,
Pleroma.Workers.Cron.PurgeExpiredActivitiesWorker,
Pleroma.Workers.Cron.ClearOauthTokenWorker
]
|> Enum.reduce(oban_config[:crontab], fn removed_worker, acc ->
with acc when is_list(acc) <- acc,
setting when is_tuple(setting) <-
Enum.find(acc, fn {_, worker} -> worker == removed_worker end) do
"""
!!!OBAN CONFIG WARNING!!!
You are using old workers in Oban crontab settings, which were removed.
Please, remove setting from crontab in your config file (prod.secret.exs): #{
inspect(setting)
}
"""
|> Logger.warn()
List.delete(acc, setting)
else
_ -> acc
end
end)
Pleroma.Config.put(Oban, Keyword.put(oban_config, :crontab, crontab))
end
end

34
lib/pleroma/ecto_type/activity_pub/object_validators/emoji.ex Normal file
View file

@ -0,0 +1,34 @@
# Pleroma: A lightweight social networking server
# Copyright © 2017-2020 Pleroma Authors <https://pleroma.social/>
# SPDX-License-Identifier: AGPL-3.0-only
defmodule Pleroma.EctoType.ActivityPub.ObjectValidators.Emoji do
use Ecto.Type
def type, do: :map
def cast(data) when is_map(data) do
has_invalid_emoji? =
Enum.find(data, fn
{name, uri} when is_binary(name) and is_binary(uri) ->
# based on ObjectValidators.Uri.cast()
case URI.parse(uri) do
%URI{host: nil} -> true
%URI{host: ""} -> true
%URI{scheme: scheme} when scheme in ["https", "http"] -> false
_ -> true
end
{_name, _uri} ->
true
end)
if has_invalid_emoji?, do: :error, else: {:ok, data}
end
def cast(_data), do: :error
def dump(data), do: {:ok, data}
def load(data), do: {:ok, data}
end

14
lib/pleroma/gun/conn.ex
View file

@ -13,7 +13,7 @@ defmodule Pleroma.Gun.Conn do
opts =
opts
|> Enum.into(%{})
|> Map.put_new(:await_up_timeout, pool_opts[:await_up_timeout] || 5_000)
|> Map.put_new(:connect_timeout, pool_opts[:connect_timeout] || 5_000)
|> Map.put_new(:supervise, false)
|> maybe_add_tls_opts(uri)
@ -50,10 +50,10 @@ defmodule Pleroma.Gun.Conn do
with open_opts <- Map.delete(opts, :tls_opts),
{:ok, conn} <- Gun.open(proxy_host, proxy_port, open_opts),
{:ok, _} <- Gun.await_up(conn, opts[:await_up_timeout]),
{:ok, protocol} <- Gun.await_up(conn, opts[:connect_timeout]),
stream <- Gun.connect(conn, connect_opts),
{:response, :fin, 200, _} <- Gun.await(conn, stream) do
{:ok, conn}
{:ok, conn, protocol}
else
error ->
Logger.warn(
@ -88,8 +88,8 @@ defmodule Pleroma.Gun.Conn do
|> Map.put(:socks_opts, socks_opts)
with {:ok, conn} <- Gun.open(proxy_host, proxy_port, opts),
{:ok, _} <- Gun.await_up(conn, opts[:await_up_timeout]) do
{:ok, conn}
{:ok, protocol} <- Gun.await_up(conn, opts[:connect_timeout]) do
{:ok, conn, protocol}
else
error ->
Logger.warn(
@ -106,8 +106,8 @@ defmodule Pleroma.Gun.Conn do
host = Pleroma.HTTP.AdapterHelper.parse_host(host)
with {:ok, conn} <- Gun.open(host, port, opts),
{:ok, _} <- Gun.await_up(conn, opts[:await_up_timeout]) do
{:ok, conn}
{:ok, protocol} <- Gun.await_up(conn, opts[:connect_timeout]) do
{:ok, conn, protocol}
else
error ->
Logger.warn(

47
lib/pleroma/gun/connection_pool/worker.ex
View file

@ -15,7 +15,7 @@ defmodule Pleroma.Gun.ConnectionPool.Worker do
@impl true
def handle_continue({:connect, [key, uri, opts, client_pid]}, _) do
with {:ok, conn_pid} <- Gun.Conn.open(uri, opts),
with {:ok, conn_pid, protocol} <- Gun.Conn.open(uri, opts),
Process.link(conn_pid) do
time = :erlang.monotonic_time(:millisecond)
@ -27,8 +27,12 @@ defmodule Pleroma.Gun.ConnectionPool.Worker do
send(client_pid, {:conn_pid, conn_pid})
{:noreply,
%{key: key, timer: nil, client_monitors: %{client_pid => Process.monitor(client_pid)}},
:hibernate}
%{
key: key,
timer: nil,
client_monitors: %{client_pid => Process.monitor(client_pid)},
protocol: protocol
}, :hibernate}
else
err ->
{:stop, {:shutdown, err}, nil}
@ -53,14 +57,20 @@ defmodule Pleroma.Gun.ConnectionPool.Worker do
end
@impl true
def handle_call(:add_client, {client_pid, _}, %{key: key} = state) do
def handle_call(:add_client, {client_pid, _}, %{key: key, protocol: protocol} = state) do
time = :erlang.monotonic_time(:millisecond)
{{conn_pid, _, _, _}, _} =
{{conn_pid, used_by, _, _}, _} =
Registry.update_value(@registry, key, fn {conn_pid, used_by, crf, last_reference} ->
{conn_pid, [client_pid | used_by], crf(time - last_reference, crf), time}
end)
:telemetry.execute(
[:pleroma, :connection_pool, :client, :add],
%{client_pid: client_pid, clients: used_by},
%{key: state.key, protocol: protocol}
)
state =
if state.timer != nil do
Process.cancel_timer(state[:timer])
@ -83,25 +93,18 @@ defmodule Pleroma.Gun.ConnectionPool.Worker do
end)
{ref, state} = pop_in(state.client_monitors[client_pid])
# DOWN message can receive right after `remove_client` call and cause worker to terminate
state =
if is_nil(ref) do
state
Process.demonitor(ref, [:flush])
timer =
if used_by == [] do
max_idle = Pleroma.Config.get([:connections_pool, :max_idle_time], 30_000)
Process.send_after(self(), :idle_close, max_idle)
else
Process.demonitor(ref)
timer =
if used_by == [] do
max_idle = Pleroma.Config.get([:connections_pool, :max_idle_time], 30_000)
Process.send_after(self(), :idle_close, max_idle)
else
nil
end
%{state | timer: timer}
nil
end
{:reply, :ok, state, :hibernate}
{:reply, :ok, %{state | timer: timer}, :hibernate}
end
@impl true
@ -131,7 +134,7 @@ defmodule Pleroma.Gun.ConnectionPool.Worker do
@impl true
def handle_info({:DOWN, _ref, :process, pid, reason}, state) do
:telemetry.execute(
[:pleroma, :connection_pool, :client_death],
[:pleroma, :connection_pool, :client, :dead],
%{client_pid: pid, reason: reason},
%{key: state.key}
)

150
lib/pleroma/helpers/media_helper.ex Normal file
View file

@ -0,0 +1,150 @@
# Pleroma: A lightweight social networking server
# Copyright © 2017-2020 Pleroma Authors <https://pleroma.social/>
# SPDX-License-Identifier: AGPL-3.0-only
defmodule Pleroma.Helpers.MediaHelper do
@moduledoc """
Handles common media-related operations.
"""
alias Pleroma.HTTP
def image_resize(url, options) do
with executable when is_binary(executable) <- System.find_executable("convert"),
{:ok, args} <- prepare_image_resize_args(options),
{:ok, env} <- HTTP.get(url, [], pool: :media),
{:ok, fifo_path} <- mkfifo() do
args = List.flatten([fifo_path, args])
run_fifo(fifo_path, env, executable, args)
else
nil -> {:error, {:convert, :command_not_found}}
{:error, _} = error -> error
end
end
defp prepare_image_resize_args(
%{max_width: max_width, max_height: max_height, format: "png"} = options
) do
quality = options[:quality] || 85
resize = Enum.join([max_width, "x", max_height, ">"])
args = [
"-resize",
resize,
"-quality",
to_string(quality),
"png:-"
]
{:ok, args}
end
defp prepare_image_resize_args(%{max_width: max_width, max_height: max_height} = options) do
quality = options[:quality] || 85
resize = Enum.join([max_width, "x", max_height, ">"])
args = [
"-interlace",
"Plane",
"-resize",
resize,
"-quality",
to_string(quality),
"jpg:-"
]
{:ok, args}
end
defp prepare_image_resize_args(_), do: {:error, :missing_options}
# Note: video thumbnail is intentionally not resized (always has original dimensions)
def video_framegrab(url) do
with executable when is_binary(executable) <- System.find_executable("ffmpeg"),
{:ok, env} <- HTTP.get(url, [], pool: :media),
{:ok, fifo_path} <- mkfifo(),
args = [
"-y",
"-i",
fifo_path,
"-vframes",
"1",
"-f",
"mjpeg",
"-loglevel",
"error",
"-"
] do
run_fifo(fifo_path, env, executable, args)
else
nil -> {:error, {:ffmpeg, :command_not_found}}
{:error, _} = error -> error
end
end
defp run_fifo(fifo_path, env, executable, args) do
pid =
Port.open({:spawn_executable, executable}, [
:use_stdio,
:stream,
:exit_status,
:binary,
args: args
])
fifo = Port.open(to_charlist(fifo_path), [:eof, :binary, :stream, :out])
fix = Pleroma.Helpers.QtFastStart.fix(env.body)
true = Port.command(fifo, fix)
:erlang.port_close(fifo)
loop_recv(pid)
after
File.rm(fifo_path)
end
defp mkfifo do
path = Path.join(System.tmp_dir!(), "pleroma-media-preview-pipe-#{Ecto.UUID.generate()}")
case System.cmd("mkfifo", [path]) do
{_, 0} ->
spawn(fifo_guard(path))
{:ok, path}
{_, err} ->
{:error, {:fifo_failed, err}}
end
end
defp fifo_guard(path) do
pid = self()
fn ->
ref = Process.monitor(pid)
receive do
{:DOWN, ^ref, :process, ^pid, _} ->
File.rm(path)
end
end
end
defp loop_recv(pid) do
loop_recv(pid, <<>>)
end
defp loop_recv(pid, acc) do
receive do
{^pid, {:data, data}} ->
loop_recv(pid, acc <> data)
{^pid, {:exit_status, 0}} ->
{:ok, acc}
{^pid, {:exit_status, status}} ->
{:error, status}
after
5000 ->
:erlang.port_close(pid)
{:error, :timeout}
end
end
end

131
lib/pleroma/helpers/qt_fast_start.ex Normal file
View file

@ -0,0 +1,131 @@
# Pleroma: A lightweight social networking server
# Copyright © 2017-2020 Pleroma Authors <https://pleroma.social/>
# SPDX-License-Identifier: AGPL-3.0-only
defmodule Pleroma.Helpers.QtFastStart do
@moduledoc """
(WIP) Converts a "slow start" (data before metadatas) mov/mp4 file to a "fast start" one (metadatas before data).
"""
# TODO: Cleanup and optimizations
# Inspirations: https://www.ffmpeg.org/doxygen/3.4/qt-faststart_8c_source.html
# https://github.com/danielgtaylor/qtfaststart/blob/master/qtfaststart/processor.py
# ISO/IEC 14496-12:2015, ISO/IEC 15444-12:2015
# Paracetamol
def fix(<<0x00, 0x00, 0x00, _, 0x66, 0x74, 0x79, 0x70, _::bits>> = binary) do
index = fix(binary, 0, nil, nil, [])
case index do
:abort -> binary
[{"ftyp", _, _, _, _}, {"mdat", _, _, _, _} | _] -> faststart(index)
[{"ftyp", _, _, _, _}, {"free", _, _, _, _}, {"mdat", _, _, _, _} | _] -> faststart(index)
_ -> binary
end
end
def fix(binary) do
binary
end
# MOOV have been seen before MDAT- abort
defp fix(<<_::bits>>, _, true, false, _) do
:abort
end
defp fix(
<<size::integer-big-size(32), fourcc::bits-size(32), rest::bits>>,
pos,
got_moov,
got_mdat,
acc
) do
full_size = (size - 8) * 8
<<data::bits-size(full_size), rest::bits>> = rest
acc = [
{fourcc, pos, pos + size, size,
<<size::integer-big-size(32), fourcc::bits-size(32), data::bits>>}
| acc
]
fix(rest, pos + size, got_moov || fourcc == "moov", got_mdat || fourcc == "mdat", acc)
end
defp fix(<<>>, _pos, _, _, acc) do
:lists.reverse(acc)
end
defp faststart(index) do
{{_ftyp, _, _, _, ftyp}, index} = List.keytake(index, "ftyp", 0)
# Skip re-writing the free fourcc as it's kind of useless.
# Why stream useless bytes when you can do without?
{free_size, index} =
case List.keytake(index, "free", 0) do
{{_, _, _, size, _}, index} -> {size, index}
_ -> {0, index}
end
{{_moov, _, _, moov_size, moov}, index} = List.keytake(index, "moov", 0)
offset = -free_size + moov_size
rest = for {_, _, _, _, data} <- index, do: data, into: []
<<moov_head::bits-size(64), moov_data::bits>> = moov
[ftyp, moov_head, fix_moov(moov_data, offset, []), rest]
end
defp fix_moov(
<<size::integer-big-size(32), fourcc::bits-size(32), rest::bits>>,
offset,
acc
) do
full_size = (size - 8) * 8
<<data::bits-size(full_size), rest::bits>> = rest
data =
cond do
fourcc in ["trak", "mdia", "minf", "stbl"] ->
# Theses contains sto or co64 part
[<<size::integer-big-size(32), fourcc::bits-size(32)>>, fix_moov(data, offset, [])]
fourcc in ["stco", "co64"] ->
# fix the damn thing
<<version::integer-big-size(32), count::integer-big-size(32), rest::bits>> = data
entry_size =
case fourcc do
"stco" -> 32
"co64" -> 64
end
[
<<size::integer-big-size(32), fourcc::bits-size(32), version::integer-big-size(32),
count::integer-big-size(32)>>,
rewrite_entries(entry_size, offset, rest, [])
]
true ->
[<<size::integer-big-size(32), fourcc::bits-size(32)>>, data]
end
acc = [acc | data]
fix_moov(rest, offset, acc)
end
defp fix_moov(<<>>, _, acc), do: acc
for size <- [32, 64] do
defp rewrite_entries(
unquote(size),
offset,
<<pos::integer-big-size(unquote(size)), rest::bits>>,
acc
) do
rewrite_entries(unquote(size), offset, rest, [
acc | <<pos + offset::integer-big-size(unquote(size))>>
])
end
end
defp rewrite_entries(_, _, <<>>, acc), do: acc
end

14
lib/pleroma/helpers/uri_helper.ex
View file

@ -3,18 +3,22 @@
# SPDX-License-Identifier: AGPL-3.0-only
defmodule Pleroma.Helpers.UriHelper do
def append_uri_params(uri, appended_params) do
def modify_uri_params(uri, overridden_params, deleted_params \\ []) do
uri = URI.parse(uri)
appended_params = for {k, v} <- appended_params, into: %{}, do: {to_string(k), v}
existing_params = URI.query_decoder(uri.query || "") |> Enum.into(%{})
updated_params_keys = Enum.uniq(Map.keys(existing_params) ++ Map.keys(appended_params))
existing_params = URI.query_decoder(uri.query || "") |> Map.new()
overridden_params = Map.new(overridden_params, fn {k, v} -> {to_string(k), v} end)
deleted_params = Enum.map(deleted_params, &to_string/1)
updated_params =
for k <- updated_params_keys, do: {k, appended_params[k] || existing_params[k]}
existing_params
|> Map.merge(overridden_params)
|> Map.drop(deleted_params)
uri
|> Map.put(:query, URI.encode_query(updated_params))
|> URI.to_string()
|> String.replace_suffix("?", "")
end
def maybe_add_base("/" <> uri, base), do: Path.join([base, uri])

6
lib/pleroma/http/adapter_helper.ex
View file

@ -6,7 +6,7 @@ defmodule Pleroma.HTTP.AdapterHelper do
@moduledoc """
Configure Tesla.Client with default and customized adapter options.
"""
@defaults [pool: :federation]
@defaults [pool: :federation, connect_timeout: 5_000, recv_timeout: 5_000]
@type proxy_type() :: :socks4 | :socks5
@type host() :: charlist() | :inet.ip_address()
@ -19,7 +19,6 @@ defmodule Pleroma.HTTP.AdapterHelper do
| {Connection.proxy_type(), Connection.host(), pos_integer()}
@callback options(keyword(), URI.t()) :: keyword()
@callback get_conn(URI.t(), keyword()) :: {:ok, term()} | {:error, term()}
@spec format_proxy(String.t() | tuple() | nil) :: proxy() | nil
def format_proxy(nil), do: nil
@ -47,9 +46,6 @@ defmodule Pleroma.HTTP.AdapterHelper do
|> adapter_helper().options(uri)
end
@spec get_conn(URI.t(), keyword()) :: {:ok, keyword()} | {:error, atom()}
def get_conn(uri, opts), do: adapter_helper().get_conn(uri, opts)
defp adapter, do: Application.get_env(:tesla, :adapter)
defp adapter_helper do

23
lib/pleroma/http/adapter_helper/gun.ex
View file

@ -6,18 +6,13 @@ defmodule Pleroma.HTTP.AdapterHelper.Gun do
@behaviour Pleroma.HTTP.AdapterHelper
alias Pleroma.Config
alias Pleroma.Gun.ConnectionPool
alias Pleroma.HTTP.AdapterHelper
require Logger
@defaults [
connect_timeout: 5_000,
domain_lookup_timeout: 5_000,
tls_handshake_timeout: 5_000,
retry: 1,
retry_timeout: 1000,
await_up_timeout: 5_000
retry_timeout: 1_000
]
@type pool() :: :federation | :upload | :media | :default
@ -46,23 +41,17 @@ defmodule Pleroma.HTTP.AdapterHelper.Gun do
end
defp put_timeout(opts) do
{recv_timeout, opts} = Keyword.pop(opts, :recv_timeout, pool_timeout(opts[:pool]))
# this is the timeout to receive a message from Gun
Keyword.put_new(opts, :timeout, pool_timeout(opts[:pool]))
# `:timeout` key is used in Tesla
Keyword.put(opts, :timeout, recv_timeout)
end
@spec pool_timeout(pool()) :: non_neg_integer()
def pool_timeout(pool) do
default = Config.get([:pools, :default, :timeout], 5_000)
default = Config.get([:pools, :default, :recv_timeout], 5_000)
Config.get([:pools, pool, :timeout], default)
end
@spec get_conn(URI.t(), keyword()) :: {:ok, keyword()} | {:error, atom()}
def get_conn(uri, opts) do
case ConnectionPool.get_conn(uri, opts) do
{:ok, conn_pid} -> {:ok, Keyword.merge(opts, conn: conn_pid, close_conn: false)}
err -> err
end
Config.get([:pools, pool, :recv_timeout], default)
end
@prefix Pleroma.Gun.ConnectionPool

15
lib/pleroma/http/adapter_helper/hackney.ex
View file

@ -2,11 +2,8 @@ defmodule Pleroma.HTTP.AdapterHelper.Hackney do
@behaviour Pleroma.HTTP.AdapterHelper
@defaults [
connect_timeout: 10_000,
recv_timeout: 20_000,
follow_redirect: true,
force_redirect: true,
pool: :federation
force_redirect: true
]
@spec options(keyword(), URI.t()) :: keyword()
@ -19,6 +16,7 @@ defmodule Pleroma.HTTP.AdapterHelper.Hackney do
|> Keyword.merge(config_opts)
|> Keyword.merge(connection_opts)
|> add_scheme_opts(uri)
|> maybe_add_with_body()
|> Pleroma.HTTP.AdapterHelper.maybe_add_proxy(proxy)
end
@ -28,6 +26,11 @@ defmodule Pleroma.HTTP.AdapterHelper.Hackney do
defp add_scheme_opts(opts, _), do: opts
@spec get_conn(URI.t(), keyword()) :: {:ok, keyword()}
def get_conn(_uri, opts), do: {:ok, opts}
defp maybe_add_with_body(opts) do
if opts[:max_body] do
Keyword.put(opts, :with_body, true)
else
opts
end
end
end

2
lib/pleroma/http/ex_aws.ex
View file

@ -11,7 +11,7 @@ defmodule Pleroma.HTTP.ExAws do
@impl true
def request(method, url, body \\ "", headers \\ [], http_opts \\ []) do
http_opts = Keyword.put_new(http_opts, :adapter, pool: :upload)
http_opts = Keyword.put_new(http_opts, :pool, :upload)
case HTTP.request(method, url, body, headers, http_opts) do
{:ok, env} ->

35
lib/pleroma/http/http.ex
View file

@ -60,30 +60,23 @@ defmodule Pleroma.HTTP do
{:ok, Env.t()} | {:error, any()}
def request(method, url, body, headers, options) when is_binary(url) do
uri = URI.parse(url)
adapter_opts = AdapterHelper.options(uri, options[:adapter] || [])
adapter_opts = AdapterHelper.options(uri, options || [])
case AdapterHelper.get_conn(uri, adapter_opts) do
{:ok, adapter_opts} ->
options = put_in(options[:adapter], adapter_opts)
params = options[:params] || []
request = build_request(method, headers, options, url, body, params)
options = put_in(options[:adapter], adapter_opts)
params = options[:params] || []
request = build_request(method, headers, options, url, body, params)
adapter = Application.get_env(:tesla, :adapter)
adapter = Application.get_env(:tesla, :adapter)
client = Tesla.client(adapter_middlewares(adapter), adapter)
client = Tesla.client(adapter_middlewares(adapter), adapter)
maybe_limit(
fn ->
request(client, request)
end,
adapter,
adapter_opts
)
# Connection release is handled in a custom FollowRedirects middleware
err ->
err
end
maybe_limit(
fn ->
request(client, request)
end,
adapter,
adapter_opts
)
end
@spec request(Client.t(), keyword()) :: {:ok, Env.t()} | {:error, any()}
@ -110,7 +103,7 @@ defmodule Pleroma.HTTP do
end
defp adapter_middlewares(Tesla.Adapter.Gun) do
[Pleroma.HTTP.Middleware.FollowRedirects]
[Tesla.Middleware.FollowRedirects, Pleroma.Tesla.Middleware.ConnectionPool]
end
defp adapter_middlewares(_), do: []

4
lib/pleroma/http/tzdata.ex
View file

@ -11,7 +11,7 @@ defmodule Pleroma.HTTP.Tzdata do
@impl true
def get(url, headers, options) do
options = Keyword.put_new(options, :adapter, pool: :default)
options = Keyword.put_new(options, :pool, :default)
with {:ok, %Tesla.Env{} = env} <- HTTP.get(url, headers, options) do
{:ok, {env.status, env.headers, env.body}}
@ -20,7 +20,7 @@ defmodule Pleroma.HTTP.Tzdata do
@impl true
def head(url, headers, options) do
options = Keyword.put_new(options, :adapter, pool: :default)
options = Keyword.put_new(options, :pool, :default)
with {:ok, %Tesla.Env{} = env} <- HTTP.head(url, headers, options) do
{:ok, {env.status, env.headers}}

16
lib/pleroma/instances/instance.ex
View file

@ -156,16 +156,12 @@ defmodule Pleroma.Instances.Instance do
defp scrape_favicon(%URI{} = instance_uri) do
try do
with {:ok, %Tesla.Env{body: html}} <-
Pleroma.HTTP.get(to_string(instance_uri), [{"accept", "text/html"}],
adapter: [pool: :media]
),
favicon_rel <-
html
|> Floki.parse_document!()
|> Floki.attribute("link[rel=icon]", "href")
|> List.first(),
favicon <- URI.merge(instance_uri, favicon_rel) |> to_string(),
true <- is_binary(favicon) do
Pleroma.HTTP.get(to_string(instance_uri), [{"accept", "text/html"}], pool: :media),
{_, [favicon_rel | _]} when is_binary(favicon_rel) <-
{:parse,
html |> Floki.parse_document!() |> Floki.attribute("link[rel=icon]", "href")},
{_, favicon} when is_binary(favicon) <-
{:merge, URI.merge(instance_uri, favicon_rel) |> to_string()} do
favicon
else
_ -> nil

71
lib/pleroma/mfa/token.ex
View file

@ -10,10 +10,11 @@ defmodule Pleroma.MFA.Token do
alias Pleroma.Repo
alias Pleroma.User
alias Pleroma.Web.OAuth.Authorization
alias Pleroma.Web.OAuth.Token, as: OAuthToken
@expires 300
@type t() :: %__MODULE__{}
schema "mfa_tokens" do
field(:token, :string)
field(:valid_until, :naive_datetime_usec)
@ -24,6 +25,7 @@ defmodule Pleroma.MFA.Token do
timestamps()
end
@spec get_by_token(String.t()) :: {:ok, t()} | {:error, :not_found}
def get_by_token(token) do
from(
t in __MODULE__,
@ -33,33 +35,40 @@ defmodule Pleroma.MFA.Token do
|> Repo.find_resource()
end
def validate(token) do
with {:fetch_token, {:ok, token}} <- {:fetch_token, get_by_token(token)},
{:expired, false} <- {:expired, is_expired?(token)} do
@spec validate(String.t()) :: {:ok, t()} | {:error, :not_found} | {:error, :expired_token}
def validate(token_str) do
with {:ok, token} <- get_by_token(token_str),
false <- expired?(token) do
{:ok, token}
else
{:expired, _} -> {:error, :expired_token}
{:fetch_token, _} -> {:error, :not_found}
error -> {:error, error}
end
end
def create_token(%User{} = user) do
%__MODULE__{}
|> change
|> assign_user(user)
|> put_token
|> put_valid_until
|> Repo.insert()
defp expired?(%__MODULE__{valid_until: valid_until}) do
with true <- NaiveDateTime.diff(NaiveDateTime.utc_now(), valid_until) > 0 do
{:error, :expired_token}
end
end
def create_token(user, authorization) do
@spec create(User.t(), Authorization.t() | nil) :: {:ok, t()} | {:error, Ecto.Changeset.t()}
def create(user, authorization \\ nil) do
with {:ok, token} <- do_create(user, authorization) do
Pleroma.Workers.PurgeExpiredToken.enqueue(%{
token_id: token.id,
valid_until: DateTime.from_naive!(token.valid_until, "Etc/UTC"),
mod: __MODULE__
})
{:ok, token}
end
end
defp do_create(user, authorization) do
%__MODULE__{}
|> change
|> change()
|> assign_user(user)
|> assign_authorization(authorization)
|> put_token
|> put_valid_until
|> maybe_assign_authorization(authorization)
|> put_token()
|> put_valid_until()
|> Repo.insert()
end
@ -69,15 +78,19 @@ defmodule Pleroma.MFA.Token do
|> validate_required([:user])
end
defp assign_authorization(changeset, authorization) do
defp maybe_assign_authorization(changeset, %Authorization{} = authorization) do
changeset
|> put_assoc(:authorization, authorization)
|> validate_required([:authorization])
end
defp maybe_assign_authorization(changeset, _), do: changeset
defp put_token(changeset) do
token = Pleroma.Web.OAuth.Token.Utils.generate_token()
changeset
|> change(%{token: OAuthToken.Utils.generate_token()})
|> change(%{token: token})
|> validate_required([:token])
|> unique_constraint(:token)
end
@ -89,18 +102,4 @@ defmodule Pleroma.MFA.Token do
|> change(%{valid_until: expires_in})
|> validate_required([:valid_until])
end
def is_expired?(%__MODULE__{valid_until: valid_until}) do
NaiveDateTime.diff(NaiveDateTime.utc_now(), valid_until) > 0
end
def is_expired?(_), do: false
def delete_expired_tokens do
from(
q in __MODULE__,
where: fragment("?", q.valid_until) < ^Timex.now()
)
|> Repo.delete_all()
end
end

15
lib/pleroma/migration_helper/notification_backfill.ex
View file

@ -19,13 +19,13 @@ defmodule Pleroma.MigrationHelper.NotificationBackfill do
query
|> Repo.chunk_stream(100)
|> Enum.each(fn notification ->
type =
notification.activity
|> type_from_activity()
if notification.activity do
type = type_from_activity(notification.activity)
notification
|> Ecto.Changeset.change(%{type: type})
|> Repo.update()
notification
|> Ecto.Changeset.change(%{type: type})
|> Repo.update()
end
end)
end
@ -72,8 +72,7 @@ defmodule Pleroma.MigrationHelper.NotificationBackfill do
"pleroma:emoji_reaction"
"Create" ->
activity
|> type_from_activity_object()
type_from_activity_object(activity)
t ->
raise "No notification type for activity type #{t}"

24
lib/pleroma/moderation_log.ex
View file

@ -320,6 +320,19 @@ defmodule Pleroma.ModerationLog do
|> insert_log_entry_with_message()
end
@spec insert_log(%{actor: User, action: String.t(), subject_id: String.t()}) ::
{:ok, ModerationLog} | {:error, any}
def insert_log(%{actor: %User{} = actor, action: "chat_message_delete", subject_id: subject_id}) do
%ModerationLog{
data: %{
"actor" => %{"nickname" => actor.nickname},
"action" => "chat_message_delete",
"subject_id" => subject_id
}
}
|> insert_log_entry_with_message()
end
@spec insert_log_entry_with_message(ModerationLog) :: {:ok, ModerationLog} | {:error, any}
defp insert_log_entry_with_message(entry) do
entry.data["message"]
@ -627,6 +640,17 @@ defmodule Pleroma.ModerationLog do
"@#{actor_nickname} updated users: #{users_to_nicknames_string(subjects)}"
end
@spec get_log_entry_message(ModerationLog) :: String.t()
def get_log_entry_message(%ModerationLog{
data: %{
"actor" => %{"nickname" => actor_nickname},
"action" => "chat_message_delete",
"subject_id" => subject_id
}
}) do
"@#{actor_nickname} deleted chat message ##{subject_id}"
end
defp nicknames_to_string(nicknames) do
nicknames
|> Enum.map(&"@#{&1}")

12
lib/pleroma/notification.ex
View file

@ -648,4 +648,16 @@ defmodule Pleroma.Notification do
)
|> Repo.one()
end
@spec mark_context_as_read(User.t(), String.t()) :: {integer(), nil | [term()]}
def mark_context_as_read(%User{id: id}, context) do
from(
n in Notification,
join: a in assoc(n, :activity),
where: n.user_id == ^id,
where: n.seen == false,
where: fragment("?->>'context'", a.data) == ^context
)
|> Repo.update_all(set: [seen: true])
end
end

7
lib/pleroma/object/containment.ex
View file

@ -44,13 +44,6 @@ defmodule Pleroma.Object.Containment do
nil
end
# TODO: We explicitly allow 'tag' URIs through, due to references to legacy OStatus
# objects being present in the test suite environment. Once these objects are
# removed, please also remove this.
if Mix.env() == :test do
defp compare_uris(_, %URI{scheme: "tag"}), do: :ok
end
defp compare_uris(%URI{host: host} = _id_uri, %URI{host: host} = _other_uri), do: :ok
defp compare_uris(_id_uri, _other_uri), do: :error

73
lib/pleroma/object/fetcher.ex
View file

@ -12,6 +12,7 @@ defmodule Pleroma.Object.Fetcher do
alias Pleroma.Web.ActivityPub.ObjectValidator
alias Pleroma.Web.ActivityPub.Transmogrifier
alias Pleroma.Web.Federator
alias Pleroma.Web.FedSockets
require Logger
require Pleroma.Constants
@ -36,8 +37,7 @@ defmodule Pleroma.Object.Fetcher do
defp reinject_object(%Object{data: %{"type" => "Question"}} = object, new_data) do
Logger.debug("Reinjecting object #{new_data["id"]}")
with new_data <- Transmogrifier.fix_object(new_data),
data <- maybe_reinject_internal_fields(object, new_data),
with data <- maybe_reinject_internal_fields(object, new_data),
{:ok, data, _} <- ObjectValidator.validate(data, %{}),
changeset <- Object.change(object, %{data: data}),
changeset <- touch_changeset(changeset),
@ -99,8 +99,8 @@ defmodule Pleroma.Object.Fetcher do
{:containment, _} ->
{:error, "Object containment failed."}
{:transmogrifier, {:error, {:reject, nil}}} ->
{:reject, nil}
{:transmogrifier, {:error, {:reject, e}}} ->
{:reject, e}
{:transmogrifier, _} = e ->
{:error, e}
@ -183,27 +183,20 @@ defmodule Pleroma.Object.Fetcher do
end
end
def fetch_and_contain_remote_object_from_id(id) when is_binary(id) do
def fetch_and_contain_remote_object_from_id(prm, opts \\ [])
def fetch_and_contain_remote_object_from_id(%{"id" => id}, opts),
do: fetch_and_contain_remote_object_from_id(id, opts)
def fetch_and_contain_remote_object_from_id(id, opts) when is_binary(id) do
Logger.debug("Fetching object #{id} via AP")
date = Pleroma.Signature.signed_date()
headers =
[{"accept", "application/activity+json"}]
|> maybe_date_fetch(date)
|> sign_fetch(id, date)
Logger.debug("Fetch headers: #{inspect(headers)}")
with {:scheme, true} <- {:scheme, String.starts_with?(id, "http")},
{:ok, %{body: body, status: code}} when code in 200..299 <- HTTP.get(id, headers),
{:ok, data} <- Jason.decode(body),
{:ok, body} <- get_object(id, opts),
{:ok, data} <- safe_json_decode(body),
:ok <- Containment.contain_origin_from_id(id, data) do
{:ok, data}
else
{:ok, %{status: code}} when code in [404, 410] ->
{:error, "Object has been deleted"}
{:scheme, _} ->
{:error, "Unsupported URI scheme"}
@ -215,8 +208,44 @@ defmodule Pleroma.Object.Fetcher do
end
end
def fetch_and_contain_remote_object_from_id(%{"id" => id}),
do: fetch_and_contain_remote_object_from_id(id)
def fetch_and_contain_remote_object_from_id(_id, _opts),
do: {:error, "id must be a string"}
def fetch_and_contain_remote_object_from_id(_id), do: {:error, "id must be a string"}
defp get_object(id, opts) do
with false <- Keyword.get(opts, :force_http, false),
{:ok, fedsocket} <- FedSockets.get_or_create_fed_socket(id) do
Logger.debug("fetching via fedsocket - #{inspect(id)}")
FedSockets.fetch(fedsocket, id)
else
_other ->
Logger.debug("fetching via http - #{inspect(id)}")
get_object_http(id)
end
end
defp get_object_http(id) do
date = Pleroma.Signature.signed_date()
headers =
[{"accept", "application/activity+json"}]
|> maybe_date_fetch(date)
|> sign_fetch(id, date)
case HTTP.get(id, headers) do
{:ok, %{body: body, status: code}} when code in 200..299 ->
{:ok, body}
{:ok, %{status: code}} when code in [404, 410] ->
{:error, "Object has been deleted"}
{:error, e} ->
{:error, e}
e ->
{:error, e}
end
end
defp safe_json_decode(nil), do: {:ok, nil}
defp safe_json_decode(json), do: Jason.decode(json)
end

2
lib/pleroma/plugs/oauth_scopes_plug.ex
View file

@ -53,7 +53,7 @@ defmodule Pleroma.Plugs.OAuthScopesPlug do
|> assign(:token, nil)
end
@doc "Filters descendants of supported scopes"
@doc "Keeps those of `scopes` which are descendants of `supported_scopes`"
def filter_descendants(scopes, supported_scopes) do
Enum.filter(
scopes,

23
lib/pleroma/repo.ex
View file

@ -49,7 +49,21 @@ defmodule Pleroma.Repo do
end
end
def chunk_stream(query, chunk_size) do
@doc """
Returns a lazy enumerable that emits all entries from the data store matching the given query.
`returns_as` use to group records. use the `batches` option to fetch records in bulk.
## Examples
# fetch records one-by-one
iex> Pleroma.Repo.chunk_stream(Pleroma.Activity.Queries.by_actor(ap_id), 500)
# fetch records in bulk
iex> Pleroma.Repo.chunk_stream(Pleroma.Activity.Queries.by_actor(ap_id), 500, :batches)
"""
@spec chunk_stream(Ecto.Query.t(), integer(), atom()) :: Enumerable.t()
def chunk_stream(query, chunk_size, returns_as \\ :one) do
# We don't actually need start and end funcitons of resource streaming,
# but it seems to be the only way to not fetch records one-by-one and
# have individual records be the elements of the stream, instead of
@ -69,7 +83,12 @@ defmodule Pleroma.Repo do
records ->
last_id = List.last(records).id
{records, last_id}
if returns_as == :one do
{records, last_id}
else
{[records], last_id}
end
end
end,
fn _ -> :ok end

34
lib/pleroma/repo_streamer.ex
View file

@ -1,34 +0,0 @@
# Pleroma: A lightweight social networking server
# Copyright © 2017-2020 Pleroma Authors <https://pleroma.social/>
# SPDX-License-Identifier: AGPL-3.0-only
defmodule Pleroma.RepoStreamer do
alias Pleroma.Repo
import Ecto.Query
def chunk_stream(query, chunk_size) do
Stream.unfold(0, fn
:halt ->
{[], :halt}
last_id ->
query
|> order_by(asc: :id)
|> where([r], r.id > ^last_id)
|> limit(^chunk_size)
|> Repo.all()
|> case do
[] ->
{[], :halt}
records ->
last_id = List.last(records).id
{records, last_id}
end
end)
|> Stream.take_while(fn
[] -> false
_ -> true
end)
end
end

2
lib/pleroma/reverse_proxy/client/tesla.ex
View file

@ -28,7 +28,7 @@ defmodule Pleroma.ReverseProxy.Client.Tesla do
url,
body,
headers,
Keyword.put(opts, :adapter, opts)
opts
) do
if is_map(response.body) and method != :head do
{:ok, response.status, response.headers, response.body}

5
lib/pleroma/reverse_proxy/reverse_proxy.ex
View file

@ -17,6 +17,9 @@ defmodule Pleroma.ReverseProxy do
@failed_request_ttl :timer.seconds(60)
@methods ~w(GET HEAD)
def max_read_duration_default, do: @max_read_duration
def default_cache_control_header, do: @default_cache_control_header
@moduledoc """
A reverse proxy.
@ -391,6 +394,8 @@ defmodule Pleroma.ReverseProxy do
defp body_size_constraint(_, _), do: :ok
defp check_read_duration(nil = _duration, max), do: check_read_duration(@max_read_duration, max)
defp check_read_duration(duration, max)
when is_integer(duration) and is_integer(max) and max > 0 do
if duration > max do

6
lib/pleroma/signature.ex
View file

@ -39,7 +39,7 @@ defmodule Pleroma.Signature do
def fetch_public_key(conn) do
with %{"keyId" => kid} <- HTTPSignatures.signature_for_conn(conn),
{:ok, actor_id} <- key_id_to_actor_id(kid),
{:ok, public_key} <- User.get_public_key_for_ap_id(actor_id) do
{:ok, public_key} <- User.get_public_key_for_ap_id(actor_id, force_http: true) do
{:ok, public_key}
else
e ->
@ -50,8 +50,8 @@ defmodule Pleroma.Signature do
def refetch_public_key(conn) do
with %{"keyId" => kid} <- HTTPSignatures.signature_for_conn(conn),
{:ok, actor_id} <- key_id_to_actor_id(kid),
{:ok, _user} <- ActivityPub.make_user_from_ap_id(actor_id),
{:ok, public_key} <- User.get_public_key_for_ap_id(actor_id) do
{:ok, _user} <- ActivityPub.make_user_from_ap_id(actor_id, force_http: true),
{:ok, public_key} <- User.get_public_key_for_ap_id(actor_id, force_http: true) do
{:ok, public_key}
else
e ->

77
lib/pleroma/stats.ex
View file

@ -3,12 +3,15 @@
# SPDX-License-Identifier: AGPL-3.0-only
defmodule Pleroma.Stats do
use GenServer
import Ecto.Query
alias Pleroma.CounterCache
alias Pleroma.Repo
alias Pleroma.User
use GenServer
@interval :timer.seconds(60)
def start_link(_) do
GenServer.start_link(
@ -18,6 +21,12 @@ defmodule Pleroma.Stats do
)
end
@impl true
def init(_args) do
if Pleroma.Config.get(:env) == :test, do: :ok = Ecto.Adapters.SQL.Sandbox.checkout(Repo)
{:ok, nil, {:continue, :calculate_stats}}
end
@doc "Performs update stats"
def force_update do
GenServer.call(__MODULE__, :force_update)
@ -29,7 +38,11 @@ defmodule Pleroma.Stats do
end
@doc "Returns stats data"
@spec get_stats() :: %{domain_count: integer(), status_count: integer(), user_count: integer()}
@spec get_stats() :: %{
domain_count: non_neg_integer(),
status_count: non_neg_integer(),
user_count: non_neg_integer()
}
def get_stats do
%{stats: stats} = GenServer.call(__MODULE__, :get_state)
@ -44,25 +57,14 @@ defmodule Pleroma.Stats do
peers
end
def init(_args) do
{:ok, calculate_stat_data()}
end
def handle_call(:force_update, _from, _state) do
new_stats = calculate_stat_data()
{:reply, new_stats, new_stats}
end
def handle_call(:get_state, _from, state) do
{:reply, state, state}
end
def handle_cast(:run_update, _state) do
new_stats = calculate_stat_data()
{:noreply, new_stats}
end
@spec calculate_stat_data() :: %{
peers: list(),
stats: %{
domain_count: non_neg_integer(),
status_count: non_neg_integer(),
user_count: non_neg_integer()
}
}
def calculate_stat_data do
peers =
from(
@ -97,6 +99,7 @@ defmodule Pleroma.Stats do
}
end
@spec get_status_visibility_count(String.t() | nil) :: map()
def get_status_visibility_count(instance \\ nil) do
if is_nil(instance) do
CounterCache.get_sum()
@ -104,4 +107,36 @@ defmodule Pleroma.Stats do
CounterCache.get_by_instance(instance)
end
end
@impl true
def handle_continue(:calculate_stats, _) do
stats = calculate_stat_data()
Process.send_after(self(), :run_update, @interval)
{:noreply, stats}
end
@impl true
def handle_call(:force_update, _from, _state) do
new_stats = calculate_stat_data()
{:reply, new_stats, new_stats}
end
@impl true
def handle_call(:get_state, _from, state) do
{:reply, state, state}
end
@impl true
def handle_cast(:run_update, _state) do
new_stats = calculate_stat_data()
{:noreply, new_stats}
end
@impl true
def handle_info(:run_update, _) do
new_stats = calculate_stat_data()
Process.send_after(self(), :run_update, @interval)
{:noreply, new_stats}
end
end

18
lib/pleroma/telemetry/logger.ex
View file

@ -7,7 +7,8 @@ defmodule Pleroma.Telemetry.Logger do
[:pleroma, :connection_pool, :reclaim, :start],
[:pleroma, :connection_pool, :reclaim, :stop],
[:pleroma, :connection_pool, :provision_failure],
[:pleroma, :connection_pool, :client_death]
[:pleroma, :connection_pool, :client, :dead],
[:pleroma, :connection_pool, :client, :add]
]
def attach do
:telemetry.attach_many("pleroma-logger", @events, &handle_event/4, [])
@ -62,7 +63,7 @@ defmodule Pleroma.Telemetry.Logger do
end
def handle_event(
[:pleroma, :connection_pool, :client_death],
[:pleroma, :connection_pool, :client, :dead],
%{client_pid: client_pid, reason: reason},
%{key: key},
_
@ -73,4 +74,17 @@ defmodule Pleroma.Telemetry.Logger do
}"
end)
end
def handle_event(
[:pleroma, :connection_pool, :client, :add],
%{clients: [_, _ | _] = clients},
%{key: key, protocol: :http},
_
) do
Logger.info(fn ->
"Pool worker for #{key}: #{length(clients)} clients are using an HTTP1 connection at the same time, head-of-line blocking might occur."
end)
end
def handle_event([:pleroma, :connection_pool, :client, :add], _, _, _), do: :ok
end

50
lib/pleroma/tesla/middleware/connection_pool.ex Normal file
View file

@ -0,0 +1,50 @@
# Pleroma: A lightweight social networking server
# Copyright © 2020 Pleroma Authors <https://pleroma.social/>
# SPDX-License-Identifier: AGPL-3.0-only
defmodule Pleroma.Tesla.Middleware.ConnectionPool do
@moduledoc """
Middleware to get/release connections from `Pleroma.Gun.ConnectionPool`
"""
@behaviour Tesla.Middleware
alias Pleroma.Gun.ConnectionPool
@impl Tesla.Middleware
def call(%Tesla.Env{url: url, opts: opts} = env, next, _) do
uri = URI.parse(url)
# Avoid leaking connections when the middleware is called twice
# with body_as: :chunks. We assume only the middleware can set
# opts[:adapter][:conn]
if opts[:adapter][:conn] do
ConnectionPool.release_conn(opts[:adapter][:conn])
end
case ConnectionPool.get_conn(uri, opts[:adapter]) do
{:ok, conn_pid} ->
adapter_opts = Keyword.merge(opts[:adapter], conn: conn_pid, close_conn: false)
opts = Keyword.put(opts, :adapter, adapter_opts)
env = %{env | opts: opts}
case Tesla.run(env, next) do
{:ok, env} ->
unless opts[:adapter][:body_as] == :chunks do
ConnectionPool.release_conn(conn_pid)
{_, res} = pop_in(env.opts[:adapter][:conn])
{:ok, res}
else
{:ok, env}
end
err ->
ConnectionPool.release_conn(conn_pid)
err
end
err ->
err
end
end
end

110
lib/pleroma/tesla/middleware/follow_redirects.ex
View file

@ -1,110 +0,0 @@
# Pleroma: A lightweight social networking server
# Copyright © 2015-2020 Tymon Tobolski <https://github.com/teamon/tesla/blob/master/lib/tesla/middleware/follow_redirects.ex>
# Copyright © 2020 Pleroma Authors <https://pleroma.social/>
# SPDX-License-Identifier: AGPL-3.0-only
defmodule Pleroma.HTTP.Middleware.FollowRedirects do
@moduledoc """
Pool-aware version of https://github.com/teamon/tesla/blob/master/lib/tesla/middleware/follow_redirects.ex
Follow 3xx redirects
## Options
- `:max_redirects` - limit number of redirects (default: `5`)
"""
alias Pleroma.Gun.ConnectionPool
@behaviour Tesla.Middleware
@max_redirects 5
@redirect_statuses [301, 302, 303, 307, 308]
@impl Tesla.Middleware
def call(env, next, opts \\ []) do
max = Keyword.get(opts, :max_redirects, @max_redirects)
redirect(env, next, max)
end
defp redirect(env, next, left) do
opts = env.opts[:adapter]
case Tesla.run(env, next) do
{:ok, %{status: status} = res} when status in @redirect_statuses and left > 0 ->
release_conn(opts)
case Tesla.get_header(res, "location") do
nil ->
{:ok, res}
location ->
location = parse_location(location, res)
case get_conn(location, opts) do
{:ok, opts} ->
%{env | opts: Keyword.put(env.opts, :adapter, opts)}
|> new_request(res.status, location)
|> redirect(next, left - 1)
e ->
e
end
end
{:ok, %{status: status}} when status in @redirect_statuses ->
release_conn(opts)
{:error, {__MODULE__, :too_many_redirects}}
{:error, _} = e ->
release_conn(opts)
e
other ->
unless opts[:body_as] == :chunks do
release_conn(opts)
end
other
end
end
defp get_conn(location, opts) do
uri = URI.parse(location)
case ConnectionPool.get_conn(uri, opts) do
{:ok, conn} ->
{:ok, Keyword.merge(opts, conn: conn)}
e ->
e
end
end
defp release_conn(opts) do
ConnectionPool.release_conn(opts[:conn])
end
# The 303 (See Other) redirect was added in HTTP/1.1 to indicate that the originally
# requested resource is not available, however a related resource (or another redirect)
# available via GET is available at the specified location.
# https://tools.ietf.org/html/rfc7231#section-6.4.4
defp new_request(env, 303, location), do: %{env | url: location, method: :get, query: []}
# The 307 (Temporary Redirect) status code indicates that the target
# resource resides temporarily under a different URI and the user agent
# MUST NOT change the request method (...)
# https://tools.ietf.org/html/rfc7231#section-6.4.7
defp new_request(env, 307, location), do: %{env | url: location}
defp new_request(env, _, location), do: %{env | url: location, query: []}
defp parse_location("https://" <> _rest = location, _env), do: location
defp parse_location("http://" <> _rest = location, _env), do: location
defp parse_location(location, env) do
env.url
|> URI.parse()
|> URI.merge(location)
|> URI.to_string()
end
end

89
lib/pleroma/user.ex
View file

@ -25,7 +25,6 @@ defmodule Pleroma.User do
alias Pleroma.Object
alias Pleroma.Registration
alias Pleroma.Repo
alias Pleroma.RepoStreamer
alias Pleroma.User
alias Pleroma.UserRelationship
alias Pleroma.Web
@ -83,7 +82,7 @@ defmodule Pleroma.User do
]
schema "users" do
field(:bio, :string)
field(:bio, :string, default: "")
field(:raw_bio, :string)
field(:email, :string)
field(:name, :string)
@ -276,9 +275,9 @@ defmodule Pleroma.User do
@spec account_status(User.t()) :: account_status()
def account_status(%User{deactivated: true}), do: :deactivated
def account_status(%User{password_reset_pending: true}), do: :password_reset_pending
def account_status(%User{approval_pending: true}), do: :approval_pending
def account_status(%User{local: true, approval_pending: true}), do: :approval_pending
def account_status(%User{confirmation_pending: true}) do
def account_status(%User{local: true, confirmation_pending: true}) do
if Config.get([:instance, :account_activation_required]) do
:confirmation_pending
else
@ -1587,7 +1586,7 @@ defmodule Pleroma.User do
# "Right to be forgotten"
# https://gdpr.eu/right-to-be-forgotten/
change(user, %{
bio: nil,
bio: "",
raw_bio: nil,
email: nil,
name: nil,
@ -1686,42 +1685,6 @@ defmodule Pleroma.User do
def perform(:deactivate_async, user, status), do: deactivate(user, status)
@spec perform(atom(), User.t(), list()) :: list() | {:error, any()}
def perform(:blocks_import, %User{} = blocker, blocked_identifiers)
when is_list(blocked_identifiers) do
Enum.map(
blocked_identifiers,
fn blocked_identifier ->
with {:ok, %User{} = blocked} <- get_or_fetch(blocked_identifier),
{:ok, _block} <- CommonAPI.block(blocker, blocked) do
blocked
else
err ->
Logger.debug("blocks_import failed for #{blocked_identifier} with: #{inspect(err)}")
err
end
end
)
end
def perform(:follow_import, %User{} = follower, followed_identifiers)
when is_list(followed_identifiers) do
Enum.map(
followed_identifiers,
fn followed_identifier ->
with {:ok, %User{} = followed} <- get_or_fetch(followed_identifier),
{:ok, follower} <- maybe_direct_follow(follower, followed),
{:ok, _, _, _} <- CommonAPI.follow(follower, followed) do
followed
else
err ->
Logger.debug("follow_import failed for #{followed_identifier} with: #{inspect(err)}")
err
end
end
)
end
@spec external_users_query() :: Ecto.Query.t()
def external_users_query do
User.Query.build(%{
@ -1750,21 +1713,6 @@ defmodule Pleroma.User do
Repo.all(query)
end
def blocks_import(%User{} = blocker, blocked_identifiers) when is_list(blocked_identifiers) do
BackgroundWorker.enqueue("blocks_import", %{
"blocker_id" => blocker.id,
"blocked_identifiers" => blocked_identifiers
})
end
def follow_import(%User{} = follower, followed_identifiers)
when is_list(followed_identifiers) do
BackgroundWorker.enqueue("follow_import", %{
"follower_id" => follower.id,
"followed_identifiers" => followed_identifiers
})
end
def delete_notifications_from_user_activities(%User{ap_id: ap_id}) do
Notification
|> join(:inner, [n], activity in assoc(n, :activity))
@ -1775,7 +1723,7 @@ defmodule Pleroma.User do
def delete_user_activities(%User{ap_id: ap_id} = user) do
ap_id
|> Activity.Queries.by_actor()
|> RepoStreamer.chunk_stream(50)
|> Repo.chunk_stream(50, :batches)
|> Stream.each(fn activities ->
Enum.each(activities, fn activity -> delete_activity(activity, user) end)
end)
@ -1821,12 +1769,12 @@ defmodule Pleroma.User do
def html_filter_policy(_), do: Config.get([:markup, :scrub_policy])
def fetch_by_ap_id(ap_id), do: ActivityPub.make_user_from_ap_id(ap_id)
def fetch_by_ap_id(ap_id, opts \\ []), do: ActivityPub.make_user_from_ap_id(ap_id, opts)
def get_or_fetch_by_ap_id(ap_id) do
def get_or_fetch_by_ap_id(ap_id, opts \\ []) do
cached_user = get_cached_by_ap_id(ap_id)
maybe_fetched_user = needs_update?(cached_user) && fetch_by_ap_id(ap_id)
maybe_fetched_user = needs_update?(cached_user) && fetch_by_ap_id(ap_id, opts)
case {cached_user, maybe_fetched_user} do
{_, {:ok, %User{} = user}} ->
@ -1899,8 +1847,8 @@ defmodule Pleroma.User do
def public_key(_), do: {:error, "key not found"}
def get_public_key_for_ap_id(ap_id) do
with {:ok, %User{} = user} <- get_or_fetch_by_ap_id(ap_id),
def get_public_key_for_ap_id(ap_id, opts \\ []) do
with {:ok, %User{} = user} <- get_or_fetch_by_ap_id(ap_id, opts),
{:ok, public_key} <- public_key(user) do
{:ok, public_key}
else
@ -2315,6 +2263,11 @@ defmodule Pleroma.User do
max_pinned_statuses = Config.get([:instance, :max_pinned_statuses], 0)
params = %{pinned_activities: user.pinned_activities ++ [id]}
# if pinned activity was scheduled for deletion, we remove job
if expiration = Pleroma.Workers.PurgeExpiredActivity.get_expiration(id) do
Oban.cancel_job(expiration.id)
end
user
|> cast(params, [:pinned_activities])
|> validate_length(:pinned_activities,
@ -2327,9 +2280,19 @@ defmodule Pleroma.User do
|> update_and_set_cache()
end
def remove_pinnned_activity(user, %Pleroma.Activity{id: id}) do
def remove_pinnned_activity(user, %Pleroma.Activity{id: id, data: data}) do
params = %{pinned_activities: List.delete(user.pinned_activities, id)}
# if pinned activity was scheduled for deletion, we reschedule it for deletion
if data["expires_at"] do
{:ok, expires_at, _} = DateTime.from_iso8601(data["expires_at"])
Pleroma.Workers.PurgeExpiredActivity.enqueue(%{
activity_id: id,
expires_at: expires_at
})
end
user
|> cast(params, [:pinned_activities])
|> update_and_set_cache()

85
lib/pleroma/user/import.ex Normal file
View file

@ -0,0 +1,85 @@
# Pleroma: A lightweight social networking server
# Copyright © 2017-2020 Pleroma Authors <https://pleroma.social/>
# SPDX-License-Identifier: AGPL-3.0-only
defmodule Pleroma.User.Import do
use Ecto.Schema
alias Pleroma.User
alias Pleroma.Web.CommonAPI
alias Pleroma.Workers.BackgroundWorker
require Logger
@spec perform(atom(), User.t(), list()) :: :ok | list() | {:error, any()}
def perform(:mutes_import, %User{} = user, [_ | _] = identifiers) do
Enum.map(
identifiers,
fn identifier ->
with {:ok, %User{} = muted_user} <- User.get_or_fetch(identifier),
{:ok, _} <- User.mute(user, muted_user) do
muted_user
else
error -> handle_error(:mutes_import, identifier, error)
end
end
)
end
def perform(:blocks_import, %User{} = blocker, [_ | _] = identifiers) do
Enum.map(
identifiers,
fn identifier ->
with {:ok, %User{} = blocked} <- User.get_or_fetch(identifier),
{:ok, _block} <- CommonAPI.block(blocker, blocked) do
blocked
else
error -> handle_error(:blocks_import, identifier, error)
end
end
)
end
def perform(:follow_import, %User{} = follower, [_ | _] = identifiers) do
Enum.map(
identifiers,
fn identifier ->
with {:ok, %User{} = followed} <- User.get_or_fetch(identifier),
{:ok, follower} <- User.maybe_direct_follow(follower, followed),
{:ok, _, _, _} <- CommonAPI.follow(follower, followed) do
followed
else
error -> handle_error(:follow_import, identifier, error)
end
end
)
end
def perform(_, _, _), do: :ok
defp handle_error(op, user_id, error) do
Logger.debug("#{op} failed for #{user_id} with: #{inspect(error)}")
error
end
def blocks_import(%User{} = blocker, [_ | _] = identifiers) do
BackgroundWorker.enqueue(
"blocks_import",
%{"user_id" => blocker.id, "identifiers" => identifiers}
)
end
def follow_import(%User{} = follower, [_ | _] = identifiers) do
BackgroundWorker.enqueue(
"follow_import",
%{"user_id" => follower.id, "identifiers" => identifiers}
)
end
def mutes_import(%User{} = user, [_ | _] = identifiers) do
BackgroundWorker.enqueue(
"mutes_import",
%{"user_id" => user.id, "identifiers" => identifiers}
)
end
end

5
lib/pleroma/user/search.ex
View file

@ -52,6 +52,7 @@ defmodule Pleroma.User.Search do
|> base_query(following)
|> filter_blocked_user(for_user)
|> filter_invisible_users()
|> filter_discoverable_users()
|> filter_internal_users()
|> filter_blocked_domains(for_user)
|> fts_search(query_string)
@ -122,6 +123,10 @@ defmodule Pleroma.User.Search do
from(q in query, where: q.invisible == false)
end
defp filter_discoverable_users(query) do
from(q in query, where: q.discoverable == true)
end
defp filter_internal_users(query) do
from(q in query, where: q.actor_type != "Application")
end

90
lib/pleroma/web/activity_pub/activity_pub.ex
View file

@ -5,7 +5,6 @@
defmodule Pleroma.Web.ActivityPub.ActivityPub do
alias Pleroma.Activity
alias Pleroma.Activity.Ir.Topics
alias Pleroma.ActivityExpiration
alias Pleroma.Config
alias Pleroma.Constants
alias Pleroma.Conversation
@ -85,7 +84,7 @@ defmodule Pleroma.Web.ActivityPub.ActivityPub do
defp increase_replies_count_if_reply(_create_data), do: :noop
@object_types ~w[ChatMessage Question Answer Audio Event]
@object_types ~w[ChatMessage Question Answer Audio Video Event Article]
@spec persist(map(), keyword()) :: {:ok, Activity.t() | Object.t()}
def persist(%{"type" => type} = object, meta) when type in @object_types do
with {:ok, object} <- Object.create(object) do
@ -102,7 +101,9 @@ defmodule Pleroma.Web.ActivityPub.ActivityPub do
local: local,
recipients: recipients,
actor: object["actor"]
}) do
}),
# TODO: add tests for expired activities, when Note type will be supported in new pipeline
{:ok, _} <- maybe_create_activity_expiration(activity) do
{:ok, activity, meta}
end
end
@ -111,23 +112,14 @@ defmodule Pleroma.Web.ActivityPub.ActivityPub do
def insert(map, local \\ true, fake \\ false, bypass_actor_check \\ false) when is_map(map) do
with nil <- Activity.normalize(map),
map <- lazy_put_activity_defaults(map, fake),
true <- bypass_actor_check || check_actor_is_active(map["actor"]),
{_, true} <- {:remote_limit_error, check_remote_limit(map)},
{_, true} <- {:actor_check, bypass_actor_check || check_actor_is_active(map["actor"])},
{_, true} <- {:remote_limit_pass, check_remote_limit(map)},
{:ok, map} <- MRF.filter(map),
{recipients, _, _} = get_recipients(map),
{:fake, false, map, recipients} <- {:fake, fake, map, recipients},
{:containment, :ok} <- {:containment, Containment.contain_child(map)},
{:ok, map, object} <- insert_full_object(map) do
{:ok, activity} =
%Activity{
data: map,
local: local,
actor: map["actor"],
recipients: recipients
}
|> Repo.insert()
|> maybe_create_activity_expiration()
{:ok, map, object} <- insert_full_object(map),
{:ok, activity} <- insert_activity_with_expiration(map, local, recipients) do
# Splice in the child object if we have one.
activity = Maps.put_if_present(activity, :object, object)
@ -138,6 +130,15 @@ defmodule Pleroma.Web.ActivityPub.ActivityPub do
%Activity{} = activity ->
{:ok, activity}
{:actor_check, _} ->
{:error, false}
{:containment, _} = error ->
error
{:error, _} = error ->
error
{:fake, true, map, recipients} ->
activity = %Activity{
data: map,
@ -150,8 +151,24 @@ defmodule Pleroma.Web.ActivityPub.ActivityPub do
Pleroma.Web.RichMedia.Helpers.fetch_data_for_activity(activity)
{:ok, activity}
error ->
{:error, error}
{:remote_limit_pass, _} ->
{:error, :remote_limit}
{:reject, _} = e ->
{:error, e}
end
end
defp insert_activity_with_expiration(data, local, recipients) do
struct = %Activity{
data: data,
local: local,
actor: data["actor"],
recipients: recipients
}
with {:ok, activity} <- Repo.insert(struct) do
maybe_create_activity_expiration(activity)
end
end
@ -164,13 +181,19 @@ defmodule Pleroma.Web.ActivityPub.ActivityPub do
stream_out_participations(participations)
end
defp maybe_create_activity_expiration({:ok, %{data: %{"expires_at" => expires_at}} = activity}) do
with {:ok, _} <- ActivityExpiration.create(activity, expires_at) do
defp maybe_create_activity_expiration(
%{data: %{"expires_at" => %DateTime{} = expires_at}} = activity
) do
with {:ok, _job} <-
Pleroma.Workers.PurgeExpiredActivity.enqueue(%{
activity_id: activity.id,
expires_at: expires_at
}) do
{:ok, activity}
end
end
defp maybe_create_activity_expiration(result), do: result
defp maybe_create_activity_expiration(activity), do: {:ok, activity}
defp create_or_bump_conversation(activity, actor) do
with {:ok, conversation} <- Conversation.create_or_bump_for(activity),
@ -818,7 +841,14 @@ defmodule Pleroma.Web.ActivityPub.ActivityPub do
from(
[activity, object: o] in query,
where: fragment("not (? = ANY(?))", activity.actor, ^blocked_ap_ids),
where: fragment("not (? && ?)", activity.recipients, ^blocked_ap_ids),
where:
fragment(
"((not (? && ?)) or ? = ?)",
activity.recipients,
^blocked_ap_ids,
activity.actor,
^user.ap_id
),
where:
fragment(
"recipients_contain_blocked_domains(?, ?) = false",
@ -1224,7 +1254,7 @@ defmodule Pleroma.Web.ActivityPub.ActivityPub do
name: data["name"],
follower_address: data["followers"],
following_address: data["following"],
bio: data["summary"],
bio: data["summary"] || "",
actor_type: actor_type,
also_known_as: Map.get(data, "alsoKnownAs", []),
public_key: public_key,
@ -1247,10 +1277,12 @@ defmodule Pleroma.Web.ActivityPub.ActivityPub do
def fetch_follow_information_for_user(user) do
with {:ok, following_data} <-
Fetcher.fetch_and_contain_remote_object_from_id(user.following_address),
Fetcher.fetch_and_contain_remote_object_from_id(user.following_address,
force_http: true
),
{:ok, hide_follows} <- collection_private(following_data),
{:ok, followers_data} <-
Fetcher.fetch_and_contain_remote_object_from_id(user.follower_address),
Fetcher.fetch_and_contain_remote_object_from_id(user.follower_address, force_http: true),
{:ok, hide_followers} <- collection_private(followers_data) do
{:ok,
%{
@ -1324,8 +1356,8 @@ defmodule Pleroma.Web.ActivityPub.ActivityPub do
end
end
def fetch_and_prepare_user_from_ap_id(ap_id) do
with {:ok, data} <- Fetcher.fetch_and_contain_remote_object_from_id(ap_id),
def fetch_and_prepare_user_from_ap_id(ap_id, opts \\ []) do
with {:ok, data} <- Fetcher.fetch_and_contain_remote_object_from_id(ap_id, opts),
{:ok, data} <- user_data_from_user_object(data) do
{:ok, maybe_update_follow_information(data)}
else
@ -1367,13 +1399,13 @@ defmodule Pleroma.Web.ActivityPub.ActivityPub do
end
end
def make_user_from_ap_id(ap_id) do
def make_user_from_ap_id(ap_id, opts \\ []) do
user = User.get_cached_by_ap_id(ap_id)
if user && !User.ap_enabled?(user) do
Transmogrifier.upgrade_user_from_ap_id(ap_id)
else
with {:ok, data} <- fetch_and_prepare_user_from_ap_id(ap_id) do
with {:ok, data} <- fetch_and_prepare_user_from_ap_id(ap_id, opts) do
if user do
user
|> User.remote_user_changeset(data)

4
lib/pleroma/web/activity_pub/mrf/activity_expiration_policy.ex
View file

@ -31,10 +31,10 @@ defmodule Pleroma.Web.ActivityPub.MRF.ActivityExpirationPolicy do
defp maybe_add_expiration(activity) do
days = Pleroma.Config.get([:mrf_activity_expiration, :days], 365)
expires_at = NaiveDateTime.utc_now() |> Timex.shift(days: days)
expires_at = DateTime.utc_now() |> Timex.shift(days: days)
with %{"expires_at" => existing_expires_at} <- activity,
:lt <- NaiveDateTime.compare(existing_expires_at, expires_at) do
:lt <- DateTime.compare(existing_expires_at, expires_at) do
activity
else
_ -> Map.put(activity, "expires_at", expires_at)

56
lib/pleroma/web/activity_pub/mrf/force_bot_unlisted_policy.ex Normal file
View file

@ -0,0 +1,56 @@
# Pleroma: A lightweight social networking server
# Copyright © 2017-2020 Pleroma Authors <https://pleroma.social/>
# SPDX-License-Identifier: AGPL-3.0-only
defmodule Pleroma.Web.ActivityPub.MRF.ForceBotUnlistedPolicy do
alias Pleroma.User
@behaviour Pleroma.Web.ActivityPub.MRF
@moduledoc "Remove bot posts from federated timeline"
require Pleroma.Constants
defp check_by_actor_type(user), do: user.actor_type in ["Application", "Service"]
defp check_by_nickname(user), do: Regex.match?(~r/bot@|ebooks@/i, user.nickname)
defp check_if_bot(user), do: check_by_actor_type(user) or check_by_nickname(user)
@impl true
def filter(
%{
"type" => "Create",
"to" => to,
"cc" => cc,
"actor" => actor,
"object" => object
} = message
) do
user = User.get_cached_by_ap_id(actor)
isbot = check_if_bot(user)
if isbot and Enum.member?(to, Pleroma.Constants.as_public()) do
to = List.delete(to, Pleroma.Constants.as_public()) ++ [user.follower_address]
cc = List.delete(cc, user.follower_address) ++ [Pleroma.Constants.as_public()]
object =
object
|> Map.put("to", to)
|> Map.put("cc", cc)
message =
message
|> Map.put("to", to)
|> Map.put("cc", cc)
|> Map.put("object", object)
{:ok, message}
else
{:ok, message}
end
end
@impl true
def filter(message), do: {:ok, message}
@impl true
def describe, do: {:ok, %{}}
end

22
lib/pleroma/web/activity_pub/mrf/media_proxy_warming_policy.ex
View file

@ -12,23 +12,21 @@ defmodule Pleroma.Web.ActivityPub.MRF.MediaProxyWarmingPolicy do
require Logger
@options [
pool: :media
@adapter_options [
pool: :media,
recv_timeout: 10_000
]
def perform(:prefetch, url) do
Logger.debug("Prefetching #{inspect(url)}")
# Fetching only proxiable resources
if MediaProxy.enabled?() and MediaProxy.url_proxiable?(url) do
# If preview proxy is enabled, it'll also hit media proxy (so we're caching both requests)
prefetch_url = MediaProxy.preview_url(url)
opts =
if Application.get_env(:tesla, :adapter) == Tesla.Adapter.Hackney do
Keyword.put(@options, :recv_timeout, 10_000)
else
@options
end
Logger.debug("Prefetching #{inspect(url)} as #{inspect(prefetch_url)}")
url
|> MediaProxy.url()
|> HTTP.get([], adapter: opts)
HTTP.get(prefetch_url, [], @adapter_options)
end
end
def perform(:preload, %{"object" => %{"attachment" => attachments}} = _message) do

31
lib/pleroma/web/activity_pub/object_validator.ex
View file

@ -12,11 +12,13 @@ defmodule Pleroma.Web.ActivityPub.ObjectValidator do
alias Pleroma.Activity
alias Pleroma.EctoType.ActivityPub.ObjectValidators
alias Pleroma.Object
alias Pleroma.Object.Containment
alias Pleroma.User
alias Pleroma.Web.ActivityPub.ObjectValidators.AcceptRejectValidator
alias Pleroma.Web.ActivityPub.ObjectValidators.AnnounceValidator
alias Pleroma.Web.ActivityPub.ObjectValidators.AnswerValidator
alias Pleroma.Web.ActivityPub.ObjectValidators.AudioValidator
alias Pleroma.Web.ActivityPub.ObjectValidators.ArticleNoteValidator
alias Pleroma.Web.ActivityPub.ObjectValidators.AudioVideoValidator
alias Pleroma.Web.ActivityPub.ObjectValidators.BlockValidator
alias Pleroma.Web.ActivityPub.ObjectValidators.ChatMessageValidator
alias Pleroma.Web.ActivityPub.ObjectValidators.CreateChatMessageValidator
@ -149,10 +151,20 @@ defmodule Pleroma.Web.ActivityPub.ObjectValidator do
end
end
def validate(%{"type" => "Audio"} = object, meta) do
def validate(%{"type" => type} = object, meta) when type in ~w[Audio Video] do
with {:ok, object} <-
object
|> AudioValidator.cast_and_validate()
|> AudioVideoValidator.cast_and_validate()
|> Ecto.Changeset.apply_action(:insert) do
object = stringify_keys(object)
{:ok, object, meta}
end
end
def validate(%{"type" => "Article"} = object, meta) do
with {:ok, object} <-
object
|> ArticleNoteValidator.cast_and_validate()
|> Ecto.Changeset.apply_action(:insert) do
object = stringify_keys(object)
{:ok, object, meta}
@ -198,7 +210,7 @@ defmodule Pleroma.Web.ActivityPub.ObjectValidator do
%{"type" => "Create", "object" => %{"type" => objtype} = object} = create_activity,
meta
)
when objtype in ~w[Question Answer Audio Event] do
when objtype in ~w[Question Answer Audio Video Event Article] do
with {:ok, object_data} <- cast_and_apply(object),
meta = Keyword.put(meta, :object_data, object_data |> stringify_keys),
{:ok, create_activity} <-
@ -232,14 +244,18 @@ defmodule Pleroma.Web.ActivityPub.ObjectValidator do
AnswerValidator.cast_and_apply(object)
end
def cast_and_apply(%{"type" => "Audio"} = object) do
AudioValidator.cast_and_apply(object)
def cast_and_apply(%{"type" => type} = object) when type in ~w[Audio Video] do
AudioVideoValidator.cast_and_apply(object)
end
def cast_and_apply(%{"type" => "Event"} = object) do
EventValidator.cast_and_apply(object)
end
def cast_and_apply(%{"type" => "Article"} = object) do
ArticleNoteValidator.cast_and_apply(object)
end
def cast_and_apply(o), do: {:error, {:validator_not_set, o}}
# is_struct/1 isn't present in Elixir 1.8.x
@ -262,7 +278,8 @@ defmodule Pleroma.Web.ActivityPub.ObjectValidator do
def stringify_keys(object), do: object
def fetch_actor(object) do
with {:ok, actor} <- ObjectValidators.ObjectID.cast(object["actor"]) do
with actor <- Containment.get_actor(object),
{:ok, actor} <- ObjectValidators.ObjectID.cast(actor) do
User.get_or_fetch_by_ap_id(actor)
end
end

40
lib/pleroma/web/activity_pub/object_validators/audio_validator.ex → lib/pleroma/web/activity_pub/object_validators/article_note_validator.ex
View file

@ -2,13 +2,14 @@
# Copyright © 2017-2020 Pleroma Authors <https://pleroma.social/>
# SPDX-License-Identifier: AGPL-3.0-only
defmodule Pleroma.Web.ActivityPub.ObjectValidators.AudioValidator do
defmodule Pleroma.Web.ActivityPub.ObjectValidators.ArticleNoteValidator do
use Ecto.Schema
alias Pleroma.EctoType.ActivityPub.ObjectValidators
alias Pleroma.Web.ActivityPub.ObjectValidators.AttachmentValidator
alias Pleroma.Web.ActivityPub.ObjectValidators.CommonFixes
alias Pleroma.Web.ActivityPub.ObjectValidators.CommonValidations
alias Pleroma.Web.ActivityPub.Transmogrifier
import Ecto.Changeset
@ -24,29 +25,31 @@ defmodule Pleroma.Web.ActivityPub.ObjectValidators.AudioValidator do
# TODO: Write type
field(:tag, {:array, :map}, default: [])
field(:type, :string)
field(:name, :string)
field(:summary, :string)
field(:content, :string)
field(:context, :string)
# short identifier for PleromaFE to group statuses by context
field(:context_id, :integer)
# TODO: Remove actor on objects
field(:actor, ObjectValidators.ObjectID)
field(:attributedTo, ObjectValidators.ObjectID)
field(:summary, :string)
field(:published, ObjectValidators.DateTime)
# TODO: Write type
field(:emoji, :map, default: %{})
field(:emoji, ObjectValidators.Emoji, default: %{})
field(:sensitive, :boolean, default: false)
embeds_many(:attachment, AttachmentValidator)
field(:replies_count, :integer, default: 0)
field(:like_count, :integer, default: 0)
field(:announcement_count, :integer, default: 0)
field(:inReplyTo, :string)
field(:inReplyTo, ObjectValidators.ObjectID)
field(:url, ObjectValidators.Uri)
# short identifier for PleromaFE to group statuses by context
field(:context_id, :integer)
field(:likes, {:array, :string}, default: [])
field(:announcements, {:array, :string}, default: [])
field(:likes, {:array, ObjectValidators.ObjectID}, default: [])
field(:announcements, {:array, ObjectValidators.ObjectID}, default: [])
end
def cast_and_apply(data) do
@ -62,19 +65,14 @@ defmodule Pleroma.Web.ActivityPub.ObjectValidators.AudioValidator do
end
def cast_data(data) do
data = fix(data)
%__MODULE__{}
|> changeset(data)
end
defp fix_url(%{"url" => url} = data) when is_list(url) do
attachment =
Enum.find(url, fn x -> is_map(x) and String.starts_with?(x["mimeType"], "audio/") end)
link_element = Enum.find(url, fn x -> is_map(x) and x["mimeType"] == "text/html" end)
data
|> Map.put("attachment", [attachment])
|> Map.put("url", link_element["href"])
defp fix_url(%{"url" => url} = data) when is_map(url) do
Map.put(data, "url", url["href"])
end
defp fix_url(data), do: data
@ -83,7 +81,9 @@ defmodule Pleroma.Web.ActivityPub.ObjectValidators.AudioValidator do
data
|> CommonFixes.fix_defaults()
|> CommonFixes.fix_attribution()
|> CommonFixes.fix_actor()
|> fix_url()
|> Transmogrifier.fix_emoji()
end
def changeset(struct, data) do
@ -96,8 +96,8 @@ defmodule Pleroma.Web.ActivityPub.ObjectValidators.AudioValidator do
def validate_data(data_cng) do
data_cng
|> validate_inclusion(:type, ["Audio"])
|> validate_required([:id, :actor, :attributedTo, :type, :context, :attachment])
|> validate_inclusion(:type, ["Article", "Note"])
|> validate_required([:id, :actor, :attributedTo, :type, :context, :context_id])
|> CommonValidations.validate_any_presence([:cc, :to])
|> CommonValidations.validate_fields_match([:actor, :attributedTo])
|> CommonValidations.validate_actor_presence()

21
lib/pleroma/web/activity_pub/object_validators/attachment_validator.ex
View file

@ -5,6 +5,7 @@
defmodule Pleroma.Web.ActivityPub.ObjectValidators.AttachmentValidator do
use Ecto.Schema
alias Pleroma.EctoType.ActivityPub.ObjectValidators
alias Pleroma.Web.ActivityPub.ObjectValidators.UrlObjectValidator
import Ecto.Changeset
@ -15,7 +16,11 @@ defmodule Pleroma.Web.ActivityPub.ObjectValidators.AttachmentValidator do
field(:mediaType, :string, default: "application/octet-stream")
field(:name, :string)
embeds_many(:url, UrlObjectValidator)
embeds_many :url, UrlObjectValidator, primary_key: false do
field(:type, :string)
field(:href, ObjectValidators.Uri)
field(:mediaType, :string, default: "application/octet-stream")
end
end
def cast_and_validate(data) do
@ -37,7 +42,18 @@ defmodule Pleroma.Web.ActivityPub.ObjectValidators.AttachmentValidator do
struct
|> cast(data, [:type, :mediaType, :name])
|> cast_embed(:url, required: true)
|> cast_embed(:url, with: &url_changeset/2)
|> validate_inclusion(:type, ~w[Link Document Audio Image Video])
|> validate_required([:type, :mediaType, :url])
end
def url_changeset(struct, data) do
data = fix_media_type(data)
struct
|> cast(data, [:type, :href, :mediaType])
|> validate_inclusion(:type, ["Link"])
|> validate_required([:type, :href, :mediaType])
end
def fix_media_type(data) do
@ -75,6 +91,7 @@ defmodule Pleroma.Web.ActivityPub.ObjectValidators.AttachmentValidator do
def validate_data(cng) do
cng
|> validate_inclusion(:type, ~w[Document Audio Image Video])
|> validate_required([:mediaType, :url, :type])
end
end

134
lib/pleroma/web/activity_pub/object_validators/audio_video_validator.ex Normal file
View file

@ -0,0 +1,134 @@
# Pleroma: A lightweight social networking server
# Copyright © 2017-2020 Pleroma Authors <https://pleroma.social/>
# SPDX-License-Identifier: AGPL-3.0-only
defmodule Pleroma.Web.ActivityPub.ObjectValidators.AudioVideoValidator do
use Ecto.Schema
alias Pleroma.EarmarkRenderer
alias Pleroma.EctoType.ActivityPub.ObjectValidators
alias Pleroma.Web.ActivityPub.ObjectValidators.AttachmentValidator
alias Pleroma.Web.ActivityPub.ObjectValidators.CommonFixes
alias Pleroma.Web.ActivityPub.ObjectValidators.CommonValidations
alias Pleroma.Web.ActivityPub.Transmogrifier
import Ecto.Changeset
@primary_key false
@derive Jason.Encoder
embedded_schema do
field(:id, ObjectValidators.ObjectID, primary_key: true)
field(:to, ObjectValidators.Recipients, default: [])
field(:cc, ObjectValidators.Recipients, default: [])
field(:bto, ObjectValidators.Recipients, default: [])
field(:bcc, ObjectValidators.Recipients, default: [])
# TODO: Write type
field(:tag, {:array, :map}, default: [])
field(:type, :string)
field(:name, :string)
field(:summary, :string)
field(:content, :string)
field(:context, :string)
# short identifier for PleromaFE to group statuses by context
field(:context_id, :integer)
# TODO: Remove actor on objects
field(:actor, ObjectValidators.ObjectID)
field(:attributedTo, ObjectValidators.ObjectID)
field(:published, ObjectValidators.DateTime)
field(:emoji, ObjectValidators.Emoji, default: %{})
field(:sensitive, :boolean, default: false)
embeds_many(:attachment, AttachmentValidator)
field(:replies_count, :integer, default: 0)
field(:like_count, :integer, default: 0)
field(:announcement_count, :integer, default: 0)
field(:inReplyTo, ObjectValidators.ObjectID)
field(:url, ObjectValidators.Uri)
field(:likes, {:array, ObjectValidators.ObjectID}, default: [])
field(:announcements, {:array, ObjectValidators.ObjectID}, default: [])
end
def cast_and_apply(data) do
data
|> cast_data
|> apply_action(:insert)
end
def cast_and_validate(data) do
data
|> cast_data()
|> validate_data()
end
def cast_data(data) do
%__MODULE__{}
|> changeset(data)
end
defp fix_url(%{"url" => url} = data) when is_list(url) do
attachment =
Enum.find(url, fn x ->
mime_type = x["mimeType"] || x["mediaType"] || ""
is_map(x) and String.starts_with?(mime_type, ["video/", "audio/"])
end)
link_element =
Enum.find(url, fn x ->
mime_type = x["mimeType"] || x["mediaType"] || ""
is_map(x) and mime_type == "text/html"
end)
data
|> Map.put("attachment", [attachment])
|> Map.put("url", link_element["href"])
end
defp fix_url(data), do: data
defp fix_content(%{"mediaType" => "text/markdown", "content" => content} = data)
when is_binary(content) do
content =
content
|> Earmark.as_html!(%Earmark.Options{renderer: EarmarkRenderer})
|> Pleroma.HTML.filter_tags()
Map.put(data, "content", content)
end
defp fix_content(data), do: data
defp fix(data) do
data
|> CommonFixes.fix_defaults()
|> CommonFixes.fix_attribution()
|> CommonFixes.fix_actor()
|> Transmogrifier.fix_emoji()
|> fix_url()
|> fix_content()
end
def changeset(struct, data) do
data = fix(data)
struct
|> cast(data, __schema__(:fields) -- [:attachment])
|> cast_embed(:attachment)
end
def validate_data(data_cng) do
data_cng
|> validate_inclusion(:type, ["Audio", "Video"])
|> validate_required([:id, :actor, :attributedTo, :type, :context, :attachment])
|> CommonValidations.validate_any_presence([:cc, :to])
|> CommonValidations.validate_fields_match([:actor, :attributedTo])
|> CommonValidations.validate_actor_presence()
|> CommonValidations.validate_host_match()
end
end

2
lib/pleroma/web/activity_pub/object_validators/chat_message_validator.ex
View file

@ -22,7 +22,7 @@ defmodule Pleroma.Web.ActivityPub.ObjectValidators.ChatMessageValidator do
field(:content, ObjectValidators.SafeText)
field(:actor, ObjectValidators.ObjectID)
field(:published, ObjectValidators.DateTime)
field(:emoji, :map, default: %{})
field(:emoji, ObjectValidators.Emoji, default: %{})
embeds_one(:attachment, AttachmentValidator)
end

13
lib/pleroma/web/activity_pub/object_validators/common_fixes.ex
View file

@ -3,6 +3,7 @@
# SPDX-License-Identifier: AGPL-3.0-only
defmodule Pleroma.Web.ActivityPub.ObjectValidators.CommonFixes do
alias Pleroma.Object.Containment
alias Pleroma.Web.ActivityPub.Utils
# based on Pleroma.Web.ActivityPub.Utils.lazy_put_objects_defaults
@ -11,12 +12,20 @@ defmodule Pleroma.Web.ActivityPub.ObjectValidators.CommonFixes do
Utils.create_context(data["context"] || data["conversation"])
data
|> Map.put_new("context", context)
|> Map.put_new("context_id", context_id)
|> Map.put("context", context)
|> Map.put("context_id", context_id)
end
def fix_attribution(data) do
data
|> Map.put_new("actor", data["attributedTo"])
end
def fix_actor(data) do
actor = Containment.get_actor(data)
data
|> Map.put("actor", actor)
|> Map.put("attributedTo", actor)
end
end

8
lib/pleroma/web/activity_pub/object_validators/create_generic_validator.ex
View file

@ -10,9 +10,10 @@ defmodule Pleroma.Web.ActivityPub.ObjectValidators.CreateGenericValidator do
alias Pleroma.EctoType.ActivityPub.ObjectValidators
alias Pleroma.Object
alias Pleroma.Web.ActivityPub.ObjectValidators.CommonFixes
alias Pleroma.Web.ActivityPub.ObjectValidators.CommonValidations
import Ecto.Changeset
import Pleroma.Web.ActivityPub.ObjectValidators.CommonValidations
@primary_key false
@ -75,14 +76,15 @@ defmodule Pleroma.Web.ActivityPub.ObjectValidators.CreateGenericValidator do
data
|> fix_context(meta)
|> fix_addressing(meta)
|> CommonFixes.fix_actor()
end
def validate_data(cng, meta \\ []) do
cng
|> validate_required([:actor, :type, :object])
|> validate_inclusion(:type, ["Create"])
|> validate_actor_presence()
|> validate_any_presence([:to, :cc])
|> CommonValidations.validate_actor_presence()
|> CommonValidations.validate_any_presence([:to, :cc])
|> validate_actors_match(meta)
|> validate_context_match(meta)
|> validate_object_nonexistence()

5
lib/pleroma/web/activity_pub/object_validators/event_validator.ex
View file

@ -9,6 +9,7 @@ defmodule Pleroma.Web.ActivityPub.ObjectValidators.EventValidator do
alias Pleroma.Web.ActivityPub.ObjectValidators.AttachmentValidator
alias Pleroma.Web.ActivityPub.ObjectValidators.CommonFixes
alias Pleroma.Web.ActivityPub.ObjectValidators.CommonValidations
alias Pleroma.Web.ActivityPub.Transmogrifier
import Ecto.Changeset
@ -39,8 +40,7 @@ defmodule Pleroma.Web.ActivityPub.ObjectValidators.EventValidator do
field(:attributedTo, ObjectValidators.ObjectID)
field(:published, ObjectValidators.DateTime)
# TODO: Write type
field(:emoji, :map, default: %{})
field(:emoji, ObjectValidators.Emoji, default: %{})
field(:sensitive, :boolean, default: false)
embeds_many(:attachment, AttachmentValidator)
field(:replies_count, :integer, default: 0)
@ -74,6 +74,7 @@ defmodule Pleroma.Web.ActivityPub.ObjectValidators.EventValidator do
data
|> CommonFixes.fix_defaults()
|> CommonFixes.fix_attribution()
|> Transmogrifier.fix_emoji()
end
def changeset(struct, data) do

66
lib/pleroma/web/activity_pub/object_validators/note_validator.ex
View file

@ -1,66 +0,0 @@
# Pleroma: A lightweight social networking server
# Copyright © 2017-2020 Pleroma Authors <https://pleroma.social/>
# SPDX-License-Identifier: AGPL-3.0-only
defmodule Pleroma.Web.ActivityPub.ObjectValidators.NoteValidator do
use Ecto.Schema
alias Pleroma.EctoType.ActivityPub.ObjectValidators
import Ecto.Changeset
@primary_key false
embedded_schema do
field(:id, ObjectValidators.ObjectID, primary_key: true)
field(:to, ObjectValidators.Recipients, default: [])
field(:cc, ObjectValidators.Recipients, default: [])
field(:bto, ObjectValidators.Recipients, default: [])
field(:bcc, ObjectValidators.Recipients, default: [])
# TODO: Write type
field(:tag, {:array, :map}, default: [])
field(:type, :string)
field(:name, :string)
field(:summary, :string)
field(:content, :string)
field(:context, :string)
# short identifier for PleromaFE to group statuses by context
field(:context_id, :integer)
field(:actor, ObjectValidators.ObjectID)
field(:attributedTo, ObjectValidators.ObjectID)
field(:published, ObjectValidators.DateTime)
# TODO: Write type
field(:emoji, :map, default: %{})
field(:sensitive, :boolean, default: false)
# TODO: Write type
field(:attachment, {:array, :map}, default: [])
field(:replies_count, :integer, default: 0)
field(:like_count, :integer, default: 0)
field(:announcement_count, :integer, default: 0)
field(:inReplyTo, ObjectValidators.ObjectID)
field(:url, ObjectValidators.Uri)
field(:likes, {:array, :string}, default: [])
field(:announcements, {:array, :string}, default: [])
end
def cast_and_validate(data) do
data
|> cast_data()
|> validate_data()
end
def cast_data(data) do
%__MODULE__{}
|> cast(data, __schema__(:fields))
end
def validate_data(data_cng) do
data_cng
|> validate_inclusion(:type, ["Note"])
|> validate_required([:id, :actor, :to, :cc, :type, :content, :context])
end
end

9
lib/pleroma/web/activity_pub/object_validators/question_validator.ex
View file

@ -10,6 +10,7 @@ defmodule Pleroma.Web.ActivityPub.ObjectValidators.QuestionValidator do
alias Pleroma.Web.ActivityPub.ObjectValidators.CommonFixes
alias Pleroma.Web.ActivityPub.ObjectValidators.CommonValidations
alias Pleroma.Web.ActivityPub.ObjectValidators.QuestionOptionsValidator
alias Pleroma.Web.ActivityPub.Transmogrifier
import Ecto.Changeset
@ -35,8 +36,7 @@ defmodule Pleroma.Web.ActivityPub.ObjectValidators.QuestionValidator do
field(:attributedTo, ObjectValidators.ObjectID)
field(:summary, :string)
field(:published, ObjectValidators.DateTime)
# TODO: Write type
field(:emoji, :map, default: %{})
field(:emoji, ObjectValidators.Emoji, default: %{})
field(:sensitive, :boolean, default: false)
embeds_many(:attachment, AttachmentValidator)
field(:replies_count, :integer, default: 0)
@ -47,8 +47,8 @@ defmodule Pleroma.Web.ActivityPub.ObjectValidators.QuestionValidator do
# short identifier for PleromaFE to group statuses by context
field(:context_id, :integer)
field(:likes, {:array, :string}, default: [])
field(:announcements, {:array, :string}, default: [])
field(:likes, {:array, ObjectValidators.ObjectID}, default: [])
field(:announcements, {:array, ObjectValidators.ObjectID}, default: [])
field(:closed, ObjectValidators.DateTime)
field(:voters, {:array, ObjectValidators.ObjectID}, default: [])
@ -85,6 +85,7 @@ defmodule Pleroma.Web.ActivityPub.ObjectValidators.QuestionValidator do
data
|> CommonFixes.fix_defaults()
|> CommonFixes.fix_attribution()
|> Transmogrifier.fix_emoji()
|> fix_closed()
end

24
lib/pleroma/web/activity_pub/object_validators/url_object_validator.ex
View file

@ -1,24 +0,0 @@
# Pleroma: A lightweight social networking server
# Copyright © 2017-2020 Pleroma Authors <https://pleroma.social/>
# SPDX-License-Identifier: AGPL-3.0-only
defmodule Pleroma.Web.ActivityPub.ObjectValidators.UrlObjectValidator do
use Ecto.Schema
alias Pleroma.EctoType.ActivityPub.ObjectValidators
import Ecto.Changeset
@primary_key false
embedded_schema do
field(:type, :string)
field(:href, ObjectValidators.Uri)
field(:mediaType, :string, default: "application/octet-stream")
end
def changeset(struct, data) do
struct
|> cast(data, __schema__(:fields))
|> validate_required([:type, :href, :mediaType])
end
end

34
lib/pleroma/web/activity_pub/publisher.ex
View file

@ -13,6 +13,7 @@ defmodule Pleroma.Web.ActivityPub.Publisher do
alias Pleroma.User
alias Pleroma.Web.ActivityPub.Relay
alias Pleroma.Web.ActivityPub.Transmogrifier
alias Pleroma.Web.FedSockets
require Pleroma.Constants
@ -50,15 +51,35 @@ defmodule Pleroma.Web.ActivityPub.Publisher do
def publish_one(%{inbox: inbox, json: json, actor: %User{} = actor, id: id} = params) do
Logger.debug("Federating #{id} to #{inbox}")
uri = URI.parse(inbox)
case FedSockets.get_or_create_fed_socket(inbox) do
{:ok, fedsocket} ->
Logger.debug("publishing via fedsockets - #{inspect(inbox)}")
FedSockets.publish(fedsocket, json)
_ ->
Logger.debug("publishing via http - #{inspect(inbox)}")
http_publish(inbox, actor, json, params)
end
end
def publish_one(%{actor_id: actor_id} = params) do
actor = User.get_cached_by_id(actor_id)
params
|> Map.delete(:actor_id)
|> Map.put(:actor, actor)
|> publish_one()
end
defp http_publish(inbox, actor, json, params) do
uri = %{path: path} = URI.parse(inbox)
digest = "SHA-256=" <> (:crypto.hash(:sha256, json) |> Base.encode64())
date = Pleroma.Signature.signed_date()
signature =
Pleroma.Signature.sign(actor, %{
"(request-target)": "post #{uri.path}",
"(request-target)": "post #{path}",
host: signature_host(uri),
"content-length": byte_size(json),
digest: digest,
@ -89,15 +110,6 @@ defmodule Pleroma.Web.ActivityPub.Publisher do
end
end
def publish_one(%{actor_id: actor_id} = params) do
actor = User.get_cached_by_id(actor_id)
params
|> Map.delete(:actor_id)
|> Map.put(:actor, actor)
|> publish_one()
end
defp signature_host(%URI{port: port, scheme: scheme, host: host}) do
if port == URI.default_port(scheme) do
host

7
lib/pleroma/web/activity_pub/side_effects.ex
View file

@ -7,7 +7,6 @@ defmodule Pleroma.Web.ActivityPub.SideEffects do
"""
alias Pleroma.Activity
alias Pleroma.Activity.Ir.Topics
alias Pleroma.ActivityExpiration
alias Pleroma.Chat
alias Pleroma.Chat.MessageReference
alias Pleroma.FollowingRelationship
@ -188,10 +187,6 @@ defmodule Pleroma.Web.ActivityPub.SideEffects do
Object.increase_replies_count(in_reply_to)
end
if expires_at = activity.data["expires_at"] do
ActivityExpiration.create(activity, expires_at)
end
BackgroundWorker.enqueue("fetch_data_for_activity", %{"activity_id" => activity.id})
meta =
@ -341,7 +336,7 @@ defmodule Pleroma.Web.ActivityPub.SideEffects do
end
def handle_object_creation(%{"type" => objtype} = object, meta)
when objtype in ~w[Audio Question Event] do
when objtype in ~w[Audio Video Question Event Article] do
with {:ok, object, meta} <- Pipeline.common_pipeline(object, meta) do
{:ok, object, meta}
end

48
lib/pleroma/web/activity_pub/transmogrifier.ex
View file

@ -7,7 +7,6 @@ defmodule Pleroma.Web.ActivityPub.Transmogrifier do
A module to handle coding from internal to wire ActivityPub and back.
"""
alias Pleroma.Activity
alias Pleroma.EarmarkRenderer
alias Pleroma.EctoType.ActivityPub.ObjectValidators
alias Pleroma.Maps
alias Pleroma.Object
@ -45,7 +44,6 @@ defmodule Pleroma.Web.ActivityPub.Transmogrifier do
|> fix_addressing
|> fix_summary
|> fix_type(options)
|> fix_content
end
def fix_summary(%{"summary" => nil} = object) do
@ -168,7 +166,6 @@ defmodule Pleroma.Web.ActivityPub.Transmogrifier do
def fix_in_reply_to(%{"inReplyTo" => in_reply_to} = object, options)
when not is_nil(in_reply_to) do
in_reply_to_id = prepare_in_reply_to(in_reply_to)
object = Map.put(object, "inReplyToAtomUri", in_reply_to_id)
depth = (options[:depth] || 0) + 1
if Federator.allowed_thread_distance?(depth) do
@ -176,9 +173,8 @@ defmodule Pleroma.Web.ActivityPub.Transmogrifier do
%Activity{} <- Activity.get_create_by_object_ap_id(replied_object.data["id"]) do
object
|> Map.put("inReplyTo", replied_object.data["id"])
|> Map.put("inReplyToAtomUri", object["inReplyToAtomUri"] || in_reply_to_id)
|> Map.put("context", replied_object.data["context"] || object["conversation"])
|> Map.drop(["conversation"])
|> Map.drop(["conversation", "inReplyToAtomUri"])
else
e ->
Logger.warn("Couldn't fetch #{inspect(in_reply_to_id)}, error: #{inspect(e)}")
@ -276,24 +272,7 @@ defmodule Pleroma.Web.ActivityPub.Transmogrifier do
Map.put(object, "url", url["href"])
end
def fix_url(%{"type" => "Video", "url" => url} = object) when is_list(url) do
attachment =
Enum.find(url, fn x ->
media_type = x["mediaType"] || x["mimeType"] || ""
is_map(x) and String.starts_with?(media_type, "video/")
end)
link_element =
Enum.find(url, fn x -> is_map(x) and (x["mediaType"] || x["mimeType"]) == "text/html" end)
object
|> Map.put("attachment", [attachment])
|> Map.put("url", link_element["href"])
end
def fix_url(%{"type" => object_type, "url" => url} = object)
when object_type != "Video" and is_list(url) do
def fix_url(%{"url" => url} = object) when is_list(url) do
first_element = Enum.at(url, 0)
url_string =
@ -318,9 +297,6 @@ defmodule Pleroma.Web.ActivityPub.Transmogrifier do
Map.put(mapping, name, data["icon"]["url"])
end)
# we merge mastodon and pleroma emoji into a single mapping, to allow for both wire formats
emoji = Map.merge(object["emoji"] || %{}, emoji)
Map.put(object, "emoji", emoji)
end
@ -376,18 +352,6 @@ defmodule Pleroma.Web.ActivityPub.Transmogrifier do
def fix_type(object, _), do: object
defp fix_content(%{"mediaType" => "text/markdown", "content" => content} = object)
when is_binary(content) do
html_content =
content
|> Earmark.as_html!(%Earmark.Options{renderer: EarmarkRenderer})
|> Pleroma.HTML.filter_tags()
Map.merge(object, %{"content" => html_content, "mediaType" => "text/html"})
end
defp fix_content(object), do: object
# Reduce the object list to find the reported user.
defp get_reported(objects) do
Enum.reduce_while(objects, nil, fn ap_id, _ ->
@ -460,7 +424,7 @@ defmodule Pleroma.Web.ActivityPub.Transmogrifier do
%{"type" => "Create", "object" => %{"type" => objtype} = object} = data,
options
)
when objtype in ~w{Article Note Video Page} do
when objtype in ~w{Note Page} do
actor = Containment.get_actor(data)
with nil <- Activity.get_create_by_object_ap_id(object["id"]),
@ -554,7 +518,9 @@ defmodule Pleroma.Web.ActivityPub.Transmogrifier do
%{"type" => "Create", "object" => %{"type" => objtype}} = data,
_options
)
when objtype in ~w{Question Answer ChatMessage Audio Event} do
when objtype in ~w{Question Answer ChatMessage Audio Video Event Article} do
data = Map.put(data, "object", strip_internal_fields(data["object"]))
with {:ok, %User{}} <- ObjectValidator.fetch_actor(data),
{:ok, activity, _} <- Pipeline.common_pipeline(data, local: false) do
{:ok, activity}
@ -1034,7 +1000,7 @@ defmodule Pleroma.Web.ActivityPub.Transmogrifier do
def upgrade_user_from_ap_id(ap_id) do
with %User{local: false} = user <- User.get_cached_by_ap_id(ap_id),
{:ok, data} <- ActivityPub.fetch_and_prepare_user_from_ap_id(ap_id),
{:ok, data} <- ActivityPub.fetch_and_prepare_user_from_ap_id(ap_id, force_http: true),
{:ok, user} <- update_user(user, data) do
TransmogrifierWorker.enqueue("user_upgrade", %{"user_id" => user.id})
{:ok, user}

22
lib/pleroma/web/admin_api/controllers/admin_api_controller.ex
View file

@ -23,8 +23,6 @@ defmodule Pleroma.Web.AdminAPI.AdminAPIController do
alias Pleroma.Web.Endpoint
alias Pleroma.Web.Router
require Logger
@users_page_size 50
plug(
@ -68,6 +66,12 @@ defmodule Pleroma.Web.AdminAPI.AdminAPIController do
when action in [:list_user_statuses, :list_instance_statuses]
)
plug(
OAuthScopesPlug,
%{scopes: ["read:chats"], admin: true}
when action in [:list_user_chats]
)
plug(
OAuthScopesPlug,
%{scopes: ["read"], admin: true}
@ -256,6 +260,20 @@ defmodule Pleroma.Web.AdminAPI.AdminAPIController do
end
end
def list_user_chats(%{assigns: %{user: admin}} = conn, %{"nickname" => nickname} = _params) do
with %User{id: user_id} <- User.get_cached_by_nickname_or_id(nickname, for: admin) do
chats =
Pleroma.Chat.for_user_query(user_id)
|> Pleroma.Repo.all()
conn
|> put_view(AdminAPI.ChatView)
|> render("index.json", chats: chats)
else
_ -> {:error, :not_found}
end
end
def user_toggle_activation(%{assigns: %{user: admin}} = conn, %{"nickname" => nickname}) do
user = User.get_cached_by_nickname(nickname)

85
lib/pleroma/web/admin_api/controllers/chat_controller.ex Normal file
View file

@ -0,0 +1,85 @@
# Pleroma: A lightweight social networking server
# Copyright © 2017-2020 Pleroma Authors <https://pleroma.social/>
# SPDX-License-Identifier: AGPL-3.0-only
defmodule Pleroma.Web.AdminAPI.ChatController do
use Pleroma.Web, :controller
alias Pleroma.Activity
alias Pleroma.Chat
alias Pleroma.Chat.MessageReference
alias Pleroma.ModerationLog
alias Pleroma.Pagination
alias Pleroma.Plugs.OAuthScopesPlug
alias Pleroma.Web.AdminAPI
alias Pleroma.Web.CommonAPI
alias Pleroma.Web.PleromaAPI.Chat.MessageReferenceView
require Logger
plug(Pleroma.Web.ApiSpec.CastAndValidate)
plug(
OAuthScopesPlug,
%{scopes: ["read:chats"], admin: true} when action in [:show, :messages]
)
plug(
OAuthScopesPlug,
%{scopes: ["write:chats"], admin: true} when action in [:delete_message]
)
action_fallback(Pleroma.Web.AdminAPI.FallbackController)
defdelegate open_api_operation(action), to: Pleroma.Web.ApiSpec.Admin.ChatOperation
def delete_message(%{assigns: %{user: user}} = conn, %{
message_id: message_id,
id: chat_id
}) do
with %MessageReference{object: %{data: %{"id" => object_ap_id}}} = cm_ref <-
MessageReference.get_by_id(message_id),
^chat_id <- to_string(cm_ref.chat_id),
%Activity{id: activity_id} <- Activity.get_create_by_object_ap_id(object_ap_id),
{:ok, _} <- CommonAPI.delete(activity_id, user) do
ModerationLog.insert_log(%{
action: "chat_message_delete",
actor: user,
subject_id: message_id
})
conn
|> put_view(MessageReferenceView)
|> render("show.json", chat_message_reference: cm_ref)
else
_e ->
{:error, :could_not_delete}
end
end
def messages(conn, %{id: id} = params) do
with %Chat{} = chat <- Chat.get_by_id(id) do
cm_refs =
chat
|> MessageReference.for_chat_query()
|> Pagination.fetch_paginated(params)
conn
|> put_view(MessageReferenceView)
|> render("index.json", chat_message_references: cm_refs)
else
_ ->
conn
|> put_status(:not_found)
|> json(%{error: "not found"})
end
end
def show(conn, %{id: id}) do
with %Chat{} = chat <- Chat.get_by_id(id) do
conn
|> put_view(AdminAPI.ChatView)
|> render("show.json", chat: chat)
end
end
end

41
lib/pleroma/web/admin_api/controllers/instance_document_controller.ex Normal file
View file

@ -0,0 +1,41 @@
# Pleroma: A lightweight social networking server
# Copyright © 2017-2020 Pleroma Authors <https://pleroma.social/>
# SPDX-License-Identifier: AGPL-3.0-only
defmodule Pleroma.Web.AdminAPI.InstanceDocumentController do
use Pleroma.Web, :controller
alias Pleroma.Plugs.InstanceStatic
alias Pleroma.Plugs.OAuthScopesPlug
alias Pleroma.Web.InstanceDocument
plug(Pleroma.Web.ApiSpec.CastAndValidate)
action_fallback(Pleroma.Web.AdminAPI.FallbackController)
defdelegate open_api_operation(action), to: Pleroma.Web.ApiSpec.Admin.InstanceDocumentOperation
plug(OAuthScopesPlug, %{scopes: ["read"], admin: true} when action == :show)
plug(OAuthScopesPlug, %{scopes: ["write"], admin: true} when action in [:update, :delete])
def show(conn, %{name: document_name}) do
with {:ok, url} <- InstanceDocument.get(document_name),
{:ok, content} <- File.read(InstanceStatic.file_path(url)) do
conn
|> put_resp_content_type("text/html")
|> send_resp(200, content)
end
end
def update(%{body_params: %{file: file}} = conn, %{name: document_name}) do
with {:ok, url} <- InstanceDocument.put(document_name, file.path) do
json(conn, %{"url" => url})
end
end
def delete(conn, %{name: document_name}) do
with :ok <- InstanceDocument.delete(document_name) do
json(conn, %{})
end
end
end

30
lib/pleroma/web/admin_api/views/chat_view.ex Normal file
View file

@ -0,0 +1,30 @@
# Pleroma: A lightweight social networking server
# Copyright © 2017-2020 Pleroma Authors <https://pleroma.social/>
# SPDX-License-Identifier: AGPL-3.0-only
defmodule Pleroma.Web.AdminAPI.ChatView do
use Pleroma.Web, :view
alias Pleroma.Chat
alias Pleroma.User
alias Pleroma.Web.MastodonAPI
alias Pleroma.Web.PleromaAPI
def render("index.json", %{chats: chats} = opts) do
render_many(chats, __MODULE__, "show.json", Map.delete(opts, :chats))
end
def render("show.json", %{chat: %Chat{user_id: user_id}} = opts) do
user = User.get_by_id(user_id)
sender = MastodonAPI.AccountView.render("show.json", user: user, skip_visibility_check: true)
serialized_chat = PleromaAPI.ChatView.render("show.json", opts)
serialized_chat
|> Map.put(:sender, sender)
|> Map.put(:receiver, serialized_chat[:account])
|> Map.delete(:account)
end
def render(view, opts), do: PleromaAPI.ChatView.render(view, opts)
end

3
lib/pleroma/web/admin_api/views/status_view.ex
View file

@ -8,6 +8,7 @@ defmodule Pleroma.Web.AdminAPI.StatusView do
require Pleroma.Constants
alias Pleroma.Web.AdminAPI
alias Pleroma.Web.CommonAPI
alias Pleroma.Web.MastodonAPI
defdelegate merge_account_views(user), to: AdminAPI.AccountView
@ -17,7 +18,7 @@ defmodule Pleroma.Web.AdminAPI.StatusView do
end
def render("show.json", %{activity: %{data: %{"object" => _object}} = activity} = opts) do
user = MastodonAPI.StatusView.get_user(activity.data["actor"])
user = CommonAPI.get_user(activity.data["actor"])
MastodonAPI.StatusView.render("show.json", opts)
|> Map.merge(%{account: merge_account_views(user)})

13
lib/pleroma/web/api_spec.ex
View file

@ -13,10 +13,15 @@ defmodule Pleroma.Web.ApiSpec do
@impl OpenApi
def spec do
%OpenApi{
servers: [
# Populate the Server info from a phoenix endpoint
OpenApiSpex.Server.from_endpoint(Endpoint)
],
servers:
if Phoenix.Endpoint.server?(:pleroma, Endpoint) do
[
# Populate the Server info from a phoenix endpoint
OpenApiSpex.Server.from_endpoint(Endpoint)
]
else
[]
end,
info: %OpenApiSpex.Info{
title: "Pleroma",
description: Application.spec(:pleroma, :description) |> to_string(),

6
lib/pleroma/web/api_spec/helpers.ex
View file

@ -72,7 +72,11 @@ defmodule Pleroma.Web.ApiSpec.Helpers do
end
def empty_array_response do
Operation.response("Empty array", "application/json", %Schema{type: :array, example: []})
Operation.response("Empty array", "application/json", %Schema{
type: :array,
items: %Schema{type: :object, example: %{}},
example: []
})
end
def no_content_response do

7
lib/pleroma/web/api_spec/operations/account_operation.ex
View file

@ -372,6 +372,10 @@ defmodule Pleroma.Web.ApiSpec.AccountOperation do
tags: ["accounts"],
summary: "Identity proofs",
operationId: "AccountController.identity_proofs",
# Validators complains about unused path params otherwise
parameters: [
%Reference{"$ref": "#/components/parameters/accountIdOrNickname"}
],
description: "Not implemented",
responses: %{
200 => empty_array_response()
@ -469,7 +473,6 @@ defmodule Pleroma.Web.ApiSpec.AccountOperation do
identifier: %Schema{type: :string},
message: %Schema{type: :string}
},
required: [],
# Note: example of successful registration with failed login response:
# example: %{
# "identifier" => "missing_confirmed_email",
@ -530,7 +533,7 @@ defmodule Pleroma.Web.ApiSpec.AccountOperation do
nullable: true,
oneOf: [
%Schema{type: :array, items: attribute_field()},
%Schema{type: :object, additionalProperties: %Schema{type: attribute_field()}}
%Schema{type: :object, additionalProperties: attribute_field()}
]
},
# NOTE: `source` field is not supported

96
lib/pleroma/web/api_spec/operations/admin/chat_operation.ex Normal file
View file

@ -0,0 +1,96 @@
# Pleroma: A lightweight social networking server
# Copyright © 2017-2020 Pleroma Authors <https://pleroma.social/>
# SPDX-License-Identifier: AGPL-3.0-only
defmodule Pleroma.Web.ApiSpec.Admin.ChatOperation do
alias OpenApiSpex.Operation
alias Pleroma.Web.ApiSpec.Schemas.Chat
alias Pleroma.Web.ApiSpec.Schemas.ChatMessage
import Pleroma.Web.ApiSpec.Helpers
def open_api_operation(action) do
operation = String.to_existing_atom("#{action}_operation")
apply(__MODULE__, operation, [])
end
def delete_message_operation do
%Operation{
tags: ["admin", "chat"],
summary: "Delete an individual chat message",
operationId: "AdminAPI.ChatController.delete_message",
parameters: [
Operation.parameter(:id, :path, :string, "The ID of the Chat"),
Operation.parameter(:message_id, :path, :string, "The ID of the message")
],
responses: %{
200 =>
Operation.response(
"The deleted ChatMessage",
"application/json",
ChatMessage
)
},
security: [
%{
"oAuth" => ["write:chats"]
}
]
}
end
def messages_operation do
%Operation{
tags: ["admin", "chat"],
summary: "Get the most recent messages of the chat",
operationId: "AdminAPI.ChatController.messages",
parameters:
[Operation.parameter(:id, :path, :string, "The ID of the Chat")] ++
pagination_params(),
responses: %{
200 =>
Operation.response(
"The messages in the chat",
"application/json",
Pleroma.Web.ApiSpec.ChatOperation.chat_messages_response()
)
},
security: [
%{
"oAuth" => ["read:chats"]
}
]
}
end
def show_operation do
%Operation{
tags: ["chat"],
summary: "Create a chat",
operationId: "AdminAPI.ChatController.show",
parameters: [
Operation.parameter(
:id,
:path,
:string,
"The id of the chat",
required: true,
example: "1234"
)
],
responses: %{
200 =>
Operation.response(
"The existing chat",
"application/json",
Chat
)
},
security: [
%{
"oAuth" => ["read"]
}
]
}
end
end

115
lib/pleroma/web/api_spec/operations/admin/instance_document_operation.ex Normal file
View file

@ -0,0 +1,115 @@
# Pleroma: A lightweight social networking server
# Copyright © 2017-2020 Pleroma Authors <https://pleroma.social/>
# SPDX-License-Identifier: AGPL-3.0-only
defmodule Pleroma.Web.ApiSpec.Admin.InstanceDocumentOperation do
alias OpenApiSpex.Operation
alias OpenApiSpex.Schema
alias Pleroma.Web.ApiSpec.Helpers
alias Pleroma.Web.ApiSpec.Schemas.ApiError
def open_api_operation(action) do
operation = String.to_existing_atom("#{action}_operation")
apply(__MODULE__, operation, [])
end
def show_operation do
%Operation{
tags: ["Admin", "InstanceDocument"],
summary: "Get the instance document",
operationId: "AdminAPI.InstanceDocumentController.show",
security: [%{"oAuth" => ["read"]}],
parameters: [
Operation.parameter(:name, :path, %Schema{type: :string}, "The document name",
required: true
)
| Helpers.admin_api_params()
],
responses: %{
200 => document_content(),
400 => Operation.response("Bad Request", "application/json", ApiError),
403 => Operation.response("Forbidden", "application/json", ApiError),
404 => Operation.response("Not Found", "application/json", ApiError)
}
}
end
def update_operation do
%Operation{
tags: ["Admin", "InstanceDocument"],
summary: "Update the instance document",
operationId: "AdminAPI.InstanceDocumentController.update",
security: [%{"oAuth" => ["write"]}],
requestBody: Helpers.request_body("Parameters", update_request()),
parameters: [
Operation.parameter(:name, :path, %Schema{type: :string}, "The document name",
required: true
)
| Helpers.admin_api_params()
],
responses: %{
200 => Operation.response("InstanceDocument", "application/json", instance_document()),
400 => Operation.response("Bad Request", "application/json", ApiError),
403 => Operation.response("Forbidden", "application/json", ApiError),
404 => Operation.response("Not Found", "application/json", ApiError)
}
}
end
defp update_request do
%Schema{
title: "UpdateRequest",
description: "POST body for uploading the file",
type: :object,
required: [:file],
properties: %{
file: %Schema{
type: :string,
format: :binary,
description: "The file to be uploaded, using multipart form data."
}
}
}
end
def delete_operation do
%Operation{
tags: ["Admin", "InstanceDocument"],
summary: "Get the instance document",
operationId: "AdminAPI.InstanceDocumentController.delete",
security: [%{"oAuth" => ["write"]}],
parameters: [
Operation.parameter(:name, :path, %Schema{type: :string}, "The document name",
required: true
)
| Helpers.admin_api_params()
],
responses: %{
200 => Operation.response("InstanceDocument", "application/json", instance_document()),
400 => Operation.response("Bad Request", "application/json", ApiError),
403 => Operation.response("Forbidden", "application/json", ApiError),
404 => Operation.response("Not Found", "application/json", ApiError)
}
}
end
defp instance_document do
%Schema{
title: "InstanceDocument",
type: :object,
properties: %{
url: %Schema{type: :string}
},
example: %{
"url" => "https://example.com/static/terms-of-service.html"
}
}
end
defp document_content do
Operation.response("InstanceDocumentContent", "text/html", %Schema{
type: :string,
example: "<h1>Instance panel</h1>"
})
end
end

2
lib/pleroma/web/api_spec/operations/custom_emoji_operation.ex
View file

@ -69,7 +69,7 @@ defmodule Pleroma.Web.ApiSpec.CustomEmojiOperation do
type: :object,
properties: %{
category: %Schema{type: :string},
tags: %Schema{type: :array}
tags: %Schema{type: :array, items: %Schema{type: :string}}
}
}
],

2
lib/pleroma/web/api_spec/operations/emoji_reaction_operation.ex
View file

@ -23,7 +23,7 @@ defmodule Pleroma.Web.ApiSpec.EmojiReactionOperation do
parameters: [
Operation.parameter(:id, :path, FlakeID, "Status ID", required: true),
Operation.parameter(:emoji, :path, :string, "Filter by a single unicode emoji",
required: false
required: nil
)
],
security: [%{"oAuth" => ["read:statuses"]}],

21
lib/pleroma/web/api_spec/operations/list_operation.ex
View file

@ -114,7 +114,7 @@ defmodule Pleroma.Web.ApiSpec.ListOperation do
description: "Add accounts to the given list.",
operationId: "ListController.add_to_list",
parameters: [id_param()],
requestBody: add_remove_accounts_request(),
requestBody: add_remove_accounts_request(true),
security: [%{"oAuth" => ["write:lists"]}],
responses: %{
200 => Operation.response("Empty object", "application/json", %Schema{type: :object})
@ -127,8 +127,16 @@ defmodule Pleroma.Web.ApiSpec.ListOperation do
tags: ["Lists"],
summary: "Remove accounts from list",
operationId: "ListController.remove_from_list",
parameters: [id_param()],
requestBody: add_remove_accounts_request(),
parameters: [
id_param(),
Operation.parameter(
:account_ids,
:query,
%Schema{type: :array, items: %Schema{type: :string}},
"Array of account IDs"
)
],
requestBody: add_remove_accounts_request(false),
security: [%{"oAuth" => ["write:lists"]}],
responses: %{
200 => Operation.response("Empty object", "application/json", %Schema{type: :object})
@ -171,7 +179,7 @@ defmodule Pleroma.Web.ApiSpec.ListOperation do
)
end
defp add_remove_accounts_request do
defp add_remove_accounts_request(required) when is_boolean(required) do
request_body(
"Parameters",
%Schema{
@ -179,10 +187,9 @@ defmodule Pleroma.Web.ApiSpec.ListOperation do
type: :object,
properties: %{
account_ids: %Schema{type: :array, description: "Array of account IDs", items: FlakeID}
},
required: [:account_ids]
}
},
required: true
required: required
)
end
end

80
lib/pleroma/web/api_spec/operations/user_import_operation.ex Normal file
View file

@ -0,0 +1,80 @@
# Pleroma: A lightweight social networking server
# Copyright © 2017-2020 Pleroma Authors <https://pleroma.social/>
# SPDX-License-Identifier: AGPL-3.0-only
defmodule Pleroma.Web.ApiSpec.UserImportOperation do
alias OpenApiSpex.Operation
alias OpenApiSpex.Schema
alias Pleroma.Web.ApiSpec.Schemas.ApiError
import Pleroma.Web.ApiSpec.Helpers
@spec open_api_operation(atom) :: Operation.t()
def open_api_operation(action) do
operation = String.to_existing_atom("#{action}_operation")
apply(__MODULE__, operation, [])
end
def follow_operation do
%Operation{
tags: ["follow_import"],
summary: "Imports your follows.",
operationId: "UserImportController.follow",
requestBody: request_body("Parameters", import_request(), required: true),
responses: %{
200 => ok_response(),
500 => Operation.response("Error", "application/json", ApiError)
},
security: [%{"oAuth" => ["write:follow"]}]
}
end
def blocks_operation do
%Operation{
tags: ["blocks_import"],
summary: "Imports your blocks.",
operationId: "UserImportController.blocks",
requestBody: request_body("Parameters", import_request(), required: true),
responses: %{
200 => ok_response(),
500 => Operation.response("Error", "application/json", ApiError)
},
security: [%{"oAuth" => ["write:blocks"]}]
}
end
def mutes_operation do
%Operation{
tags: ["mutes_import"],
summary: "Imports your mutes.",
operationId: "UserImportController.mutes",
requestBody: request_body("Parameters", import_request(), required: true),
responses: %{
200 => ok_response(),
500 => Operation.response("Error", "application/json", ApiError)
},
security: [%{"oAuth" => ["write:mutes"]}]
}
end
defp import_request do
%Schema{
type: :object,
required: [:list],
properties: %{
list: %Schema{
description:
"STRING or FILE containing a whitespace-separated list of accounts to import.",
anyOf: [
%Schema{type: :string, format: :binary},
%Schema{type: :string}
]
}
}
}
end
defp ok_response do
Operation.response("Ok", "application/json", %Schema{type: :string, example: "ok"})
end
end

3
lib/pleroma/web/api_spec/schemas/chat_message.ex
View file

@ -4,6 +4,7 @@
defmodule Pleroma.Web.ApiSpec.Schemas.ChatMessage do
alias OpenApiSpex.Schema
alias Pleroma.Web.ApiSpec.Schemas.Emoji
require OpenApiSpex
@ -18,7 +19,7 @@ defmodule Pleroma.Web.ApiSpec.Schemas.ChatMessage do
chat_id: %Schema{type: :string},
content: %Schema{type: :string, nullable: true},
created_at: %Schema{type: :string, format: :"date-time"},
emojis: %Schema{type: :array},
emojis: %Schema{type: :array, items: Emoji},
attachment: %Schema{type: :object, nullable: true},
card: %Schema{
type: :object,

4
lib/pleroma/web/api_spec/schemas/scheduled_status.ex
View file

@ -27,9 +27,9 @@ defmodule Pleroma.Web.ApiSpec.Schemas.ScheduledStatus do
media_ids: %Schema{type: :array, nullable: true, items: %Schema{type: :string}},
sensitive: %Schema{type: :boolean, nullable: true},
spoiler_text: %Schema{type: :string, nullable: true},
visibility: %Schema{type: VisibilityScope, nullable: true},
visibility: %Schema{allOf: [VisibilityScope], nullable: true},
scheduled_at: %Schema{type: :string, format: :"date-time", nullable: true},
poll: %Schema{type: Poll, nullable: true},
poll: %Schema{allOf: [Poll], nullable: true},
in_reply_to_id: %Schema{type: :string, nullable: true}
}
}

2
lib/pleroma/web/auth/pleroma_authenticator.ex
View file

@ -68,7 +68,7 @@ defmodule Pleroma.Web.Auth.PleromaAuthenticator do
nickname = value([registration_attrs["nickname"], Registration.nickname(registration)])
email = value([registration_attrs["email"], Registration.email(registration)])
name = value([registration_attrs["name"], Registration.name(registration)]) || nickname
bio = value([registration_attrs["bio"], Registration.description(registration)])
bio = value([registration_attrs["bio"], Registration.description(registration)]) || ""
random_password = :crypto.strong_rand_bytes(64) |> Base.encode64()

2
lib/pleroma/web/common_api/activity_draft.ex
View file

@ -202,7 +202,7 @@ defmodule Pleroma.Web.CommonAPI.ActivityDraft do
additional =
case draft.expires_at do
%NaiveDateTime{} = expires_at -> Map.put(additional, "expires_at", expires_at)
%DateTime{} = expires_at -> Map.put(additional, "expires_at", expires_at)
_ -> additional
end

25
lib/pleroma/web/common_api/common_api.ex
View file

@ -4,7 +4,6 @@
defmodule Pleroma.Web.CommonAPI do
alias Pleroma.Activity
alias Pleroma.ActivityExpiration
alias Pleroma.Conversation.Participation
alias Pleroma.Formatter
alias Pleroma.Object
@ -384,9 +383,9 @@ defmodule Pleroma.Web.CommonAPI do
def check_expiry_date({:ok, nil} = res), do: res
def check_expiry_date({:ok, in_seconds}) do
expiry = NaiveDateTime.utc_now() |> NaiveDateTime.add(in_seconds)
expiry = DateTime.add(DateTime.utc_now(), in_seconds)
if ActivityExpiration.expires_late_enough?(expiry) do
if Pleroma.Workers.PurgeExpiredActivity.expires_late_enough?(expiry) do
{:ok, expiry}
else
{:error, "Expiry date is too soon"}
@ -455,7 +454,8 @@ defmodule Pleroma.Web.CommonAPI do
end
def add_mute(user, activity) do
with {:ok, _} <- ThreadMute.add_mute(user.id, activity.data["context"]) do
with {:ok, _} <- ThreadMute.add_mute(user.id, activity.data["context"]),
_ <- Pleroma.Notification.mark_context_as_read(user, activity.data["context"]) do
{:ok, activity}
else
{:error, _} -> {:error, dgettext("errors", "conversation is already muted")}
@ -553,4 +553,21 @@ defmodule Pleroma.Web.CommonAPI do
def show_reblogs(%User{} = user, %User{} = target) do
UserRelationship.delete_reblog_mute(user, target)
end
def get_user(ap_id, fake_record_fallback \\ true) do
cond do
user = User.get_cached_by_ap_id(ap_id) ->
user
user = User.get_by_guessed_nickname(ap_id) ->
user
fake_record_fallback ->
# TODO: refactor (fake records is never a good idea)
User.error_user(ap_id)
true ->
nil
end
end
end

185
lib/pleroma/web/fed_sockets/fed_registry.ex Normal file
View file

@ -0,0 +1,185 @@
# Pleroma: A lightweight social networking server
# Copyright © 2017-2020 Pleroma Authors <https://pleroma.social/>
# SPDX-License-Identifier: AGPL-3.0-only
defmodule Pleroma.Web.FedSockets.FedRegistry do
@moduledoc """
The FedRegistry stores the active FedSockets for quick retrieval.
The storage and retrieval portion of the FedRegistry is done in process through
elixir's `Registry` module for speed and its ability to monitor for terminated processes.
Dropped connections will be caught by `Registry` and deleted. Since the next
message will initiate a new connection there is no reason to try and reconnect at that point.
Normally outside modules should have no need to call or use the FedRegistry themselves.
"""
alias Pleroma.Web.FedSockets.FedSocket
alias Pleroma.Web.FedSockets.SocketInfo
require Logger
@default_rejection_duration 15 * 60 * 1000
@rejections :fed_socket_rejections
@doc """
Retrieves a FedSocket from the Registry given it's origin.
The origin is expected to be a string identifying the endpoint "example.com" or "example2.com:8080"
Will return:
* {:ok, fed_socket} for working FedSockets
* {:error, :rejected} for origins that have been tried and refused within the rejection duration interval
* {:error, some_reason} usually :missing for unknown origins
"""
def get_fed_socket(origin) do
case get_registry_data(origin) do
{:error, reason} ->
{:error, reason}
{:ok, %{state: :connected} = socket_info} ->
{:ok, socket_info}
end
end
@doc """
Adds a connected FedSocket to the Registry.
Always returns {:ok, fed_socket}
"""
def add_fed_socket(origin, pid \\ nil) do
origin
|> SocketInfo.build(pid)
|> SocketInfo.connect()
|> add_socket_info
end
defp add_socket_info(%{origin: origin, state: :connected} = socket_info) do
case Registry.register(FedSockets.Registry, origin, socket_info) do
{:ok, _owner} ->
clear_prior_rejection(origin)
Logger.debug("fedsocket added: #{inspect(origin)}")
{:ok, socket_info}
{:error, {:already_registered, _pid}} ->
FedSocket.close(socket_info)
existing_socket_info = Registry.lookup(FedSockets.Registry, origin)
{:ok, existing_socket_info}
_ ->
{:error, :error_adding_socket}
end
end
@doc """
Mark this origin as having rejected a connection attempt.
This will keep it from getting additional connection attempts
for a period of time specified in the config.
Always returns {:ok, new_reg_data}
"""
def set_host_rejected(uri) do
new_reg_data =
uri
|> SocketInfo.origin()
|> get_or_create_registry_data()
|> set_to_rejected()
|> save_registry_data()
{:ok, new_reg_data}
end
@doc """
Retrieves the FedRegistryData from the Registry given it's origin.
The origin is expected to be a string identifying the endpoint "example.com" or "example2.com:8080"
Will return:
* {:ok, fed_registry_data} for known origins
* {:error, :missing} for uniknown origins
* {:error, :cache_error} indicating some low level runtime issues
"""
def get_registry_data(origin) do
case Registry.lookup(FedSockets.Registry, origin) do
[] ->
if is_rejected?(origin) do
Logger.debug("previously rejected fedsocket requested")
{:error, :rejected}
else
{:error, :missing}
end
[{_pid, %{state: :connected} = socket_info}] ->
{:ok, socket_info}
_ ->
{:error, :cache_error}
end
end
@doc """
Retrieves a map of all sockets from the Registry. The keys are the origins and the values are the corresponding SocketInfo
"""
def list_all do
(list_all_connected() ++ list_all_rejected())
|> Enum.into(%{})
end
defp list_all_connected do
FedSockets.Registry
|> Registry.select([{{:"$1", :_, :"$3"}, [], [{{:"$1", :"$3"}}]}])
end
defp list_all_rejected do
{:ok, keys} = Cachex.keys(@rejections)
{:ok, registry_data} =
Cachex.execute(@rejections, fn worker ->
Enum.map(keys, fn k -> {k, Cachex.get!(worker, k)} end)
end)
registry_data
end
defp clear_prior_rejection(origin),
do: Cachex.del(@rejections, origin)
defp is_rejected?(origin) do
case Cachex.get(@rejections, origin) do
{:ok, nil} ->
false
{:ok, _} ->
true
end
end
defp get_or_create_registry_data(origin) do
case get_registry_data(origin) do
{:error, :missing} ->
%SocketInfo{origin: origin}
{:ok, socket_info} ->
socket_info
end
end
defp save_registry_data(%SocketInfo{origin: origin, state: :connected} = socket_info) do
{:ok, true} = Registry.update_value(FedSockets.Registry, origin, fn _ -> socket_info end)
socket_info
end
defp save_registry_data(%SocketInfo{origin: origin, state: :rejected} = socket_info) do
rejection_expiration =
Pleroma.Config.get([:fed_sockets, :rejection_duration], @default_rejection_duration)
{:ok, true} = Cachex.put(@rejections, origin, socket_info, ttl: rejection_expiration)
socket_info
end
defp set_to_rejected(%SocketInfo{} = socket_info),
do: %SocketInfo{socket_info | state: :rejected}
end

137
lib/pleroma/web/fed_sockets/fed_socket.ex Normal file
View file

@ -0,0 +1,137 @@
# Pleroma: A lightweight social networking server
# Copyright © 2017-2020 Pleroma Authors <https://pleroma.social/>
# SPDX-License-Identifier: AGPL-3.0-only
defmodule Pleroma.Web.FedSockets.FedSocket do
@moduledoc """
The FedSocket module abstracts the actions to be taken taken on connections regardless of
whether the connection started as inbound or outbound.
Normally outside modules will have no need to call the FedSocket module directly.
"""
alias Pleroma.Object
alias Pleroma.Object.Containment
alias Pleroma.User
alias Pleroma.Web.ActivityPub.ObjectView
alias Pleroma.Web.ActivityPub.UserView
alias Pleroma.Web.ActivityPub.Visibility
alias Pleroma.Web.FedSockets.FetchRegistry
alias Pleroma.Web.FedSockets.IngesterWorker
alias Pleroma.Web.FedSockets.OutgoingHandler
alias Pleroma.Web.FedSockets.SocketInfo
require Logger
@shake "61dd18f7-f1e6-49a4-939a-a749fcdc1103"
def connect_to_host(uri) do
case OutgoingHandler.start_link(uri) do
{:ok, pid} ->
{:ok, pid}
error ->
{:error, error}
end
end
def close(%SocketInfo{pid: socket_pid}),
do: Process.send(socket_pid, :close, [])
def publish(%SocketInfo{pid: socket_pid}, json) do
%{action: :publish, data: json}
|> Jason.encode!()
|> send_packet(socket_pid)
end
def fetch(%SocketInfo{pid: socket_pid}, id) do
fetch_uuid = FetchRegistry.register_fetch(id)
%{action: :fetch, data: id, uuid: fetch_uuid}
|> Jason.encode!()
|> send_packet(socket_pid)
wait_for_fetch_to_return(fetch_uuid, 0)
end
def receive_package(%SocketInfo{} = fed_socket, json) do
json
|> Jason.decode!()
|> process_package(fed_socket)
end
defp wait_for_fetch_to_return(uuid, cntr) do
case FetchRegistry.check_fetch(uuid) do
{:error, :waiting} ->
Process.sleep(:math.pow(cntr, 3) |> Kernel.trunc())
wait_for_fetch_to_return(uuid, cntr + 1)
{:error, :missing} ->
Logger.error("FedSocket fetch timed out - #{inspect(uuid)}")
{:error, :timeout}
{:ok, _fr} ->
FetchRegistry.pop_fetch(uuid)
end
end
defp process_package(%{"action" => "publish", "data" => data}, %{origin: origin} = _fed_socket) do
if Containment.contain_origin(origin, data) do
IngesterWorker.enqueue("ingest", %{"object" => data})
end
{:reply, %{"action" => "publish_reply", "status" => "processed"}}
end
defp process_package(%{"action" => "fetch_reply", "uuid" => uuid, "data" => data}, _fed_socket) do
FetchRegistry.register_fetch_received(uuid, data)
{:noreply, nil}
end
defp process_package(%{"action" => "fetch", "uuid" => uuid, "data" => ap_id}, _fed_socket) do
{:ok, data} = render_fetched_data(ap_id, uuid)
{:reply, data}
end
defp process_package(%{"action" => "publish_reply"}, _fed_socket) do
{:noreply, nil}
end
defp process_package(other, _fed_socket) do
Logger.warn("unknown json packages received #{inspect(other)}")
{:noreply, nil}
end
defp render_fetched_data(ap_id, uuid) do
{:ok,
%{
"action" => "fetch_reply",
"status" => "processed",
"uuid" => uuid,
"data" => represent_item(ap_id)
}}
end
defp represent_item(ap_id) do
case User.get_by_ap_id(ap_id) do
nil ->
object = Object.get_cached_by_ap_id(ap_id)
if Visibility.is_public?(object) do
Phoenix.View.render_to_string(ObjectView, "object.json", object: object)
else
nil
end
user ->
Phoenix.View.render_to_string(UserView, "user.json", user: user)
end
end
defp send_packet(data, socket_pid) do
Process.send(socket_pid, {:send, data}, [])
end
def shake, do: @shake
end

185
lib/pleroma/web/fed_sockets/fed_sockets.ex Normal file
View file

@ -0,0 +1,185 @@
# Pleroma: A lightweight social networking server
# Copyright © 2017-2020 Pleroma Authors <https://pleroma.social/>
# SPDX-License-Identifier: AGPL-3.0-only
defmodule Pleroma.Web.FedSockets do
@moduledoc """
This documents the FedSockets framework. A framework for federating
ActivityPub objects between servers via persistant WebSocket connections.
FedSockets allow servers to authenticate on first contact and maintain that
connection, eliminating the need to authenticate every time data needs to be shared.
## Protocol
FedSockets currently support 2 types of data transfer:
* `publish` method which doesn't require a response
* `fetch` method requires a response be sent
### Publish
The publish operation sends a json encoded map of the shape:
%{action: :publish, data: json}
and accepts (but does not require) a reply of form:
%{"action" => "publish_reply"}
The outgoing params represent
* data: ActivityPub object encoded into json
### Fetch
The fetch operation sends a json encoded map of the shape:
%{action: :fetch, data: id, uuid: fetch_uuid}
and requires a reply of form:
%{"action" => "fetch_reply", "uuid" => uuid, "data" => data}
The outgoing params represent
* id: an ActivityPub object URI
* uuid: a unique uuid generated by the sender
The reply params represent
* data: an ActivityPub object encoded into json
* uuid: the uuid sent along with the fetch request
## Examples
Clients of FedSocket transfers shouldn't need to use any of the functions outside of this module.
A typical publish operation can be performed through the following code, and a fetch operation in a similar manner.
case FedSockets.get_or_create_fed_socket(inbox) do
{:ok, fedsocket} ->
FedSockets.publish(fedsocket, json)
_ ->
alternative_publish(inbox, actor, json, params)
end
## Configuration
FedSockets have the following config settings
config :pleroma, :fed_sockets,
enabled: true,
ping_interval: :timer.seconds(15),
connection_duration: :timer.hours(1),
rejection_duration: :timer.hours(1),
fed_socket_fetches: [
default: 12_000,
interval: 3_000,
lazy: false
]
* enabled - turn FedSockets on or off with this flag. Can be toggled at runtime.
* connection_duration - How long a FedSocket can sit idle before it's culled.
* rejection_duration - After failing to make a FedSocket connection a host will be excluded
from further connections for this amount of time
* fed_socket_fetches - Use these parameters to pass options to the Cachex queue backing the FetchRegistry
* fed_socket_rejections - Use these parameters to pass options to the Cachex queue backing the FedRegistry
Cachex options are
* default: the minimum amount of time a fetch can wait before it times out.
* interval: the interval between checks for timed out entries. This plus the default represent the maximum time allowed
* lazy: leave at false for consistant and fast lookups, set to true for stricter timeout enforcement
"""
require Logger
alias Pleroma.Web.FedSockets.FedRegistry
alias Pleroma.Web.FedSockets.FedSocket
alias Pleroma.Web.FedSockets.SocketInfo
@doc """
returns a FedSocket for the given origin. Will reuse an existing one or create a new one.
address is expected to be a fully formed URL such as:
"http://www.example.com" or "http://www.example.com:8080"
It can and usually does include additional path parameters,
but these are ignored as the FedSockets are organized by host and port info alone.
"""
def get_or_create_fed_socket(address) do
with {:cache, {:error, :missing}} <- {:cache, get_fed_socket(address)},
{:connect, {:ok, _pid}} <- {:connect, FedSocket.connect_to_host(address)},
{:cache, {:ok, fed_socket}} <- {:cache, get_fed_socket(address)} do
Logger.debug("fedsocket created for - #{inspect(address)}")
{:ok, fed_socket}
else
{:cache, {:ok, socket}} ->
Logger.debug("fedsocket found in cache - #{inspect(address)}")
{:ok, socket}
{:cache, {:error, :rejected} = e} ->
e
{:connect, {:error, _host}} ->
Logger.debug("set host rejected for - #{inspect(address)}")
FedRegistry.set_host_rejected(address)
{:error, :rejected}
{_, {:error, :disabled}} ->
{:error, :disabled}
{_, {:error, reason}} ->
Logger.warn("get_or_create_fed_socket error - #{inspect(reason)}")
{:error, reason}
end
end
@doc """
returns a FedSocket for the given origin. Will not create a new FedSocket if one does not exist.
address is expected to be a fully formed URL such as:
"http://www.example.com" or "http://www.example.com:8080"
"""
def get_fed_socket(address) do
origin = SocketInfo.origin(address)
with {:config, true} <- {:config, Pleroma.Config.get([:fed_sockets, :enabled], false)},
{:ok, socket} <- FedRegistry.get_fed_socket(origin) do
{:ok, socket}
else
{:config, _} ->
{:error, :disabled}
{:error, :rejected} ->
Logger.debug("FedSocket previously rejected - #{inspect(origin)}")
{:error, :rejected}
{:error, reason} ->
{:error, reason}
end
end
@doc """
Sends the supplied data via the publish protocol.
It will not block waiting for a reply.
Returns :ok but this is not an indication of a successful transfer.
the data is expected to be JSON encoded binary data.
"""
def publish(%SocketInfo{} = fed_socket, json) do
FedSocket.publish(fed_socket, json)
end
@doc """
Sends the supplied data via the fetch protocol.
It will block waiting for a reply or timeout.
Returns {:ok, object} where object is the requested object (or nil)
{:error, :timeout} in the event the message was not responded to
the id is expected to be the URI of an ActivityPub object.
"""
def fetch(%SocketInfo{} = fed_socket, id) do
FedSocket.fetch(fed_socket, id)
end
@doc """
Disconnect all and restart FedSockets.
This is mainly used in development and testing but could be useful in production.
"""
def reset do
FedRegistry
|> Process.whereis()
|> Process.exit(:testing)
end
def uri_for_origin(origin),
do: "ws://#{origin}/api/fedsocket/v1"
end

151
lib/pleroma/web/fed_sockets/fetch_registry.ex Normal file
View file

@ -0,0 +1,151 @@
# Pleroma: A lightweight social networking server
# Copyright © 2017-2020 Pleroma Authors <https://pleroma.social/>
# SPDX-License-Identifier: AGPL-3.0-only
defmodule Pleroma.Web.FedSockets.FetchRegistry do
@moduledoc """
The FetchRegistry acts as a broker for fetch requests and return values.
This allows calling processes to block while waiting for a reply.
It doesn't impose it's own process instead using `Cachex` to handle fetches in process, allowing
multi threaded processes to avoid bottlenecking.
Normally outside modules will have no need to call or use the FetchRegistry themselves.
The `Cachex` parameters can be controlled from the config. Since exact timeout intervals
aren't necessary the following settings are used by default:
config :pleroma, :fed_sockets,
fed_socket_fetches: [
default: 12_000,
interval: 3_000,
lazy: false
]
"""
defmodule FetchRegistryData do
defstruct uuid: nil,
sent_json: nil,
received_json: nil,
sent_at: nil,
received_at: nil
end
alias Ecto.UUID
require Logger
@fetches :fed_socket_fetches
@doc """
Registers a json request wth the FetchRegistry and returns the identifying UUID.
"""
def register_fetch(json) do
%FetchRegistryData{uuid: uuid} =
json
|> new_registry_data
|> save_registry_data
uuid
end
@doc """
Reports on the status of a Fetch given the identifying UUID.
Will return
* {:ok, fetched_object} if a fetch has completed
* {:error, :waiting} if a fetch is still pending
* {:error, other_error} usually :missing to indicate a fetch that has timed out
"""
def check_fetch(uuid) do
case get_registry_data(uuid) do
{:ok, %FetchRegistryData{received_at: nil}} ->
{:error, :waiting}
{:ok, %FetchRegistryData{} = reg_data} ->
{:ok, reg_data}
e ->
e
end
end
@doc """
Retrieves the response to a fetch given the identifying UUID.
The completed fetch will be deleted from the FetchRegistry
Will return
* {:ok, fetched_object} if a fetch has completed
* {:error, :waiting} if a fetch is still pending
* {:error, other_error} usually :missing to indicate a fetch that has timed out
"""
def pop_fetch(uuid) do
case check_fetch(uuid) do
{:ok, %FetchRegistryData{received_json: received_json}} ->
delete_registry_data(uuid)
{:ok, received_json}
e ->
e
end
end
@doc """
This is called to register a fetch has returned.
It expects the result data along with the UUID that was sent in the request
Will return the fetched object or :error
"""
def register_fetch_received(uuid, data) do
case get_registry_data(uuid) do
{:ok, %FetchRegistryData{received_at: nil} = reg_data} ->
reg_data
|> set_fetch_received(data)
|> save_registry_data()
{:ok, %FetchRegistryData{} = reg_data} ->
Logger.warn("tried to add fetched data twice - #{uuid}")
reg_data
{:error, _} ->
Logger.warn("Error adding fetch to registry - #{uuid}")
:error
end
end
defp new_registry_data(json) do
%FetchRegistryData{
uuid: UUID.generate(),
sent_json: json,
sent_at: :erlang.monotonic_time(:millisecond)
}
end
defp get_registry_data(origin) do
case Cachex.get(@fetches, origin) do
{:ok, nil} ->
{:error, :missing}
{:ok, reg_data} ->
{:ok, reg_data}
_ ->
{:error, :cache_error}
end
end
defp set_fetch_received(%FetchRegistryData{} = reg_data, data),
do: %FetchRegistryData{
reg_data
| received_at: :erlang.monotonic_time(:millisecond),
received_json: data
}
defp save_registry_data(%FetchRegistryData{uuid: uuid} = reg_data) do
{:ok, true} = Cachex.put(@fetches, uuid, reg_data)
reg_data
end
defp delete_registry_data(origin),
do: {:ok, true} = Cachex.del(@fetches, origin)
end

88
lib/pleroma/web/fed_sockets/incoming_handler.ex Normal file
View file

@ -0,0 +1,88 @@
# Pleroma: A lightweight social networking server
# Copyright © 2017-2020 Pleroma Authors <https://pleroma.social/>
# SPDX-License-Identifier: AGPL-3.0-only
defmodule Pleroma.Web.FedSockets.IncomingHandler do
require Logger
alias Pleroma.Web.FedSockets.FedRegistry
alias Pleroma.Web.FedSockets.FedSocket
alias Pleroma.Web.FedSockets.SocketInfo
import HTTPSignatures, only: [validate_conn: 1, split_signature: 1]
@behaviour :cowboy_websocket
def init(req, state) do
shake = FedSocket.shake()
with true <- Pleroma.Config.get([:fed_sockets, :enabled]),
sec_protocol <- :cowboy_req.header("sec-websocket-protocol", req, nil),
headers = %{"(request-target)" => ^shake} <- :cowboy_req.headers(req),
true <- validate_conn(%{req_headers: headers}),
%{"keyId" => origin} <- split_signature(headers["signature"]) do
req =
if is_nil(sec_protocol) do
req
else
:cowboy_req.set_resp_header("sec-websocket-protocol", sec_protocol, req)
end
{:cowboy_websocket, req, %{origin: origin}, %{}}
else
_ ->
{:ok, req, state}
end
end
def websocket_init(%{origin: origin}) do
case FedRegistry.add_fed_socket(origin) do
{:ok, socket_info} ->
{:ok, socket_info}
e ->
Logger.error("FedSocket websocket_init failed - #{inspect(e)}")
{:error, inspect(e)}
end
end
# Use the ping to check if the connection should be expired
def websocket_handle(:ping, socket_info) do
if SocketInfo.expired?(socket_info) do
{:stop, socket_info}
else
{:ok, socket_info, :hibernate}
end
end
def websocket_handle({:text, data}, socket_info) do
socket_info = SocketInfo.touch(socket_info)
case FedSocket.receive_package(socket_info, data) do
{:noreply, _} ->
{:ok, socket_info}
{:reply, reply} ->
{:reply, {:text, Jason.encode!(reply)}, socket_info}
{:error, reason} ->
Logger.error("incoming error - receive_package: #{inspect(reason)}")
{:ok, socket_info}
end
end
def websocket_info({:send, message}, socket_info) do
socket_info = SocketInfo.touch(socket_info)
{:reply, {:text, message}, socket_info}
end
def websocket_info(:close, state) do
{:stop, state}
end
def websocket_info(message, state) do
Logger.debug("#{__MODULE__} unknown message #{inspect(message)}")
{:ok, state}
end
end

Some files were not shown because too many files have changed in this diff Show more