* Fix#2473 - Use sidekiq scheduler to refresh PuSH subscriptions instead of cron
Fix an issue where / in domain would raise exception in TagManager#normalize_domain
PuSH subscriptions refresh done in a round-robin way to avoid hammering a single
server's hub in sequence. Correct handling of failures/retries through Sidekiq (see
also #2613). Optimize Account#with_followers scope. Also, since subscriptions
are now delegated to Sidekiq jobs, an uncaught exception will not stop the entire
refreshing operation halfway through
Fix#2702 - Correct user agent header on outgoing http requests
* Add test for SubscribeService
* Extract #expiring_accounts into method
* Make mastodon:push:refresh no-op
* Queues are now defined in sidekiq.yml
* Queues are now in sidekiq.yml
* Likely fix#2458, fix#2031 - handle out-of-order deletes for statuses
If a delete arrives before the original status, cache that information
for 6h, and if the original status arrives in that window, ignore it
* Add test case
Fix#2196 - Respond with 201 when Salmon accepted, 400 when unverified
Fix#2629 - Correctly handle confirm_domain? for local accounts
Unify rules for extracting author acct from XML, prefer <email>, fall back
to <name> + <uri> (see also #2017, #2172)
Follow-up to #2599. When a domain block with `reject_media` is
added or `rake mastodon:media:remove_remote` is invoked, mastodon
deletes the locally cached attachments and avatars but does not
reflect that change in the database, causing the `file` fields to
still have values. This change persists the deletion in the
database and sets the attachment type to unknown.
This also introduces a one-off rake task that sets all attachments
without a local file to the "unknown" type. The upgrade notes for
the next release should contain a post-upgrade step with
`rake mastodon:media:set_unknown`.
* Instead of parsing shared status contents verbatim, make roundtrip
to purported original URL. Confirm that the "original" URL is from the
same domain as the author it claims to be from.
* Fix obvious typo, add comment
* Use URI look-up first
* Add test, update Goldfinger dependency to make less useless HTTP requests per Webfinger lookup
* OEmbed support for PreviewCard
* Improve ProviderDiscovery code failure treatment
* Do not crawl links if there is a content warning, since those
don't display a link card anyway
* Reset db schema
* Fresh migrate
* Fix rubocop style issues
Fix#1681 - return existing access token when applicable instead of creating new
* Fix test
* Extract http client to helper
* Improve oembed controller
* Add coverage for domain block service with silence
* Get rid of warning about find_each and order
* Move domain_block to attr_reader
* Move optional clear_media into silence_accounts method
* Use blocked_domain method to reduce passed vars
* Extract blocked_domain_accounts method to find accounts on the domain
* Extract media_from_blocked_domain method to find relevant attachments
* Separate destruction of account images and account attachments
duplicates. Web UI regenerates UUID for that header every time the compose
form is changed or successfully submitted
Also, fix Farsi i18n overwriting the English one
* Fix#2119 - Whenever about to send a HTTP request, normalize the URI
* Add test for IDN request in FetchLinkCardService
* Perform IDN normalization on domains before they are stored in the DB
* Fix#1057 (close#1819) - Move HTML-formatted bio from <poco:note /> to <summary type="html" />
* Ensure <poco:note /> is plaintext for remote accounts, also, by stripping out HTML
* Extract detect_language to separate class
* Use default locale, not just en
* Add spec to confirm that whatlanguage cant identify empty string
* Allow account locale to override default in language detector
* PostStatusService supplies an account to detect language
* reduce unneed query when post without attachements.
This commit reduce following query:
```
MediaAttachment Load (0.9ms) SELECT "media_attachments".* FROM "media_attachments" WHERE "media_attachments"."status
_id" IS NULL AND 1=0 ORDER BY id asc
```
* fixed for more simple changes.
* Add language detection via WhatLanguage and (de)serialization of it through Atom
* Fix default language in ProcessFeedService
* Re-add newline before 'react-rails' Gem to fix groupings
Fixes Code Climate issue
The <link rel="mentioned" /> tag refers to accounts by href. So we were
matching the DB by the url attribute, and falling back to HTTP look-up.
However, GS and Mastodon use profile URLs as URIs, too, and the match
for that was missing. This could potentially alleviate some extra network
requests
Create MediaAttachment but without actual file download when domain is blocked with reject_media set to true
Clean up old media files when creating a new domain block with reject_media set to true
Return remote_url in media attachments API if local file is not present
Undo domain block action in admin UI
Ability to enable reject_media from admin UI
* Begin coverage for account search service
* Coverage for hashtag query
* Coverage for calling local vs remote find based on domain presence
* Spec to check that exact matches are not duped
* Coverage of resolve option
* Coverage for account being provided
* Start to refactor account search service
* Isolate query username and domain methods
* Isolate exact_match method
* Extract methods for local and remote results
* Simplify local vs remote and account isoliation
* Extract methods for local and remote results
* Simplify de-dupe of exact match
* Simplify logic to check for non exact remotes
* Cache some methods
* Remove nil from exact_match from results array
* Return exact matches first
* Use find_remote even with no domain
Account.find_local is just an alias for Account.find_remote(user, nil) - so we
can not bother with the conditional here, and call find_remote directly.
* Refresh local info for remote accounts when webfinger returns new values
It only refreshes account info if one of the URLs or the public-key changes,
in which cases it refreshes the full info, re-downloading the feeds from that
user.
Some special handling should probably be done when the public key changes,
but I have been unable to find any use for it in Mastodon yet.
* Re-fetch remote users we aren't subscribed to.
This might induce performance issues, we might want to only do that for users
we explicitly attempted to subscribe but failed to.
* Refactor changes
* Do not refresh existing remote account details more than once a day
* Avoid re-fetching webfinger info in tests unless otherwise specified
* Allow running mastodon on a different domain as the one used for identifying users
* Alter documentation of WEB_DOMAIN to make clear it shouldn't be used unless the admin knows what they are doing
* Compare to web_domain instead of local_domain when dealing with feeds/API
* Correctly identify mentions to local accounts
Mentions URLs point to the person's web profile, i.e., the user page served on WEB_DOMAIN.
The goal of this change is to enhance Mastodon's handling of remote domains
for which the APIs reside on a different host (see issue #1032).
Indeed, when a remote user unknown to Mastodon is mentionned, only its profile
URL (e.g. https://social.example.org/users/User) is known, and Mastodon has to
build a @username@domain handle for it. To do so, Mastodon fetches the user's
atom feed (e.g., https://social.example.org/users/User.atom) and uses its
content to get the username part of the handle, and the URL's host part to
build the domain (e.g., @User@social.example.org). This handle is then used
for a Webfinger request.
In the case where example.org serves the Webfinger info for @User@example.org
and all feeds and APIs are hosted at social.example.org, Mastodon will still
build @User@social.example.org and fail at resolving the account's details
through Webfinger.
This patch changes this behaviour by using the author's email address from
the atom feed to build the handle. In Mastodon-generated atom feeds, the
email address is always the handle it expects for federation.
* Fix#1141, fix#1126 - Work through UpdateRemoteProfileService for both <feed> and <entry> top-level tags
* Improve code quality, remove line unrelated to fix
* Rewrite Atom generation from stream entries to use Ox instead of Nokogiri::Builder
StreamEntry is now limited to only statuses, which allows some optimization. Removed
extra queries on AccountsController#show. AtomSerializer instead of AtomBuilderHelper
used in AccountsController#show, StreamEntriesController#show, StreamEntryRenderer
and PubSubHubbub::DistributionWorker
PubSubHubbub::DistributionWorker moves n+1 DomainBlock query to PubSubHubbub::DeliveryWorker
instead.
All Salmon slaps that aren't based on StreamEntry still use AtomBuilderHelper and Nokogiri
* All Salmon slaps now use Ox instead of Nokogiri. No touch from status on account
FanOutOnWriteService. The only recipient-specific part on them
is reblogged/favourited. But since only newly created statuses
appear on them, it is safe to assume that both attributes would
be false
fix ProcessFeedService pushing status into distribution if called a second time
while the first is still running (i.e. when a PuSH comes after a Salmon slap),
fix not running escape on spoiler text before emojify
Federate spoiler_text using warning attribute on <content /> instead of a <category term="spoiler" />
Clean up schema file from accidental development migrations
application website validation, don't link to app website if website isn't set,
also comment out animated boost icon from #464 until it's consistent with non-animated version
Filters out hidden stream entries from Atom feed
Blocks now generate hidden stream entries, can be used to federate blocks
Private statuses cannot be reblogged (generates generic 422 error for now)
POST /api/v1/statuses now takes visibility=(public|unlisted|private) param instead of unlisted boolean
Statuses JSON now contains visibility=(public|unlisted|private) field
Fetching atom extracted from FetchRemoteAccountService and FetchRemoteStatusService
into FetchAtomService. Mentions of the constant "http://activityschema.org/collection/public"
skipped as it's not a real URL/user.
This is a big one, so let me enumerate:
Accounts as well as stream entry pages now contain Link headers that
reference the Atom feed and Webfinger URL for the former and Atom entry
for the latter. So you only need to HEAD those resources to get that
information, no need to download and parse HTML <link>s.
ProcessFeedService will now queue ThreadResolveWorker for each remote
status that it cannot find otherwise. Furthermore, entries are now
processed in reverse order (from bottom to top) in case a newer entry
references a chronologically previous one.
ThreadResolveWorker uses FetchRemoteStatusService to obtain a status
and attach the child status it was queued for to it.
FetchRemoteStatusService looks up the URL, first with a HEAD, tests
if it's an Atom feed, in which case it processes it directly. Next
for Link headers to the Atom feed, in which case that is fetched
and processed. Lastly if it's HTML, it is checked for <link>s to the Atom
feed, and if such is found, that is fetched and processed. The account for
the status is derived from author/name attribute in the XML and the hostname
in the URL (domain). FollowRemoteAccountService and ProcessFeedService
are used.
This means that potentially threads are resolved recursively until a dead-end
is encountered, however it is performed asynchronously over background jobs,
so it should be ok.
