* Fix performance of account timelines
* Various fixes and improvements
* Fix duplicate results being returned
Co-authored-by: Claire <claire.github-309c@sitedethib.com>
* Fix grouping for pinned statuses scope
Co-authored-by: Claire <claire.github-309c@sitedethib.com>
* Add `/api/v1/accounts/familiar_followers` to REST API
* Change hide network preference to be stored consistently for local and remote accounts
* Add dummy classes to migration
* Apply suggestions from code review
Co-authored-by: Claire <claire.github-309c@sitedethib.com>
Co-authored-by: Claire <claire.github-309c@sitedethib.com>
* Add trending statuses
* Fix dangling items with stale scores in localized sets
* Various fixes and improvements
- Change approve_all/reject_all to approve_accounts/reject_accounts
- Change Trends::Query methods to not mutate the original query
- Change Trends::Query#skip to offset
- Change follow recommendations to be refreshed in a transaction
* Add tests for trending statuses filtering behaviour
* Fix not applying filtering scope in controller
* Add notifications when a reblogged status has been updated
* Change wording to say "edit" instead of "update" and add missing controls
* Replace previous update notifications with the most up-to-date one
* Add editing for published statuses
* Fix change of multiple-choice boolean in poll not resetting votes
* Remove the ability to update existing media attachments for now
* Fix Sidekiq warnings about JSON serialization
This occurs on every symbol argument we pass, and every symbol key in hashes,
because Sidekiq expects strings instead.
See https://github.com/mperham/sidekiq/pull/5071
We do not need to change how workers parse their arguments because this has
not changed and we were already converting to symbols adequately or using
`with_indifferent_access`.
* Set Sidekiq to raise on unsafe arguments in test mode
In order to more easily catch issues that would produce warnings in production
code.
* Remove support for OAUTH_REDIRECT_AT_SIGN_IN
Fixes#15959
Introduced in #6540, OAUTH_REDIRECT_AT_SIGN_IN allowed skipping the log-in form
to instead redirect to the external OmniAuth login provider.
However, it did not prevent the log-in form on /about introduced by #10232 from
appearing, and completely broke with the introduction of #15228.
As I restoring that previous log-in flow without introducing a security
vulnerability may require extensive care and knowledge of how OmniAuth works,
this commit removes support for OAUTH_REDIRECT_AT_SIGN_IN instead for the time
being.
* Add OMNIAUTH_ONLY environment variable to enforce external log-in only
* Disable user registration when OMNIAUTH_ONLY is set to true
* Replace log-in links When OMNIAUTH_ONLY is set with exactly one OmniAuth provider
* Add support for editing for published statuses
* Fix references to stripped-out code
* Various fixes and improvements
* Further fixes and improvements
* Fix updates being potentially sent to unauthorized recipients
* Various fixes and improvements
* Fix wrong words in test
* Fix notifying accounts that were tagged but were not in the audience
* Fix mistake
* Add trending links
* Add overriding specific links trendability
* Add link type to preview cards and only trend articles
Change trends review notifications from being sent every 5 minutes to being sent every 2 hours
Change threshold from 5 unique accounts to 15 unique accounts
* Fix tests
* Add followed_by? to account_interactions
* Add RemoveFromFollowersService
* Fix AccountBatch to use RemoveFromFollowersService
* Add remove from followers API
* Fix issues with POSIX::Spawn, Terrapin and Ruby 3.0
Also improve the Terrapin monkey-patch for the stderr/stdout issue.
* Fix keyword argument handling throughout the codebase
* Monkey-patch Paperclip to fix keyword arguments handling in validators
* Change validation_extensions to please CodeClimate
* Bump microformats from 4.2.1 to 4.3.1
* Allow Ruby 3.0
* Add Ruby 3.0 test target to CircleCI
* Add test for admin dashboard warnings
* Fix admin dashboard warnings on Ruby 3.0
If someone tries logging in to an account and is prompted for a 2FA
code or sign-in token, even if the account's password or e-mail is
updated in the meantime, the session will show the prompt and allow
the login process to complete with a valid 2FA code or sign-in token
* Add bell button
Fix#4890
* Remove duplicate type from post-deployment migration
* Fix legacy class type mappings
* Improve query performance with better index
* Fix validation
* Remove redundant index from notifications
* Add database support for list show-reply preferences
* Add backend support to read and update list-specific show_replies settings
* Add basic UI to set list replies setting
* Add specs for list replies policy
* Switch "cycling" reply policy link to a set of radio inputs
* Capitalize replies_policy strings
* Change radio button design to be consistent with that of the directory explorer
* Make Array-creation behavior of Paginable more predictable
Paginable.paginate_by_id usually returns ActiveRecord::Relation, but it
returns an Array if min_id option is present. The behavior caused problems
fixed with the following commits:
- 552e886b64
- b63ede5005
- 64ef37b89d
To prevent from recurring similar problems, this commit introduces two
changes:
- The scope now always returns an Array whether min_id option is present
or not.
- The scope is renamed to to_a_paginated_by_id to clarify it returns an
Array.
* Transform Paginable.to_a_paginated_by_id from a scope to a class method
https://api.rubyonrails.org/classes/ActiveRecord/Scoping/Named/ClassMethods.html#method-i-scope
> The method is intended to return an ActiveRecord::Relation object, which
> is composable with other scopes.
Paginable.to_a_paginated_by_id returns an Array and is not appropriate
as a scope.
* Replace incorrect use of distinct with group
Some uses of ActiveRecord::QueryMethods#distinct pass field names but they
are incorrect for the current version of Rails.
ActiveRecord::QueryMethods#group provides the expected behavior and
benefits performance. See commit 6da24aad4c.
* Introduce ApplicationController#cache_collection_paginated_by_id
ApplicationController#cache_collection_paginated_by_id fuses
ApplicationController#cache_collection and Paginable.paginate_by_id.
An advantage of this method is that it prevents from modifying scope which
Paginable.paginate_by_id may provide.
ApplicationController#cache_collection always return an array and there
is no possibility of the scope modification. It is also clear for a
programmer, considering the implication of "cache".
This method can also emit more efficient queries by using
Cacheable.cache_ids before calling Paginable.paginate_by_id.
Some uses of ActiveRecord::QueryMethods#distinct pass field names but they
are incorrect for the current version of Rails.
ActiveRecord::QueryMethods#group provides the expected behavior and
benefits performance. See commit 6da24aad4c.
The old implementation had two queries:
1. The query constructed in Api::V1::FavouritesController#results
2. The query constructed in #cached_favourites, which is merged with 1.
Both of them are issued againt PostgreSQL. The combination of the two
queries caused the following problems:
- The small window between the two queries involves race conditions.
- Minor performance inefficiency.
Moreover, the construction of query 2, which involves merging with query
1 has a bug. Query 1 is finalized with paginate_by_id, but paginate_by_id
returns an array when min_id parameter is specified. The behavior prevents
from merging the query, and in the real world, ActiveRecord simply ignores
the merge (!), which results in querying the entire scan of statuses and
favourites table.
This change fixes these issues by simply letting query 1 get all the works
done.
DISTINCT clause removes duplicated records according to all the selected
attributes. In reality, it can remove duplicated records only looking at
statuses.id, but the clause confuses the query planner and yields
insufficient performance.
The behavior is also problematic if the scope produced by HashQueryService
is used to query columns without id (using pluck method, for example). The
scope is expected to contain unique statuses, but the uniquness will be
evaluated with some arbitrary columns other than id.
GROUP BY clause resolves those problem by explicitly specifying the
column to take into account for the record distinction.
A workaround for the problem of DISTINCT clause in
Api::V1::Timelines::TagController is no longer necessary and removed.
* Add toot send by current user at local state after send a new toot
Related to #14021
* Decrement toot counter at profile when remove a toot
Related to #14021
* Remove semicolon at end of line
* Add UserNote model
* Add UI for user notes
* Put comment in relationships entity
* Add API to create user notes
* Copy user notes to new account when receiving a Move activity
* Address some of the review remarks
* Replace modal by inline edition
* Please CodeClimate
* Button design changes
* Change design again
* Cancel note edition when pressing Escape
* Fixes
* Tweak design again
* Move “Add note” item, and allow users to add notes to themselves
* Rename UserNote into AccountNote, rename “comment” Relationship attribute to “note”
- Change audio files to not be stripped of metadata
- Automatically extract cover art from audio if it exists
- Add `thumbnail` parameter to `POST /api/v1/media`, `POST /api/v2/media` and `PUT /api/v1/media/:id`
- Add `icon` to represent it in attachments in ActivityPub
- Fix `preview_url` containing URL of missing missing image when there is no thumbnail instead of null
- Fix duration of audio not being displayed on public pages until the file is loaded
* Fix 404 and 410 API errors being silently discarded in WebUI
Fixes#13278
* Return more appropriate error when user replies to a deleted toot
* Please CodeClimate
* Fix 404/410 errors on fetching account timelines & identity proofs
* Refactor error handling
* Move error message string to statuses.errors
* Change ActivityPub follower/following collections to not link first page
* Add support for hiding followers and following of remote users
* Switch to using a single `hide_collections` column
* Address code style remarks
* Change meaning of /api/v1/announcements/:id/dismiss to mark an announcement as read
* Change how unread announcements are counted in UI
* Add unread marker to announcements and mark announcements as unread as they are displayed
* Fixups
* Add announcements
Fix#11006
* Add reactions to announcements
* Add admin UI for announcements
* Add unit tests
* Fix issues
- Add `with_dismissed` param to announcements API
- Fix end date not being formatted when time range is given
- Fix announcement delete causing reactions to send streaming updates
- Fix announcements container growing too wide and mascot too small
- Fix `all_day` being settable when no time range is given
- Change text "Update" to "Announcement"
* Fix scheduler unpublishing announcements before they are due
* Fix filter params not being passed to announcements filter
* Hide blocked, muted, and blocked-by users from toot favourite lists
* Hide blocked, muted, and blocked-by users from toot reblog lists
* Hide blocked, muted, and blocked-by users from followers/following (API)
* Fix tests
* Hide blocked, muted, and blocked-by users from followers/following on public pages
* Add follow_request notification type
The notification type already existed in the backend but was never pushed
to the front-end. This also means translation strings were also available
for the backend, from the notification mailer.
Unlike other notification types, these are off by default, to match what
I remember of Gargron's view on the topic: that follow requests should not
clutter notifications and should instead be reviewed at the user's own
leisure in the dedicated column.
Since follow requests have their own column, I've deemed it unnecessary to
add a specific tab for them in the notification quick filter.
* Show follow request link in single-column if there are pending requests, even if account isn't locked
* Push follow requests from notifications to the follow_requests list
* Offer to accept or reject follow request from the notification
* Redesign follow request notification
* make it not return http 400 when passing and empty source argument
* create a spec for the empty source hash bug
* compact checks for nil, empty? parameters
* use nil.blank? instead checking for nil
* Add backend support for bookmarks
Bookmarks behave like favourites, except they aren't shared with other
users and do not have an associated counter.
* Add spec for bookmark endpoints
* Add front-end support for bookmarks
* Introduce OAuth scopes for bookmarks
* Add bookmarks to archive takeout
* Fix migration
* Coding style fixes
* Fix rebase issue
* Update bookmarked_statuses to latest UI changes
* Update bookmark actions to properly reflect status changes in state
* Add bookmarks item to single-column layout
* Make active bookmarks red
Change the behaviour of remotable concern. Previously, it would skip
downloading an attachment if the stored remote URL is identical to
the new one. Now it would not be skipped if the attachment is not
actually currently stored by Paperclip.
Fetching statuses from all followed accounts at once takes too long
within Postgres. Fetching them one by one and merging in Ruby
could be a lot less resource-intensive
Because the query for dynamically fetching the home timeline is so
heavy, we can no longer offer it when the home timeline is missing
* Change silenced accounts to require approval on follow
* Also require approval for follows by people explicitly muted by target accounts
* Do not auto-accept silenced or muted accounts when switching from locked to unlocked
* Add `follow_requests_count` to verify_credentials
* Show “Follow requests” menu item if needed even if account is locked
* Add tests
* Correctly reflect that follow requests weren't auto-accepted when local account is silenced
* Accept follow requests from user-muted accounts to avoid leaking mutes
* Add soft delete for statuses to allow them to appear instant
* Allow reporting soft-deleted statuses and show them in the admin UI
* Change index for getting an account's statuses
The reason for unattaching media instead of removing it is to support
delete & redraft functionality, but remote or staff-removed statuses
will never be redrafted, so the media should be deleted immediately
Fix#271
Add back the `GET /api/v1/trends` API with the caveat that it does
not return tags that have not been allowed to trend by the staff.
When a hashtag begins to trend (internally) and that hashtag has
not been previously reviewed by the staff, the staff is notified.
The new admin UI for hashtags allows filtering hashtags by where
they are used (e.g. in the profile directory), whether they have
been reviewed or are pending reviewal, they show by how many people
the hashtag is used in the directory, how many people used it
today, how many statuses with it have been created today, and it
allows fixing the name of the hashtag to make it more readable.
The disallowed hashtags feature has been reworked. It is now
controlled from the admin UI for hashtags instead of from
the file `config/settings.yml`
Allow access to account settings, 2FA, authorized applications, and
account deletions to unconfirmed and pending users, as well as
users who had their accounts disabled. Suspended users cannot update
their e-mail or password or delete their account.
Display account status on account settings page, for example, when
an account is frozen, limited, unconfirmed or pending review.
After sign up, login users straight away and show a simple page that
tells them the status of their account with links to account settings
and logout, to reduce onboarding friction and allow users to correct
wrongly typed e-mail addresses.
Move the final sign-up step of SSO integrations to be the same
as above to reduce code duplication.
* Remove Salmon and PubSubHubbub endpoints
* Add error when trying to follow OStatus accounts
* Fix new accounts not being created in ResolveAccountService
* Fix poll API not requiring authentication on non-public polls
That API does not reveal the content of the status, i.e. the question
itself, nor who the author is, nor which status it belongs to, but it
does reveal the poll options and how many answers they got
Fix#10959
* Add test
* Add `from_account` to notifications API
this adds the ability to filter notifications by the account they
originated from
* passing a non-existent user should cause none to be returned
* Fix codeclimate warnings
* fix more codeclimate warnings
* make requested changes:
* use account id instead of user@domain
* name the param `account_id` instead of `from_account`
* Don't use `return` in a lambda
* Return Status with raw text in raw_content when deleting a status
* Use raw content if available on delete & redraft
* Rename raw_content to text; do not serialize formatted content when source is requested
* Revert "Fix filtering of favourited_by, reblogged_by, followers and following (#10447)"
This reverts commit 120544067f.
* Revert "Hide blocking accounts from blocked users (#10442)"
This reverts commit 62bafa20a1.
* Improve blocked view of profiles
- Change "You are blocked" to "Profile unavailable"
- Hide following/followers in API when blocked
- Disable follow button and show "Profile unavailable" on public profile as well
* Revert "Add indication that you have been blocked in web UI (#10420)"
This reverts commit bd02ec6daa.
* Revert "Add `blocked_by` relationship to the REST API (#10373)"
This reverts commit 9745de883b.
* Hide blocking accounts from search results
* Filter blocking accouts from account followers
* Filter blocking accouts from account's following accounts
* Filter blocking accounts from “reblogged by” and “favourited by” lists
* Remove blocking account from URL search
* Return 410 on trying to fetch user data from a user who blocked us
* Return 410 in /api/v1/account/statuses for suspended or blocking accounts
* Fix status filtering when performing URL search
* Restore some React improvements
Restore some cleanup from bd02ec6daa
* Refactor by adding `without_blocking` scope
* create account_identity_proofs table
* add endpoint for keybase to check local proofs
* add async task to update validity and liveness of proofs from keybase
* first pass keybase proof CRUD
* second pass keybase proof creation
* clean up proof list and add badges
* add avatar url to keybase api
* Always highlight the “Identity Proofs” navigation item when interacting with proofs.
* Update translations.
* Add profile URL.
* Reorder proofs.
* Add proofs to bio.
* Update settings/identity_proofs front-end.
* Use `link_to`.
* Only encode query params if they exist.
URLs without params had a trailing `?`.
* Only show live proofs.
* change valid to active in proof list and update liveness before displaying
* minor fixes
* add keybase config at well-known path
* extremely naive feature flagging off the identity proof UI
* fixes for rubocop
* make identity proofs page resilient to potential keybase issues
* normalize i18n
* tweaks for brakeman
* remove two unused translations
* cleanup and add more localizations
* make keybase_contacts an admin setting
* fix ExternalProofService my_domain
* use Addressable::URI in identity proofs
* use active model serializer for keybase proof config
* more cleanup of keybase proof config
* rename proof is_valid and is_live to proof_valid and proof_live
* cleanup
* assorted tweaks for more robust communication with keybase
* Clean up
* Small fixes
* Display verified identity identically to verified links
* Clean up unused CSS
* Add caching for Keybase avatar URLs
* Remove keybase_contacts setting
* Add polls
Fix#1629
* Add tests
* Fixes
* Change API for creating polls
* Use name instead of content for votes
* Remove poll validation for remote polls
* Add polls to public pages
* When updating the poll, update options just in case they were changed
* Fix public pages showing both poll and other media
* Add type, limit, offset, min_id, max_id, account_id to search API
Fix#8939
* Make the offset work on accounts and hashtags search as well
* Assure brakeman we are not doing mass assignment here
* Do not allow paginating unless a type is chosen
* Fix search query and index id field on statuses instead of created_at
* Add hashtag filter to profiles
GET /@:username/tagged/:hashtag
GET /api/v1/accounts/:id/statuses?tagged=:hashtag
* Display featured hashtags on public profile
* Use separate model for featured tags
* Update featured hashtag counters on-write
* Limit featured tags to 10
* Add REST API for creating an account
The method is available to apps with a token obtained via the client
credentials grant. It creates a user and account records, as well as
an access token for the app that initiated the request. The user is
unconfirmed, and an e-mail is sent as usual.
The method returns the access token, which the app should save for
later. The REST API is not available to users with unconfirmed
accounts, so the app must be smart to wait for the user to click a
link in their e-mail inbox.
The method is rate-limited by IP to 5 requests per 30 minutes.
* Redirect users back to app from confirmation if they were created with an app
* Add tests
* Return 403 on the method if registrations are not open
* Require agreement param to be true in the API when creating an account
* Eliminate extra accounts select query from FollowService
* Optimistically update follow state in web UI and hide loading bar
Fix#6205
* Asynchronize NotifyService in FollowService
And fix failing test
* Skip Webfinger resolve routine when called from FollowService if possible
If an account is ActivityPub, then webfinger re-resolving is not necessary
when called from FollowService. Improve options of ResolveAccountService
* Nascent tag menu on frontend
* Hook up frontend to search
* Tag intersection backend first pass
* Update yarnlock
* WIP
* Fix for tags not searching correctly
* Make radio buttons function
* Simplify radio buttons with modeOption
* Better naming
* Rearrange options
* Add all/any/none functionality on backend
* Small PR cleanup
* Move to service from scope
* Small cleanup, add proper service tests
* Don't use send with user input :D
* Set appropriate column header
* Handle auto updating timeline
* Fix up toggle function
* Use tag value correctly
* A bit more correct to use 'self' rather than 'all' in status scope
* Fix some style issues
* Fix more code style issues
* Style select dropdown more better
* Only use to_id'ed value to ensure no SQL injection
* Revamp frontend to allow for multiple selects
* Update backend / col header to account for more flexible tagging
* Update brakeman ignore
* Codeclimate suggestions
* Fix presenter tag_url
* Implement initial PR feedback
* Handle additional tag streaming
* CodeClimate tweak
* Add conversations API
* Add web UI for conversations
* Add test for conversations API
* Add tests for ConversationAccount
* Improve web UI
* Rename ConversationAccount to AccountConversation
* Remove conversations on block and mute
* Change last_status_id to be a denormalization of status_ids
* Add optimistic locking
* Allow min_id pagination in Feed#get
* Add min_id pagination to home and list timeline APIs
* Add min_id pagination to account statuses, public and tag APIs
* Remove unused stub in reports API
* Use min_id pagination in notifications, favourites, and fix order
* Fix HomeFeed#from_database not using paginate_by_id
* Add force_login option to OAuth authorize page
For when a user needs to sign into an app from multiple accounts
on the same server
* When logging out from modal header, redirect back after re-login
* Allow moderators to disable/enable login
* Instead of rejecting login, show forbidden error when login disabled
Avoid confusion because when login is rejected, the message is that
the account is not activated, which is wrong.
* Fix tests
DELETE /api/v1/suggestions/:account_id
When blocking, remove suggestion from both sides. Muting not affected,
since muting is supposed to be invisible to the target.
* Add more granular OAuth scopes
* Add human-readable descriptions of the new scopes
* Ensure new scopes look good on the app UI
* Add tests
* Group scopes in screen and color-code dangerous ones
* Fix wrong extra scope
* Re-add follow recommendations API
GET /api/v1/suggestions
Removed in 8efa081f21 due to Neo4J
dependency. The algorithm uses triadic closures, takes into account
suspensions, blocks, mutes, domain blocks, excludes locked and moved
accounts, and prefers more recently updated accounts.
* Track interactions with people you don't follow
Replying to, favouriting and reblogging someone you're not following
will make them show up in follow recommendations. The interactions
have different weights:
- Replying is 1
- Favouriting is 10 (decidedly positive interaction, but private)
- Reblogging is 20
Following them, muting or blocking will remove them from the list,
obviously.
* Remove triadic closures, ensure potential friendships are trimmed
* Add keyword filtering
GET|POST /api/v1/filters
GET|PUT|DELETE /api/v1/filters/:id
- Irreversible filters can drop toots from home or notifications
- Other filters can hide toots through the client app
- Filters use a phrase valid in particular contexts, expiration
* Make sure expired filters don't get applied client-side
* Add missing API methods
* Remove "regex filter" from column settings
* Add tests
* Add test for FeedManager
* Add CustomFilter test
* Add UI for managing filters
* Add streaming API event to allow syncing filters
* Fix tests
* Send rejections to followers when user hides domain they're on
* Use account domain blocks for "authorized followers" action
Replace soft-blocking (block & unblock) behaviour with follow rejection
* Split sync and async work of account domain blocking
Do not create domain block when removing followers by domain, that
is probably unexpected from the user's perspective.
* Adjust confirmation message for domain block
* yarn manage:translations
* optimize direct timeline
* fix typo in class name
* change filter condition for direct timeline
* fix codestyle issue
* revoke index_accounts_not_silenced because direct timeline does not use it.
* revoke index_accounts_not_silenced because direct timeline does not use it.
* fix rspec test condition.
* fix rspec test condition.
* fix rspec test condition.
* revoke adding column and partial index
* (direct timeline) move merging logic to model
* fix pagination parameter
* add method arguments that switches return array of status or cache_ids
* fix order by
* returns ActiveRecord.Relation in default behavor
* fix codestyle issue
* Track trending tags
- Half-life of 1 day
- Historical usage in daily buckets (last 7 days stored)
- GET /api/v1/trends
Fix#271
* Add trends to web UI
* Don't render compose form on search route, adjust search results header
* Disqualify tag from trends if it's in disallowed hashtags setting
* Count distinct accounts using tag, ignore silenced accounts
* Add preference to hide following/followers lists
- Public pages
- ActivityPub collections (does not return pages but does give total)
- REST API (unless it's your own) (does not federate)
Fix#6901
* Add preference
* Add delegation
* Fix issue
* Fix issue
permitted_for scope is slow when combined with pinned status scope.
Fortunately permitted_for scope can safely be removed because a pinned
status is always public.
- POST /api/v1/push/subscription
- PUT /api/v1/push/subscription
- DELETE /api/v1/push/subscription
- New OAuth scope: "push" (required for the above methods)
* No need to re-require sidekiq plugins, they are required via Gemfile
* Add derailed_benchmarks tool, no need to require TTY gems in Gemfile
* Replace ruby-oembed with FetchOEmbedService
Reduce startup by 45382 allocated objects
* Remove preloaded JSON-LD in favour of caching HTTP responses
Reduce boot RAM by about 6 MiB
* Fix tests
* Fix test suite by stubbing out JSON-LD contexts
