support Bluesky / AT Protocol

[snarfed]

Background: https://bsky.social/ , https://atproto.com/docs, https://github.com/bluesky-social/atproto.

...switched to the now label for tracking roadmap and current tasks.

---

Blocking:

• ATProto: render DIDs on user pages as handles #948
• Cron jobs for /queue/atproto-poll-notifs, -commits #942
• Opt in/out prompt #880
• Polish AP and Bluesky user pages #825
• Bluesky: update docs #823
• Change Bluesky abbreviation from atproto to bsky #961
• Support for moderation activities granary#596
• Bluesky: truncate string values to length limits granary#717
• Entryway PDS resolves handles after account migration bluesky-social/atproto#2280

Nice to have:

• Protocol icons #939
• ActivityPub => ATProto @-mentions #957
• ActivityPub => ATProto hashtags #980
• ATProto => ActivityPub link previews #958
• ActivityPub => ATProto profile update #960
• Opt in via follow or DM should override profile opt-outs (#nobridge, #nobot, Bluesky "hide from logged out view") #974
• Follows from non-opted-in users don't work #975
• Only deliver a reply to a given protocol if its in-reply-to is also on that protocol #981
• ActivityPub => ATProto: handle usernames with _ and ~ chars #982
• ATProto: convert polling for posts/notifs to task per user #943
• Use Bluesky firehose instead of polling #978
• DM bot user to prompt remote users to enable (opt into) the bridge #966
• AP => ATProto: Support custom handle domains for bridged accounts #826

---

Manual tests:

• user pages

  • ActivityPub
  • ATProto
    • support handle in URL as well as DID

• web => ATProto

  • create bridged user in Bluesky
    • ...with complete profile (name, bio, picture, banner, links)
    • update profile, ActivityPub => ATProto profile update #960?
  • follow (original)
  • post (original)
    • ...with image (original)
  • reply (original)
  • like (original)
  • repost (original)
  • @-mention (original)

• ATProto => web

  • discover user (same as create bridged user in Bluesky above)
  • follow (original)
  • post (seen in Atom/RSS feeds)
    • ...with image
  • reply (original)
  • like (original)
  • repost (original)

• ActivityPub => ATProto

  • create bridged user in Bluesky (original)
    • ...with complete profile (name, bio, picture, banner, links)
    • update profile, ActivityPub => ATProto profile update #960
  • follow (original)
  • post, with image (original)
  • reply (original)
    • ...with @-mention: ActivityPub => ATProto @-mentions #957
  • like (original)
  • repost (original)
    Haven't gotten this working yet, not sure why. Tried reposting this post and this post on indieweb.social, UI shows that it works fine, but BF never got inbox delivery of the Announces, and when I click the boosts link on those posts in indieweb.social, eg here, they don't show any reposts. Especially odd since likes are working, and reposts are so similar. Finally got one! Maybe indieweb.social's queue was just backed up.
  • @-mention (original)

• ATProto => ActivityPub

  • discover user (original)
  • follow (original)
  • post (original)
    • ...with image (original)
    • ...with link preview: ATProto => ActivityPub link previews #958
  • reply (original)
  • like (original)
  • repost (original)
  • @-mention, ATProto => ActivityPub @-mentions #959

---

Implementation tasks:

• Fully implement ATProto protocol class
• Separate service for serving ATProto's subscribeRepos #630
  • [ ] port arroba's websocket server to asyncio?
  • Look into Eventarc for delivering new commits to websocket subscribers arroba#6
• map object ids to originals with Object.copies. (in Object.as1? or Protocol._targets? or...?)
• add User.copies, use it similarly to map actor/author etc targets
• receive task queue. (later hopefully we'll migrate incoming webmentions and AP inbox deliveries to this too, but that's complicated.)
• poll for notifications periodically, per user (How do federated notifications work? bluesky-social/atproto#1538 (reply in thread) )
• Poll for posts in pull-oriented protocols #694
• convert ids and handles, both directions
• serve WebFinger for ATP users
  • milestone: BF user profiles are visible in the fediverse
• User discovery for push based protocols #647 (Will user search/discovery in clients ever trigger handle resolution? bluesky-social/atproto#1650)
• create _atproto TXT DNS records on the fly in GCP DNS (000b08f)
• send requestCrawl to the BGS on startup
• ATProto.send: ignore Accepts of Follows
• milestone: successfully federate a non-Bluesky user into the federation sandbox
• milestone: successfully federate a Bluesky user to another protocol
• milestone: successfully federate a non-Bluesky post into the federation sandbox
• milestone: successfully federate a federation sandbox post to another protocol
• decide whether we have a BF protocol per lexicon/app, eg Bluesky, or just one for all of ATProto. We'd prefer the latter, but we need some app-specific semantics. Not sure yet if we can avoid all of those...
• Event subscription handler hangs after a while lexrpc#8
• [ ] backfill k256_pem into existing Web and ActivityPub users
• XRPC: https://lexrpc.readthedocs.io/, https://github.com/snarfed/lexrpc
  • [ ] Validate data against lexicon schemas (optional)
• Data conversion to/from app.bsky lexicons in granary:
• Bridgy Fed serves most app.bsky read XRPCs, eg https://fed.brid.gy/xrpc/app.bsky.feed.getAuthorFeed?author=snarfed.org
• finish unifying protocol.receive across protocols?
• Federation
  • MST
  • signed commit chain
  • blob serving
  • guarantee that commit chains we serve are immutable (except for rebases)
  • ingest/sync other PDSes' repos
  • [ ] DID resolution
  • [ ] did:web serving?
  • com.atproto.sync XRPC methods:
    • [ ] getBlob
    • [ ] getBlocks
    • getCheckout
    • [ ] getCommitPath
    • getHead
    • getRecord
    • getRepo
    • [ ] listBlobs
    • listRepos
    • [ ] notifyOfUpdate
    • [ ] requestCrawl
    • [ ] subscribeRepos
  • com.atproto.server XRPC methods:
    • [ ] createAccount
    • createSession
    • getSession
    • refreshSession
  • com.atproto.repo XRPC methods:
    • applyWrites
    • createRecord
    • deleteRecord
    • describeRepo
    • getRecord
    • listRecords
    • putRecord
    • [ ] rebaseRepo
    • [ ] uploadBlob
• UI
  • support multiple user (protocol) types
  • support Followers with different protocols

[snarfed]

Looks like https://bsky.social/ may be the backend serving the app.bsky XRPC API for the iOS app. Most of the XRPC methods require auth though, eg https://bsky.social/xrpc/app.bsky.actor.getProfile?user=snarfed@bsky.social.app returns 401 Authentication Required, and they haven't documented auth yet. Could probably dumpster dive for it in the repo if I get bored enough!

[snarfed]

Movement! They've been working out and implementing federation, eg in https://github.com/bluesky-social/indigo , and they're about to invite more people to the beta app.

[snarfed]

They still haven't published yet, but I'm digging into this based on com.atproto.sync.getRepo/updateRepo. Eg {services.atproto_pds.endpoint}/xrpc/com.atproto.sync.getRepo?did={did}&earliest={cid_last_retrieved_root}, and from their chat:

Assuming you're on the one.social PDS and your friend's is on two.social using the handle friend.two.social.
When you tell your PDS to follow your friend, the PDS will call the com.atproto.handle method on two.social with your friend's handle, will get their DID and the CID of their repo and can then call com.atproto.sync.{get,update}Repo to get their posts etc.
When your client then calls your PDS to show posts etc, your friend's posts should be included with the rest.

[snarfed]

First pass at MST implementation is done! Now working on implementing the com.atproto.sync XRPC methods..

[snarfed]

(PDS implementation is in https://github.com/snarfed/arroba/ )

[maietta]

Blue Sky, the same service that is lead by the guy who let the government access Twitter when he was in charge? I HIGHLY OPPOSE the integration of Blue Sky service because of this. Perhaps it's time to create a rolling fork that purposely removes Blue Sky and potentially other networks from the plugin.

Yeah, I'm unsubscribing from this product altogether. I won't support this in any way shape or form.

[snarfed]

Sorry to hear that! You're obviously welcome to fork or stop using Bridgy Fed at any time, that's your prerogative, and the beauty of both open source code and free (ie no cost to you) services like this.

Fwiw, Jack and others at Twitter initially kicked off the Bluesky project, but neither he nor Twitter have ever led it or had meaningful ownership or individual control over it since then. More: https://snarfed.org/bluesky-corporate-ownership-and-structure

[snarfed]

Moved to snarfed/arroba#2, and the BF-specific bits to #512.

[leo60228]

Is the intention just to allow following websites from Bluesky (like already live for ActivityPub), or to allow following Bluesky users from ActivityPub? If the latter, how will opt-in be handled?

I would personally find the ability to follow Bluesky users from Mastodon very useful, but I can see it being controversial for a number of reasons.

[snarfed]

Thanks for asking! The intent is to expand Bridgy Fed to be able to bridge any decentralized social protocol to any other protocol, eg web sites <=> Nostr, ActivityPub <=> Bluesky, and all other combinations. You can see some of the detailed background prep work in https://fed.brid.gy/docs#translate, along with tracking issues like #512 and #529.

Glad to hear you'd find it useful! And yup, I definitely know that it may be controversial. That's definitely not my intent, and I plan to be as careful as possible, and take all feedback into account, including experiences with existing bridges like https://mostr.pub/ ...but I expect I may still take some heat. I guess I'm ok with that.

[snarfed]

Reopening for tracking, since we haven't actually shipped this yet. 😆

[ghobs91]

Super cool! Just to clarify, is this specific issue meant to introduce bridging between ActivityPub and Bluesky?

[snarfed]

@ghobs91 yes! Contingent on Bluesky turning on federation in prod, of course.

Latest status: https://snarfed.org/2023-07-11_bridgy-fed-status-update-2

[snarfed]

Oh and notably this would bridge Bluesky to all of the protocols that BF supports. Right now that's just AP and web (microformats2 + webmention), but we hope to add others too, eg Nostr.

[snarfed]

Big milestone yesterday, https://github.com/snarfed/arroba is now successfully federating with the ATProto sandbox! Demo PDS is https://arroba-pds.appspot.com/ , user is arroba5.snarfed.org. Woot!

[snarfed]

Thinking through architecture here. With the other protocols we're working with so far, individual objects are at most loosely connected. We can federate profiles, follows, posts, etc independently of each other.

ATProto, on the other hand, ties all of a given user's objects together more tightly in their repo. When a non-ATProto user follows or otherwise interacts with a non-ATProto user, we can construct their repo on demand, but that's heavyweight. Should we construct them for everyone, ahead of time?

...except most of the time we'll be seeing a given non-ATProto user for the first time, so we need to be able to construct repos on the fly anyway. So maybe we just start with that, and see whether/how we can optimize later.

[snarfed]

Big milestone, shipped Bluesky => fediverse user discovery! Search for @[handle]@atproto.brid.gy (eg @jay.bsky.team@atproto.brid.gy) in any fediverse server.

[ghobs91]

Big milestone, shipped Bluesky => fediverse user discovery! Search for @[handle]@atproto.brid.gy (eg @jay.bsky.team@atproto.brid.gy) in any fediverse server.

Amazing! I only seem to be able to find @jay.bsky.team@atproto.brid.gy though, if i try any other bluesky handle it doesn't work.

[leo60228]

Same, WebFinger gives this response for other users:

<!doctype html>
<html lang=en>
<title>400 Bad Request</title>
<h1>Bad Request</h1>
<p>None is not a valid atproto id</p>

[snarfed]

Argh, looks like I deployed a bug that affects *.bsky.social handles specifically. Thanks for the nudge, fixing now.

[snarfed]

Previous progress reports: https://snarfed.org/2023-11-07_bridgy-fed-status-update-8, and the "previously" links at the bottom there.

[snarfed]

I guess I should leave this open until we can actually ship it 😎

[leo60228]

Is this functionality only enabled for some accounts? My Mastodon and ATProto sandbox accounts can see each other, but follows and posts don't seem to be going through in either direction.

[snarfed]

@leo60228 it's not on for any accounts, hence "until we can actually ship it" and "Not real Bluesky, only in the federation sandbox" 😁.

[leo60228]

I meant in the sandbox, I know that real Bluesky accounts won't work yet.

[snarfed]

Ah! No, it's not fully on in the sandbox either. I don't currently have plans to turn it back on there and leave it on, since the sandbox is just for testing and gets wiped semi-regularly. Glad you're interested though!

[ch0ccyra1n]

Is there anything I can do to contribute to this?

[snarfed]

Hey, thank you for offering! Sasquatch is awesome btw!

At an alpha level at least, this is basically done. We're just waiting for Bluesky team to turn on external federation. After they do that and we ship this, we'll discover a ton of bugs and missing things, as usual, but I'm not trying too hard to find all of those beforehand.

The one main thing I'm thinking about is #744, which currently caps us at 10k external accounts bridged into Bluesky. That sounds high, but I suspect it will arrive sooner than we expect.

Otherwise, feel free to look at the open issues and see if any of them grab you, I'd love the help!

[snarfed]

Oh, there's also #728, which somewhat blocks us from supporting reposts until bluesky-social/atproto#1811 is addressed. Feel free to evangelize or even look into fixing that!

[snarfed]

Bluesky is federating soon! "This month."

[ch0ccyra1n]

So I guess it's federating now.

[afontenot]

Just curious about the status of Bluesky support? Given this:

We're just waiting for Bluesky team to turn on external federation. After they do that and we ship this, we'll discover a ton of bugs and missing things, as usual, but I'm not trying too hard to find all of those beforehand.

Given that federation has been active over a month, I'm wondering when this will get activated. There are people on Bluesky I'd love to be able to follow from a Mastodon server (even passively, without fully functional federation), but this doesn't seem to be working yet, e.g. searching for @username.bsky.social@atproto.brid.gy from Mastodon doesn't return a profile for me to follow.

[snarfed]

@afontenot yes! Bluesky's federation is currently only in beta, limited to just 10 accounts per server, so we can't launch until they lift that limit.

We also have some work left to do here too, but that limit is probably still the main blocker right now.

[afontenot]

@snarfed thanks very much for the update!

[ch0ccyra1n]

Regarding task

decide whether we have a BF protocol per lexicon/app, eg Bluesky, or just one for all of ATProto. We'd prefer the latter, but we need some app-specific semantics. Not sure yet if we can avoid all of those...

hasn't a decision been de-facto already made, at least for the time-being?

[snarfed]

Hmm! Surprisingly no, I don't think so. We only have the one atproto BF protocol right now, which hard codes the app.bsky lexicons and semantics. If/when we want to support new lexicons, like eg the new long-form writing service being built on ATProto (can't remember the name), I still don't know how we'd do it in BF. Fortunately we don't need to figure that out any time soon.

[RandomDSdevel]

the new long-form writing service being built on ATProto (can't remember the name)

     I don't think this is what you were referring to, but the owner/author of the blog hosted at snorre.io also built their site's comment system on top of BlueSky.

[snarfed]

Fun! Always nice to see other backfeed implementations like https://brid.gy/ (non-Fed). Agreed though, that's different, and it also still uses the existing app.bsky.* lexicons.

[snarfed]

Adding more todos and manual tests to the description at the top here, pinning this issue.

[snarfed]

Closing this! Initial implementation is complete. I'm now tracking ongoing bugs and updates in the now label.