Announcing the iroh-blobs canary series

Today we are releasing iroh-blobs 0.104. It contains a number of bug fixes and only a few API changes, most of which won't affect users.

This will be the last release before we start doing larger API changes to prepare for iroh-blobs 1.0. The process will be somewhat similar to the canary series we did for iroh itself, but much shorter.

Since irpc is a public dependency of blobs, we will also release a 1.0 version of irpc soon.

Some fixes, like hash sequences not being loaded lazily (#275), became more urgent now that applications such as iroh global content discovery publish endpoint IDs globally. When endpoint IDs are public, a request can come from anywhere, so we cannot have requests that load an entire blob into memory. Also, now that iroh has reached 1.0, we have a much sturdier foundation for stabilizing the iroh-blobs API—which is what the canary series is about.

Others were discovered by the many external users of iroh-blobs. Originally, iroh-blobs was a core part of iroh and was used in iroh-docs and sendme, but these days there are so many external projects that it is hard to keep track. Fortunately, that means that not just obvious bugs but also bugs in rare corner cases get found.

Projects using iroh-blobs

Here are a few of the projects that use iroh-blobs today:

• deCDN is a peer-to-peer CDN for large files such as AI models and datasets. Nodes cache and serve content-addressed blobs over iroh. The deCDN team also sent us a comprehensive list of bugs for this release—thank you!
• DashBeam, previously AltSendme, is a cross-platform file transfer app with no accounts and no size limits, inspired by sendme.
• Flipflop does file sharing over persistent iroh tunnels between trusted devices.
• Dash Chat by darksoil studio is an offline-first peer-to-peer messenger built on p2panda and iroh.
• Radicle Artifact is Radicle's take on Git LFS: large files bound to commits, seeded and fetched peer-to-peer with iroh-blobs.

What is next

The next step is to make sure the public API of iroh-blobs is minimal, so we can support iroh-blobs 1.0 for a long time without breaking changes.

We also want to do some internal refactors prior to 1.0 and add an abstraction to make it easier to write new sync or async blob stores. The latter may or may not make it into 1.0, possibly under an experimental feature.

The changes in detail

New features

• #283 Downloader::builder allows connecting to providers with a custom ALPN, e.g. for protocols like p2panda that derive ALPNs from a private network ID. Downloader::new_with_opts is deprecated in favor of the builder. Based on work by @maackle.

Fixes

• #275 Reading the children of a hash seq loaded the entire root blob into memory, so a peer could make a provider run out of memory by requesting the children of a large blob. Provider and client get and push now read hash seqs lazily, in 1 KiB chunks. Stores now also send ExportRangesItem::Size first, as documented.
• #309 Decoding a RangeSpecSeq accepted non-canonical forms. Zero deltas produced duplicate offsets, and large deltas or widths overflowed, which panicked in debug builds and wrapped in release builds. Non-canonical encodings are now rejected. Our own encoders never produce them, so well-behaved peers are not affected.
• #295 A get into a local store could report success even though the store failed to import the data, e.g. when writing the final batch failed. This affected fetch, execute_get, execute_get_many and the downloader. The store error is now reported as GetError::LocalFailure.
• #297 A truncated export_ranges stream, e.g. because the store shut down mid-export, looked like a complete one, so concatenate() returned Ok with truncated data. The end of the stream is now inferred from the size and the requested ranges, and a stream that ends early is an error.
• #298 The same for export_bao: data_to_vec, data_to_bytes, write and stream() could silently return truncated data. The end is now inferred from the requested ranges instead of relying on the Done item.
• #293 Since tokio 1.53, dropping the fs store deadlocked, because the store's private runtime was dropped from one of its own threads. The runtime is now shut down in the background.
• #294 Observing a poisoned entry in the fs store, e.g. one whose data file was deleted, panicked and killed the entity actor for that hash, which stayed down until the store was restarted. Writing to a poisoned entry reported success while storing nothing. Observe streams now end for poisoned entries, writes fail with the actual io error, and the cause of poisoning is logged.
• #296 Re-adding the content of a poisoned entry reported success, but the blob stayed unreadable. A finished import now replaces a poisoned entry.
• #302 An entry with external data only tried the first of its paths, so it stayed poisoned if that file was deleted, even when the same content was available at another path. All paths are now tried.
• #317 It was possible for the garbage collector to remove the children of a hash seq root even though it was already marked as live, deleting data that should have been kept. Thanks to Damian Miles for finding this in production and contributing a test!
• #299 Blobs::add_stream hung forever when the source stream yielded an error. The source error is now returned to the caller.
• #282 Hash::from_str panicked on base32 input that does not decode to exactly 32 bytes. This was fixed before, but the fix got lost when the crate was replaced with the blobs2 code.
• #313 Bitfield::total_bytes underflowed for a range starting past the size: a panic in debug builds, a huge value in release builds.
• #285 The fs store used max_read_duration instead of max_write_duration to limit write batches, so changing max_write_duration had no effect. Thanks to Parth Sarthi Mishra!
• #280 iroh-blobs without the rpc feature failed to build when another crate in the build enabled irpc/rpc, e.g. in a wasm browser client using irpc-iroh.

Breaking changes

• #304 Hash::fmt_short now returns impl Display + Copy + 'static instead of arrayvec::ArrayString<10>, the same way iroh_base::PublicKey::fmt_short does. This removes the pre-1.0 arrayvec crate from our public API. If you used the result as a &str, add .to_string().
• #312 Update iroh-util to 0.7. iroh_blobs::util::connection_pool re-exports iroh_util::connection_pool, so its changes are visible to blobs users: Options and the error enums are now #[non_exhaustive], on_connected is set via Options::with_on_connected, connection attempts count against max_connections, and the new max_pending_connections (default 64) limits concurrent attempts, failing with PoolConnectError::TooManyPendingConnections at the cap.
• #307 Update bao-tree to 0.17. The serde impl of Parent announced a sequence of 2 elements but wrote 3, which broke export_bao and import_bao over rpc for blobs larger than 16 KiB. The fix changes the rpc wire format of Parent and of the io error in EncodeError, so rpc clients and servers need to be updated together.