iroh 0.29 - net is the new iroh

Welcome to a new release of iroh, a library for building on direct connections between devices, putting more control in the hands of your users.

We're excited to announce that the iroh crate has been streamlined, fulfilling the goal we outlined in our blog post, "Smaller is Better." Iroh is now leaner and more focused, serving as a core library for building direct connections between devices.

The protocols we've developed on top of iroh have been split into their own dedicated crates for iroh-blobs, iroh-docs, and iroh-gossip, respectively.

Additionally, iroh is no longer a CLI. We've also updated our terminology to better reflect its purpose: what we previously referred to as the iroh "Node" is now called the iroh "Router"—a more accurate name for its role in stacking protocols on top of iroh's peer-to-peer connections. All terminology is updated on our docs site to reflect these changes.

♻️ Upgrade Guide

With a smaller iroh, you get way more flexibility. You can still recreate the setups that were once required-bundled into iroh, and this blog will guide you through upgrading your code! We've swept out a bunch of code from the main iroh codebase. Here is an upgrade guide to the biggest changes in this new release, you can find a full list of breaking changes below.

iroh-net is now just iroh

This is a pretty straightforward migration that’s more spiritual than a change in functionality. If your code depends on iroh-net, change that dependency to iroh@v0.29.0, and change any instance where you import from iroh_net to iroh.

Protocol Registry

We now have a short list of the protocols we've factored out of iroh at https://iroh.computer/proto. We've moved blobs, docs, and gossip here, while also making room for some new protocols in the works from the number 0 team & others!

No more Node in iroh

Iroh previously provided you with a Node that came bundled with the iroh-docs, iroh-blobs, and iroh-gossip protocols enabled. This also meant that iroh was not only a networking library, but also responsible for storage for these different protocols. Now, we've shifted our thinking. The iroh library provides you with an endpoint, a light framework for implementing your own protocols, and a router that combines that endpoint and those protocols (as well as the n0-developed protocols).

That means we have no more Node in iroh. If you rely on the node, we've added migration instructions for how to get the n0-developed protocols into iroh, as well as custom protocols that you have developed yourself.

iroh::Endpoint

No matter what protocol(s) you use, you will start with building an iroh::Endpoint. This is what creates, drives, and hole-punches the connections that undergird the protocols.

Note that, by default, the endpoint::Builder will bind to 0.0.0.0:0 and disable discovery, so please check out the build options for more in-depth ways to configure the endpoint.

If you are converting straight from a default iroh::Node, make sure to include the Builder::discovery_n0 option, which will enable the discovery mechanisms it had on by default.

use iroh::Endpoint;
use anyhow::Result;

#[tokio::main]
async fn main() -> Result<()> {
	let endpoint = Endpoint::builder().discovery_n0().bind().await?
	println!("my node id: {}", endpoint.node_id());
	Ok(())
}

iroh-blobs

Two important parts are needed when building a Blobs protocol handler to use with the Router: a local pool and a store.

The local pool manages all tasks spawned when working with iroh-blobs. It makes sure they are completed and closed once dropped. When building a Blobs handler (or if you also need to build a Downloader or the iroh-docs protocol), you will pass a handle to the LocalPool. Make sure you don’t accidentally drop the local pool early, as that will close all the tasks before you mean to.

The store is where the blobs are kept, this can be loaded from a file or be an in-memory store.

use anyhow::Result;
use iroh::{protocol::Router, Endpoint};
use iroh_blobs::{net_protocol::Blobs, util::local_pool::LocalPool};

#[tokio::main]
async fn main() -> Result<()> {
    // create an iroh endpoint that includes the standard discovery mechanisms
    // we've built at number0
    let endpoint = Endpoint::builder().discovery_n0().bind().await?;
    println!("my node id: {}", endpoint.node_id());
    // spawns a local pool with one thread per CPU
    // for a single threaded pool use `LocalPool::single`
    let local_pool = LocalPool::default();

    // create an in-memory blob store
    // use `iroh_blobs::net_protocol::Blobs::persistent` to load or create a
    // persistent blob store from a path
    let blobs = Blobs::memory().build(local_pool.handle(), &endpoint);

    // turn on the "rpc" feature if you need to create blobs and tags clients
    let blobs_client = blobs.clone().client();
    let tags_client = blobs_client.tags();

    // build the router
    let router = Router::builder(endpoint)
        .accept(iroh_blobs::ALPN, blobs)
        .spawn();

    // make sure not to drop the local_pool before you are finished
    Ok(())
}

iroh-gossip

use std::sync::Arc;

use anyhow::Result;
use iroh::{protocol::Router, Endpoint};
use iroh_gossip::{net::Gossip, proto::Config};

#[tokio::main]
async fn main() -> Result<()> {
    // create an iroh endpoint that includes the standard discovery mechanisms
    // we've built at number0
    let endpoint = Endpoint::builder().discovery_n0().bind().await?;
    println!("my node id: {}", endpoint.node_id());

    // get your address
    let node_addr = endpoint.node_addr().await?;
    let addr_info = node_addr.info;

    // look at `iroh_gossip::proto::topic::Config`
    // for more configuration details.
    let config = Config::default();

    // create gossip
    let gossip = Arc::new(Gossip::from_endpoint(endpoint.clone(), config, &addr_info));

    // turn on the "rpc" feature to use the gossip client
    let gossip_client = gossip.client();

    // create router and add gossip protocol
    let router = Router::builder(endpoint)
        .accept(iroh_gossip::ALPN, gossip)
        .spawn().await?;

    Ok(())
}

iroh-docs

Setting up the docs protocol is a funny little beast. It’s a “meta protocol” that includes both the iroh-blobs and iroh-gossip protocols and so requires some involved set up.

use std::{path::PathBuf, sync::Arc};

use anyhow::Result;
use iroh::{Endpoint, protocol::Router};
use iroh_blobs::{
    downloader::Downloader, net_protocol::Blobs, util::local_pool::LocalPool,
};
use iroh_docs::engine::Engine as Docs;
use iroh_gossip::net::Gossip;

#[tokio::main]
async fn main() -> Result<()> {
    // local thread pool manager for blobs and docs
    let local_pool = LocalPool::default();

    // create endpoint
    let endpoint = Endpoint::builder()
        .discovery_n0()
        .bind()
        .await?;

    // build the protocol router
    let mut builder = Router::builder(endpoint);

    // add iroh-gossip
    let addr = builder.endpoint().node_addr().await?;
    let gossip = Gossip::from_endpoint(
        builder.endpoint().clone(),
        Default::default(),
        &addr.info,
    );
    builder = builder.accept(iroh_gossip::ALPN, Arc::new(gossip.clone()));

    // add iroh-blobs
    let blobs = Blobs::memory().build(local_pool.handle(), builder.endpoint());
    builder = builder.accept(iroh_blobs::ALPN, blobs.clone());

    // add iroh-docs

    // setup docs storage
    let docs_store = iroh_docs::store::Store::memory();
    let author_store = iroh_docs::engine::DefaultAuthorStorage::Mem;

    let docs = Arc::new(Docs::spawn(
        builder.endpoint().clone(),
        gossip,
        docs_store,
        blobs.store().clone(),
        blobs.downloader().clone(),
        author_store,
        local_pool.handle().clone(),
    )
    .await?);
    builder = builder.accept(iroh_docs::ALPN, docs.clone());

    // spawn the router
    let router = builder.spawn().await?;

    // enable the `rpc` feature to get the memory rpc client
    let docs_client = docs.client().clone();
    let authors_client = docs_client.authors();

    Ok(())
}

Your very own protocol

It’s straightforward to add your own protocol to the iroh Router.

use iroh::{Endpoint, protocols::Router};

mod cool_protocol {
    /// The ALPN
    pub const ALPN: &[u8] = b"/my/cool/protocol/1";

    pub struct Protocol;

    /// Implement ProtocolHandler
    impl iroh::protocol::ProtocolHandler {
        /// Needs to at least implement the `accept` function
        fn accept(self: Arc<Self>, conn: iroh::endpoint::Connecting) -> BoxedFuture<anyhow::Result<()>> {
            Box::pin(async move {
                // Here you implement what your protocol does when a new connection is attempted
                println!("got a new incoming connection");
            })
        }
    }
}

// create your protocol
let my_cool_protocol = Arc::new(cool_protocol::Protocol::new());

// build an endpoint
let endpoint = Endpoint::builder()
    .discovery_n0()
    .bind()
    .await?;

// build a router that connects the endpoint to the protocol
let router = Router::builder(endpoint)
    .accept(cool_protocol::ALPN, my_cool_protocol.clone())
    .spawn()
    .await?;

// from here, you can use iroh.endpoint().connect() to connect via a NodeId
// or use your protocol to handle incoming connections

For a full example of a custom protocol take a look at the echo protocol.

If you are looking for examples of how to create an RPC client for your protocol using quic-rpc, take a look at the iroh-gossip repo for a simple example and the iroh-blobs repo for a more complex one that includes streaming rpc (server side & bidirectional).

And that’s it! Full list of breaking changes follows. This is a big release the n0 team has been working toward on our road to 1.0, and can’t wait for you to try it out!

⚠️ Breaking Changes

• net-tools now has its own repo, which includes portmapper and netwatch

• iroh

  • changed
    • module iroh::client::net is now a reexport of iroh_node_util::rpc::client::net
    • module iroh::client::node is now a reexport of iroh_node_util::rpc::client::node
    • fn iroh::client::Iroh::net returns a net::Client instead of a &net::Client
    • fn iroh::client::Iroh::node returns a node::Client instead of a &node::Client
    • iroh-net got renamed to iroh
    • iroh-router moved into iroh
    • iroh-router::ProtocolHandler is now iroh::protocol::ProtocolHandler
    • iroh-router::ProtocolMap is now iroh::protocol::ProtocolMap
    • iroh-router::Router is now iroh::protocol::Router
    • iroh-router::RouterBuilder is now iroh::protocol::RouterBuilder
    • iroh-net's NetcheckMetrics are now called NetReportMetrics
    • iroh::Endpoint::close that takes no arguments now, it default to using code 0 and an empty message
    • iroh::Endpoint::close now takes &self rather than self. This can, in some situations, mean an existing (clone of an) endpoint might be dropped too early as a temporary variable.
    • iroh::test_utils::run_relay_server_with(stun: Option<StunConfig>) => iroh::test_utils::run_relay_server_with(stun: Option<StunConfig>, quic: bool)
    • Events are emitted on different tracing targets: iroh::events instead of events.net.
    • iroh::node::Builder has no more generic parameters anymore
    • iroh::node::Node has no more generic parameters anymore
    • iroh::protocol::Router::shutdown takes &self instead of self
    • RouterBuilder::accept takes impl AsRef<[u8]>. Existing code should still work!
    • Node::accept takes impl AsRef<[u8]>. Existing code should still work!
  • removed
    • iroh::cli, look at iroh-blobs, iroh-docs, and iroh-doctor for cli examples
    • iroh::node, use iroh-node-utils
    • iroh::metrics, use iroh-metrics
    • iroh::blobs use iroh-blobs crate
    • iroh::docs use iroh-docs crate
    • iroh::gossip use iroh-gossip crate
    • iroh::util
    • the ability to run iroh itself in a docker container, as there is no binary anymore
    • iroh::client::blobs use iroh_blobs::rpc::client. a memory client is available on Blobs
    • iroh::client::tags use iroh_blobs::rpc::client. a memory client is available on Blobs
    • iroh::client::gossip use iroh_gossip::rpc::client. a memory client is available on Gossip
    • iroh::client::docs use iroh_docs::rpc::client. a memory client is available on Docs
    • iroh::client::authors use iroh_docs::rpc::client. a memory clientis available on Docs
    • iroh::node::MemNode, use Node directly
    • iroh::node::FsNode, use Node directly
    • iroh::node::Node::local_pool_handle
    • iroh::node::builder::DocsStorage
    • iroh::node::builder::Builder::enable_gc_policy
    • iroh::node::builder::Builder::enable_docs
    • iroh::node::builder::Builder::register_cb_done
    • iroh::node::builder::ProtocolBuilder::local_pool_handle
    • iroh::node::builder::GcPolicy
    • iroh::util::progress
    • iroh::util::path::IrohPaths::BaoStoreDir
    • iroh::util::path::IrohPaths::DocsDatabase
    • iroh::util::path::IrohPaths::Console
    • iroh::util::path::IrohPaths::DefaultAuthor
    • util::fs::PathContent, use iroh_blobs::util::fs::PathContent
    • util::fs::path_content_info, use iroh_blobs::util::fs::path_content_info
    • util::fs::key_to_path , use iroh_blobs::util::fs::key_to_path
    • util::fs::path_to_key, use iroh_blobs::util::fs::path_to_key
    • util::fs::canonicalized_path_to_string, use iroh_blobs::util::fs::canonicalized_path_to_string
    • util::io::*, use iroh_blobs::util::io
    • util::progress::ProgressEmitter
    • util::progress::ProgressAsyncReader
    • util::progress::Progress
    • util::progress::ProgressReader
    • util::progress::ProgressReaderUpdate
  • added
    • added iroh::Endpoint::is_closed

• iroh-node-util

  • module iroh-cli::commands::net moved to iroh-node-util::cli::net
  • logic in iroh-cli::commands::rpc moved to iroh-node-util::cli::node
  • module iroh_cli::logging has moved to iroh-node-util to make it available for other people building nodes
  • iroh_config_root, iroh_data_root and iroh_cache_root in iroh-cli have been replaced with generic config_root, data_root and cache_root in iroh-node-util::config
  • load_secret_key moved to iroh_node_utils to preserve it

• iroh-relay

  • changed The URLs served by the relay changed:
    • /relay/probe has moved to /ping
    • /derp/probe has been removed. Unless you were manually using those URLs you will not notice these changes, nothing in the iroh codebase ever used the changed URLs.
    • RelayMap now can be created with an iterator of Arcs directly.
    • iroh-relay now uses NodeGone instead of PeerGone in some enums
    • If not configured there is now a default rate limit for incoming data from client connections: 4KiB/s steady-stream and 16MiB burst capacity.
  • removed
    • iroh_net::relay is removed. RelayUrl, RelayMode, RelayNode and RelayMap are moved to the top (iroh_net). All other members of this module are now moved to the new crate iroh-relay
    • field config has been removed from variant iroh_relay::server::CertConfig::LetsEncrypt
    • variant iroh_relay::server::CertConfig::Manual no longer has field private_key
  • added
    • iroh_base::relay_map::RelayNode now has field quic that takes a Option<iroh_base::relay_map::QuicConfig>
    • iroh::test_utils::run_relay_server_with(stun: Option<StunConfig>) => iroh::test_utils::run_relay_server_with(stun: Option<StunConfig>, quic: bool), when quic is true, it will start a quic server for QUIC address discovery, that has self signed tls certs for testing.
    • iroh_relay::server::ServerConfig has new field quic that takes a Option<iroh_relay::server::QuicConfig>
    • iroh_relay::server::TlsConfig.quic_bind_addr is a new field that takes a SocketAddr
    • iroh_relay::server::TlsConfig.server_config is a new field that takes a rustls::ServerConfig
    • variant iroh_relay::server::CertConfig::LetsEncrypt has a new field state that takes a tokio_rustls_acme::AcmeState<EC, EA>

But wait, there's more!

Many bugs were squashed, and smaller features were added. For all those details, check out the full changelog: https://github.com/n0-computer/iroh/releases/tag/v0.29.0.

If you want to know what is coming up, check out the v0.30.0 milestone, and if you have any wishes, let us know about the issues! If you need help using iroh or just want to chat, please join us on discord! And to keep up with all things iroh, check out our Twitter.