\input texinfo   @c -*-texinfo-*-
@c %**start of header
@setfilename gnunet.info
@documentencoding UTF-8
@ifinfo
@*Generated by Sphinx 8.1.3.@*
@end ifinfo
@settitle GNUnet
@defindex ge
@paragraphindent 0
@exampleindent 4
@finalout
@dircategory Miscellaneous
@direntry
* gnunet: (gnunet.info). One line description of project
@end direntry

@c %**end of header

@copying
@quotation
GNUnet , Sep 17, 2025

GNUnet Project

Copyright @copyright{} 2024, GNUnet Project
@end quotation

@end copying

@titlepage
@title GNUnet
@insertcopying
@end titlepage
@contents

@c %** start of user preamble

@c %** end of user preamble

@ifnottex
@node Top
@top GNUnet
@insertcopying
@end ifnottex

@c %**start of body
@anchor{index doc}@anchor{0}
@menu
* About GNUnet:: 
* Installation:: 
* User Manual:: 
* Developer Handbook:: 
* GUIs:: 
* FAQs:: 
* Index:: 

@detailmenu
 --- The Detailed Node Listing ---

About GNUnet

* Project governance:: 
* Philosophy:: 
* Key Concepts:: 

Philosophy

* Design Principles:: 
* Privacy and Anonymity:: 
* Practicality:: 

Key Concepts

* Cryptography:: 
* Anonymity:: 
* Deniability:: 
* Accounting to Encourage Resource Sharing:: 

Cryptography

* Peer Identities:: 
* Egos:: 

Egos

* Zones in the GNU Name System:: 

Anonymity

* How file-sharing achieves Anonymity:: 
* How messaging provided Anonymity:: 

Installation

* Dependencies:: 
* Getting the Source Code:: 
* Create user and groups for the system services:: 
* Preparing and Compiling the Source Code:: 
* Minimal configuration:: 

Preparing and Compiling the Source Code

* NSS plugin (optional): NSS plugin optional. 
* Installing the GNS Certificate Authority (Optional): Installing the GNS Certificate Authority Optional. 

User Manual

* Getting started:: 
* Subsystems:: 
* The GNU Name System:: 
* re;claimID: re claimID. 
* File-sharing:: 
* Virtual Public Network:: 
* Messenger:: 
* Advanced Configuration:: 

Subsystems

* CADET - Decentralized End-to-end Transport:: 
* CORE - GNUnet link layer:: 
* DHT - Distributed Hash Table:: 
* FS — File sharing over GNUnet:: 
* NAMECACHE — DHT caching of GNS results:: 
* NAMESTORE — Storage of local GNS zones:: 
* HOSTLIST — HELLO bootstrapping and gossip:: 
* IDENTITY — Ego management:: 
* MESSENGER — Room-based end-to-end messaging:: 
* NSE — Network size estimation:: 
* PEERINFO — Persistent HELLO storage:: 
* PEERSTORE — Extensible local persistent data storage:: 
* REGEX — Service discovery using regular expressions:: 
* REST — RESTful GNUnet Web APIs:: 
* REVOCATION — Ego key revocation:: 
* RPS — Random peer sampling:: 
* STATISTICS — Runtime statistics publication:: 
* TRANSPORT-NG — Next-generation transport management:: 
* TRANSPORT — Overlay transport management:: 
* SET — Peer to peer set operations (Deprecated): SET — Peer to peer set operations Deprecated. 
* SETI — Peer to peer set intersections:: 
* SETU — Peer to peer set unions:: 

CORE - GNUnet link layer

* When is a peer "connected"?:: 

DHT - Distributed Hash Table

* Block library and plugins:: 

Block library and plugins

* What is a Block?:: 

FS — File sharing over GNUnet

* GNS - the GNU Name system:: 

HOSTLIST — HELLO bootstrapping and gossip

* HELLOs:: 
* Overview for the HOSTLIST subsystem:: 
* Interacting with the HOSTLIST daemon:: 
* Hostlist security address validation:: 
* The HOSTLIST client:: 
* Usage:: 

Overview for the HOSTLIST subsystem

* Features:: 
* HOSTLIST - Limitations:: 

Hostlist security address validation

* The HTTP Server:: 
* Advertising the URL:: 

The HOSTLIST client

* Bootstrapping:: 

NSE — Network size estimation

* Motivation:: 

Motivation

* Example:: 
* Algorithm:: 
* Target value:: 
* Timing:: 
* Controlled Flooding:: 
* Calculating the estimate:: 

PEERINFO — Persistent HELLO storage

* PEERINFO - Features:: 
* PEERINFO - Limitations:: 
* DeveloperPeer Information:: 
* Startup:: 
* Managing Information:: 
* Obtaining Information:: 

PEERSTORE — Extensible local persistent data storage

* Functionality:: 
* Architecture:: 

REGEX — Service discovery using regular expressions

* How to run the regex profiler:: 

REVOCATION — Ego key revocation

* Dissemination:: 
* Revocation Message Design Requirements:: 

RPS — Random peer sampling

* Brahms:: 

TRANSPORT-NG — Next-generation transport management

* Design goals of TNG:: 
* HELLO-NG:: 
* Priorities and preferences:: 
* Communicators:: 

Communicators

* UDP communicator:: 
* TCP communicator:: 
* QUIC communicator:: 

SET — Peer to peer set operations (Deprecated)

* Local Sets:: 
* Set Modifications:: 
* Set Operations:: 
* Result Elements:: 

SETI — Peer to peer set intersections

* Intersection Sets:: 
* Set Intersection Modifications:: 
* Set Intersection Operations:: 
* Intersection Result Elements:: 

SETU — Peer to peer set unions

* Union Sets:: 
* Set Union Modifications:: 
* Set Union Operations:: 
* Union Result Elements:: 

The GNU Name System

* Start Zones:: 
* Local zone maintenance:: 
* Application integration:: 
* Zone dissemination:: 
* Resource Records:: 
* DNS Migration:: 

Start Zones

* Remote:: 
* Local:: 
* Zone Top-Level-Domains:: 

Local zone maintenance

* Orphaning:: 
* Records:: 
* Revocation:: 
* Backup:: 
* Large Zones:: 

Application integration

* Integration with Browsers (DNS2GNS service): Integration with Browsers DNS2GNS service. 
* Integration with Browsers (SOCKS proxy): Integration with Browsers SOCKS proxy. 
* NSS plugin:: 

Zone dissemination

* The .pin zone: The pin zone. 
* Be Social:: 
* Creating a business card:: 
* The GNUnet default Start Zones:: 
* The GNUnet GNS Registrar:: 

Resource Records

* VPN:: 

DNS Migration

* Zoneimport:: 
* Zonefile:: 
* Ascension:: 

re:claimID

* Managing Attributes:: 
* Managing Credentials:: 
* Sharing Attributes with Third Parties:: 
* Revoking Authorizations of Third Parties:: 
* OpenID Connect:: 
* Providing Third Party Attestation:: 

File-sharing

* Searching:: 
* Downloading:: 
* Publishing:: 
* Indexing vs Inserting:: 
* Concepts:: 
* Namespace Management:: 
* File-Sharing URIs:: 

Concepts

* Files:: 
* Keywords:: 
* Directories:: 
* Egos and File-Sharing:: 
* Namespaces:: 
* Advertisements:: 
* Anonymity level:: 
* Content Priority:: 
* Replication:: 

Namespace Management

* Creating Egos:: 
* Deleting Egos:: 

File-Sharing URIs

* Encoding of hash values in URIs:: 
* Content Hash Key (chk): Content Hash Key chk. 
* Location identifiers (loc): Location identifiers loc. 
* Keyword queries (ksk): Keyword queries ksk. 
* Namespace content (sks): Namespace content sks. 

Virtual Public Network

* Setting up an Exit node:: 
* Fedora and the Firewall:: 
* Setting up VPN node for protocol translation and tunneling:: 

Messenger

* Current state:: 
* Entering a room:: 
* Opening a room:: 
* Messaging in a room:: 
* Private messaging:: 

Advanced Configuration

* Config file format:: 
* The Single-User Setup:: 
* The Multi-User Setup:: 
* Access Control for GNUnet:: 
* Configuring the hostlist to bootstrap:: 
* Disable default bootstrap (private network): Disable default bootstrap private network. 
* Manually connecting peers:: 
* Configuration of the HOSTLIST proxy settings:: 
* Configuring your peer to provide a hostlist:: 
* Configuring the datastore:: 
* Configuring the Postgres database:: 
* Reasons to use Postgres:: 
* Reasons not to use Postgres:: 
* Manual setup instructions:: 
* Testing the setup manually:: 
* Configuring the datacache:: 
* Configuring the file-sharing service:: 
* Configuring logging:: 
* Configuring the transport service and plugins:: 
* Configuring the WLAN transport plugin:: 
* Configuring HTTP(S) reverse proxy functionality using Apache or nginx: Configuring HTTP S reverse proxy functionality using Apache or nginx. 
* Blacklisting peers:: 
* Configuration of the HTTP and HTTPS transport plugins:: 
* Configuring the GNUnet VPN:: 
* Bandwidth Configuration:: 
* Configuring NAT:: 
* Peer configuration for distributors (e.g. Operating Systems): Peer configuration for distributors e g Operating Systems. 

Access Control for GNUnet

* Recommendation - Disable access to services via TCP:: 
* Recommendation - Run most services as system user "gnunet":: 
* Recommendation - Control access to services using group "gnunet":: 
* Recommendation - Limit access to certain SUID binaries by group "gnunet":: 
* Recommendation - Limit access to critical gnunet-helper-dns to group "gnunetdns":: 
* Differences between "make install" and these recommendations:: 

Configuring the WLAN transport plugin

* Requirements for the WLAN plugin:: 
* Configuration:: 
* Before starting GNUnet:: 
* Limitations and known bugs:: 

Configuring HTTP(S) reverse proxy functionality using Apache or nginx

* Reverse Proxy - Configure your Apache2 HTTP webserver:: 
* Reverse Proxy - Configure your Apache2 HTTPS webserver:: 
* Reverse Proxy - Configure your nginx HTTPS webserver:: 
* Reverse Proxy - Configure your nginx HTTP webserver:: 
* Reverse Proxy - Configure your GNUnet peer:: 

Configuring the GNUnet VPN

* IPv4 address for interface:: 
* IPv6 address for interface:: 
* Configuring the GNUnet VPN DNS:: 
* Configuring the GNUnet VPN Exit Service:: 
* IP Address of external DNS resolver:: 
* IPv4 address for Exit interface:: 
* IPv6 address for Exit interface:: 

Developer Handbook

* Contributing:: 
* Style and Workflow:: 
* libgnunetutil:: 
* System Architecture:: 
* Cryptographic Material:: 
* Subsystem stability:: 
* Service APIs:: 
* REST API:: 
* C Tutorial:: 
* Specifications:: 
* Doxygen:: 

Contributing

* Licenses of contributions:: 
* Minor contributions:: 
* Major contributions:: 
* Copyright Assignment:: 
* Reference Manual:: 

Style and Workflow

* Coding style:: 
* Naming conventions:: 
* Continuous integration:: 
* Developer access and commit messages:: 
* Releases:: 

Naming conventions

* Header files:: 
* Binaries:: 
* Logging:: 
* Symbols:: 
* Tests:: 

Service APIs

* CADET:: 
* CORE:: 
* CONG:: 
* DHT: DHT<2>. 
* FS:: 
* GNS: GNS<2>. 
* IDENTITY:: 
* MESSENGER:: 
* NAMECACHE:: 
* NAMESTORE:: 
* NSE:: 
* PEERSTORE:: 
* REST:: 
* REVOCATION:: 
* Peer-to-Peer Set Operations:: 
* STATISTICS:: 
* TRANSPORT:: 

CORE

* The CORE Client-Service Protocol:: 
* The CORE Peer-to-Peer Protocol:: 

The CORE Client-Service Protocol

* Setup2:: 
* Notifications:: 
* Sending:: 

The CORE Peer-to-Peer Protocol

* Creating the EphemeralKeyMessage:: 
* Establishing a connection:: 
* Encryption and Decryption:: 
* Type maps:: 

CONG

* Key exchange:: 
* Unified Address Format for L2O and libp2p:: 
* Peer IDs:: 
* libp2p Underlay:: 
* Protocol-Versioning:: 
* Open Design Questions:: 
* Requirements:: 
* Use Cases and Scenarios:: 

Key exchange

* Next Steps:: 
* Handshake Protocol (Current): Handshake Protocol Current. 
* Draft; CORE Authenticated Key Exchange (CAKE): Draft CORE Authenticated Key Exchange CAKE. 

Draft: CORE Authenticated Key Exchange (CAKE)

* First Message (RTT=0): First Message RTT=0. 
* Second Message (RTT=0.5): Second Message RTT=0 5. 
* Third Message (RTT=1.5): Third Message RTT=1 5. 
* Confirmation (RTT=1.5): Confirmation RTT=1 5. 
* Rekey / Service Status change:: 
* Edge Cases:: 
* Glossary:: 

Peer IDs

* Reasoning:: 
* Implications:: 
* Details on how:: 
* Open questions:: 
* Ownership of Peer IDs:: 
* Implications: Implications<2>. 
* Details on how: Details on how<2>. 

Reasoning

* Attacker Model:: 

Implications

* DHT:: 
* Scope of Peer ID for higher Layers:: 
* Reconnects:: 
* Messenger: Messenger<2>. 

Implications

* Transport:: 
* Other:: 

Open Design Questions

* Peer ID changes and connectivity:: 
* Peer ID Terminology:: 

Use Cases and Scenarios

* Peer ID:: 
* GNS:: 
* CADET: CADET<2>. 
* Messenger: Messenger<3>. 

DHT

* The API of libgnunetblock:: 
* Queries:: 
* Sample Code:: 
* Conclusion2:: 
* GET:: 
* PUT:: 
* MONITOR:: 
* DHT Routing Options:: 
* PUTting data into the DHT:: 
* GETting data from the DHT:: 
* Monitoring the DHT:: 
* Routing GETs or PUTs:: 
* PUTting data into the DHT: PUTting data into the DHT<2>. 
* GETting data from the DHT: GETting data from the DHT<2>. 

DHT Routing Options

* The DHT Client-Service Protocol:: 

Monitoring the DHT

* The DHT Peer-to-Peer Protocol:: 

FS

* Namespace Advertisements:: 
* KSBlocks:: 

KSBlocks

* File-sharing persistence directory structure:: 

GNS

* Looking up records:: 
* Accessing the records:: 
* Creating records:: 
* Future work:: 
* Value handling:: 
* Type handling:: 
* Network Setup Details:: 
* Conversions between DNS and GNS:: 
* DNS Zone Size:: 
* Performance:: 

Future work

* libgnunetgnsrecord:: 

Type handling

* GNS plugins:: 
* The GNS Client-Service Protocol:: 
* Hijacking the DNS-Traffic using gnunet-service-dns:: 

Network Setup Details

* Importing DNS Zones into GNS:: 

IDENTITY

* Connecting to the service:: 
* Operations on Egos:: 
* The anonymous Ego:: 
* Convenience API to lookup a single ego:: 

Convenience API to lookup a single ego

* The IDENTITY Client-Service Protocol:: 

MESSENGER

* Messages:: 
* Member sessions:: 

Messages

* NAME-message:: 
* INVITE-message:: 
* TEXT-message:: 
* FILE-message:: 
* DELETE-message:: 
* TICKET-message:: 
* TAG-message:: 
* SUBSCRIBE-message:: 
* TALK-message:: 

NAMECACHE

* The NAMECACHE Client-Service Protocol:: 
* The NAMECACHE Plugin API:: 

The NAMECACHE Client-Service Protocol

* Lookup:: 
* Store:: 

The NAMECACHE Plugin API

* Lookup2:: 
* Store2:: 

NAMESTORE

* Editing Zone Information:: 
* Transactional operations:: 
* Iterating Zone Information:: 
* Monitoring Zone Information:: 

NSE

* Results:: 
* Examples:: 

Examples

* The NSE Client-Service Protocol:: 
* The NSE Peer-to-Peer Protocol:: 

PEERSTORE

* libgnunetpeerstore:: 

REST

* Namespace considerations:: 
* Endpoint documentation:: 

REVOCATION

* Querying for revoked keys:: 
* Preparing revocations:: 
* Issuing revocations:: 

Issuing revocations

* The REVOCATION Client-Service Protocol:: 
* The REVOCATION Peer-to-Peer Protocol:: 

Peer-to-Peer Set Operations

* SET:: 
* SETI:: 
* SETU:: 

SET

* Sets:: 
* Listeners:: 
* Operations:: 
* Supplying a Set:: 
* The Result Callback:: 
* Creating Sets:: 
* Listeners: Listeners<2>. 
* Initiating Operations:: 
* Modifying Sets:: 
* Results and Operation Status:: 
* Iterating Sets:: 
* The Bloom filter exchange:: 
* Salt:: 

The Result Callback

* The SET Client-Service Protocol:: 

Iterating Sets

* The SET Intersection Peer-to-Peer Protocol:: 

Salt

* The SET Union Peer-to-Peer Protocol:: 

SETI

* Intersection Set API:: 
* Intersection Listeners:: 
* Intersection Operations:: 
* Supplying a Set for Intersection:: 
* The Intersection Result Callback:: 
* Creating Intersection Sets:: 
* Listeners for Intersection:: 
* Initiating Intersection Operations:: 
* Modifying Intersection Sets:: 
* Intersection Results and Operation Status:: 
* The Bloom filter exchange in SETI:: 
* Intersection Salt:: 

The Intersection Result Callback

* The SETI Client-Service Protocol:: 

Intersection Results and Operation Status

* The SETI Intersection Peer-to-Peer Protocol:: 

SETU

* Union Set API:: 
* Union Listeners:: 
* Union Operations:: 
* Supplying a Set for Union:: 
* The Union Result Callback:: 
* Creating Union Sets:: 
* Listeners for Union:: 
* Initiating Union Operations:: 
* Modifying Union Sets:: 
* Union Results and Operation Status:: 
* The SETU Union Peer-to-Peer Protocol:: 

The Union Result Callback

* The SETU Client-Service Protocol:: 

STATISTICS

* Statistics retrieval:: 
* Setting statistics and updating them:: 
* Watches:: 
* Statistics retrieval: Statistics retrieval<2>. 
* Setting and updating statistics:: 
* Watching for updates:: 

Watches

* The STATISTICS Client-Service Protocol:: 

TRANSPORT

* Address validation protocol:: 
* NAT library:: 
* Distance-Vector plugin:: 
* SMTP plugin:: 
* Why use SMTP for a peer-to-peer transport?:: 
* How does it work?:: 
* How do I configure my peer?:: 
* How do I test if it works?:: 
* How fast is it?:: 
* Bluetooth plugin:: 
* What do I need to use the Bluetooth plugin transport?:: 
* How does it work2?:: 
* What possible errors should I be aware of?:: 
* How do I configure my peer2?:: 
* How can I test it?:: 
* The implementation of the Bluetooth transport plugin:: 
* WLAN plugin:: 

The implementation of the Bluetooth transport plugin

* Linux functionality:: 
* THE INITIALIZATION:: 
* THE LOOP:: 
* Details about the broadcast implementation:: 
* Pending features:: 

REST API

* Configuration: Configuration<2>. 
* Conventions:: 
* Identity:: 
* Namestore:: 
* GNS: GNS<3>. 
* Peerinfo:: 
* Config:: 

Identity

* Definition:: 
* Requests:: 

Definition

* Identity: Identity<2>. 
* Error Response:: 
* Response Code:: 

Requests

* GET Requests:: 
* POST Request:: 
* PUT Request:: 
* DELETE Request:: 
* OPTIONS Request:: 

Namestore

* Data model:: 
* Requests: Requests<2>. 

GNS

* Error Response: Error Response<2>. 
* GET Request:: 
* OPTIONS Request: OPTIONS Request<2>. 

Error Response

* Requests: Requests<3>. 

Peerinfo

* Definition: Definition<2>. 
* Requests: Requests<4>. 

Definition

* Peer:: 
* Peerinfo Response:: 
* Error Response: Error Response<3>. 

Requests

* GET Request: GET Request<2>. 
* OPTIONS Request: OPTIONS Request<3>. 

Config

* Definition: Definition<3>. 
* Requests: Requests<5>. 

Definition

* Configuration: Configuration<3>. 
* Error Response: Error Response<4>. 
* Response Code: Response Code<2>. 

Requests

* GET Requests: GET Requests<2>. 
* POST Request: POST Request<2>. 
* OPTIONS Request: OPTIONS Request<4>. 

C Tutorial

* Introduction to GNUnet Architecture:: 
* First Steps with GNUnet:: 
* Developing Applications:: 
* GNU Free Documentation License:: 

First Steps with GNUnet

* Configure your peer:: 
* Start a peer:: 
* Monitor a peer:: 
* Starting Two Peers by Hand:: 
* Starting Peers Using the Testbed Service:: 

Starting Two Peers by Hand

* Setup a second peer:: 
* Start the second peer and connect the peers:: 
* How to connect manually:: 

Developing Applications

* gnunet-ext:: 
* Adapting the Template:: 
* Writing a Client Application:: 
* Writing a Service:: 
* Interacting directly with other Peers using the CORE Service:: 
* Storing peer-specific data using the PEERSTORE service:: 
* Using the DHT:: 
* Debugging with gnunet-arm:: 

Writing a Client Application

* Handling command-line options:: 
* Writing a Client Library:: 
* Writing a user interface:: 

Writing a Client Library

* Connecting to the Service:: 
* Sending messages:: 
* Receiving Replies from the Service:: 

Writing a Service

* Code Placement:: 
* Starting a Service:: 

Interacting directly with other Peers using the CORE Service

* New P2P connections:: 
* Receiving P2P Messages:: 
* Sending P2P Messages:: 
* End of P2P connections:: 

Storing peer-specific data using the PEERSTORE service

* Storing records:: 
* Retrieving records:: 
* Monitoring records:: 
* Disconnecting from PEERSTORE:: 

Using the DHT

* Storing data in the DHT:: 
* Obtaining data from the DHT:: 
* Implementing a block plugin:: 
* Monitoring the DHT: Monitoring the DHT<2>. 

Implementing a block plugin

* Validating requests and replies:: 
* Deriving a key from a reply:: 
* Initialization of the plugin:: 
* Shutdown of the plugin:: 
* Integration of the plugin with the build system:: 

GUIs

* The graphical configuration interface:: 
* GTK File-sharing User Interface:: 

The graphical configuration interface

* Configuring your peer:: 
* Configuration of the HOSTLIST proxy settings: Configuration of the HOSTLIST proxy settings<2>. 
* Configuration of the HTTP and HTTPS transport plugins: Configuration of the HTTP and HTTPS transport plugins<2>. 

GTK File-sharing User Interface

* Publishing: Publishing<2>. 
* Searching: Searching<2>. 
* Downloading: Downloading<2>. 

FAQs

* General:: 
* Features: Features<2>. 
* GNU Name System:: 
* Error messages:: 
* File-sharing: File-sharing<2>. 
* Contributing: Contributing<2>. 

@end detailmenu
@end menu

@node About GNUnet,Installation,Top,Top
@anchor{about doc}@anchor{1}@anchor{about about-gnunet}@anchor{2}@anchor{about welcome-to-gnunets-documentation}@anchor{3}
@chapter About GNUnet


GNUnet in its current version is the result of over 20 years of work
from many contributors. So far, most contributions were made by
volunteers or people paid to do fundamental research. At this stage,
GNUnet remains an experimental system where significant parts of the
software lack a reasonable degree of professionalism in its
implementation. Furthermore, we are aware of a significant number of
existing bugs and critical design flaws, as some unfortunate early
design decisions remain to be rectified. There are still known open
problems; GNUnet remains an active research project.

The project was started in 2001 when some initial ideas for improving
Freenet’s file-sharing turned out to be too radical to be easily
realized within the scope of the existing Freenet project. We lost our
first contributor on 11.9.2001 as the contributor realized that privacy
may help terrorists. The rest of the team concluded that it was now even
more important to fight for civil liberties. The first release was
called “GNet” – already with the name GNUnet in mind, but without the
blessing of GNU we did not dare to call it GNUnet immediately. A few
months after the first release we contacted the GNU project, happily
agreed to their governance model and became an official GNU package.

Within the first year, we created GNU libextractor, a helper library for
meta data extraction which has been used by a few other projects as
well. 2003 saw the emergence of pluggable transports, the ability for
GNUnet to use different mechanisms for communication, starting with TCP,
UDP and SMTP (support for the latter was later dropped due to a lack of
maintenance). In 2005, the project first started to evolve beyond the
original file-sharing application with a first simple P2P chat. In 2007,
we created GNU libmicrohttpd to support a pluggable transport based on
HTTP. In 2009, the architecture was radically modularized into the
multi-process system that exists today. Coincidentally, the first
version of the ARM service (ARM: Automatic Restart Manager) was
implemented a day before systemd was announced. From 2009 to 2014 work
progressed rapidly thanks to a significant research grant from the
Deutsche Forschungsgesellschaft. This resulted in particular in the
creation of the R5N DHT, CADET, ATS and the GNU Name System. In 2010,
GNUnet was selected as the basis for the secushare online social
network, resulting in a significant growth of the core team. In 2013, we
launched GNU Taler to address the challenge of convenient and
privacy-preserving online payments. In 2015, the pretty Easy privacy
(pEp) project announced that they will use GNUnet as the technology for
their meta-data protection layer, ultimately resulting in GNUnet e.V.
entering into a formal long-term collaboration with the pEp Foundation.
In 2016, Taler Systems SA, a first startup using GNUnet technology, was
founded with support from the community.

GNUnet is not merely a technical project, but also a political mission:
like the GNU project as a whole, we are writing software to achieve
political goals with a focus on the human right of informational
self-determination. Putting users in control of their computing has been
the core driver of the GNU project. With GNUnet we are focusing on
informational self-determination for collaborative computing and
communication over networks.

The Internet is shaped as much by code and protocols as it is by its
associated political processes (IETF, ICANN, IEEE, etc.). Similarly its
flaws are not limited to the protocol design. Thus, technical excellence
by itself will not suffice to create a better network. We also need to
build a community that is wise, humble and has a sense of humor to
achieve our goal to create a technical foundation for a society we would
like to live in.

@menu
* Project governance:: 
* Philosophy:: 
* Key Concepts:: 

@end menu

@node Project governance,Philosophy,,About GNUnet
@anchor{about project-governance}@anchor{4}
@section Project governance


GNUnet, like the GNU project and many other free software projects,
follows the governance model of a benevolent dictator. This means that
ultimately, the GNU project appoints the GNU maintainer and can overrule
decisions made by the GNUnet maintainer. Similarly, the GNUnet
maintainer can overrule any decisions made by individual developers.
Still, in practice neither has happened in the last 20 years for GNUnet,
and we hope to keep it that way.

The current maintainers of GNUnet are:


@itemize -

@item 
Christian Grothoff@footnote{https://grothoff.org/christian/}

@item 
Martin Schanzenbach@footnote{https://schanzen.eu/}
@end itemize

The GNUnet project is supported by GNUnet e.V., a German association
where any developer can become a member. GNUnet e.V. serves as a legal
entity to hold the copyrights to GNUnet. GNUnet e.V. may also choose to
pay for project resources, and can collect donations as well as choose
to adjust the license of the software (with the constraint that it has
to remain free software). In 2018 we switched from GPL3 to AGPL3, in
practice these changes do not happen very often.

@node Philosophy,Key Concepts,Project governance,About GNUnet
@anchor{about philosophy}@anchor{5}
@section Philosophy


The primary goal of the GNUnet project is to provide a reliable, open,
non-discriminating and censorship-resistant system for information
exchange. We value free speech above state interests and intellectual
monopoly. GNUnet’s long-term goal is to serve as a development platform
for the next generation of Internet protocols.

Participants are encouraged to contribute at least as much resources
(storage, bandwidth) to the network as they consume, so that their
participation does not have a negative impact on other users.

@menu
* Design Principles:: 
* Privacy and Anonymity:: 
* Practicality:: 

@end menu

@node Design Principles,Privacy and Anonymity,,Philosophy
@anchor{about design-principles}@anchor{6}
@subsection Design Principles


These are the GNUnet design principles, in order of importance:


@itemize -

@item 
GNUnet must be implemented as Free Software — This means that you
have the four essential freedoms: to run the program, to study and
change the program in source code form, to redistribute exact copies,
and to distribute modified versions.
(@indicateurl{https://www.gnu.org/philosophy/free-sw.html}).

@item 
GNUnet must minimize the amount of personally identifiable
information exposed.

@item 
GNUnet must be fully distributed and resilient to external attacks
and rogue participants.

@item 
GNUnet must be self-organizing and not depend on administrators or
centralized infrastructure.

@item 
GNUnet must inform the user which other participants have to be
trusted when establishing private communications.

@item 
GNUnet must be open and permit new peers to join.

@item 
GNUnet must support a diverse range of applications and devices.

@item 
GNUnet must use compartmentalization to protect sensitive
information.

@item 
The GNUnet architecture must be resource efficient.

@item 
GNUnet must provide incentives for peers to contribute more resources
than they consume.
@end itemize

@node Privacy and Anonymity,Practicality,Design Principles,Philosophy
@anchor{about privacy-and-anonymity}@anchor{7}
@subsection Privacy and Anonymity


The GNUnet protocols minimize the leakage of personally identifiable
information of participants and do not allow adversaries to control,
track, monitor or censor users activities. The GNUnet protocols also
make it as hard as possible to disrupt operations by participating in
the network with malicious intent.

Analyzing participant’s activities becomes more difficult as the number
of peers and applications that generate traffic on the network grows,
even if the additional traffic generated is not related to anonymous
communication. This is one of the reasons why GNUnet is developed as a
peer-to-peer framework where many applications share the lower layers of
an increasingly complex protocol stack. The GNUnet architecture
encourages many different forms of peer-to-peer applications.

@node Practicality,,Privacy and Anonymity,Philosophy
@anchor{about practicality}@anchor{8}
@subsection Practicality


Wherever possible GNUnet allows the peer to adjust its operations and
functionalities to specific use cases. A GNUnet peer running on a mobile
device with limited battery for example might choose not to relay
traffic for other participants.

For certain applications like file-sharing GNUnet allows participants to
trade degrees of anonymity in exchange for increased efficiency.
However, it is not possible for any user’s efficiency requirements to
compromise the anonymity of any other user.

@node Key Concepts,,Philosophy,About GNUnet
@anchor{about key-concepts}@anchor{9}
@section Key Concepts


In this section, the fundamental concepts of GNUnet are explained. Most
of them are also described in our research papers. First, some of the
concepts used in the GNUnet framework are detailed. The second part
describes concepts specific to anonymous file-sharing.

@menu
* Cryptography:: 
* Anonymity:: 
* Deniability:: 
* Accounting to Encourage Resource Sharing:: 

@end menu

@node Cryptography,Anonymity,,Key Concepts
@anchor{about cryptography}@anchor{a}
@subsection Cryptography


Adversaries (malicious, bad actors) outside of GNUnet are not supposed
to know what kind of actions a peer is involved in. Only the specific
neighbor of a peer that is the corresponding sender or recipient of a
message may know its contents, and even then application protocols may
place further restrictions on that knowledge. In order to ensure
confidentiality, GNUnet uses link encryption, that is each message
exchanged between two peers is encrypted using a pair of keys only known
to these two peers. Encrypting traffic like this makes any kind of
traffic analysis much harder. Naturally, for some applications, it may
still be desirable if even neighbors cannot determine the concrete
contents of a message. In GNUnet, this problem is addressed by the
specific application-level protocols. See for example the following
sections: Anonymity@footnote{about.md#anonymity}, see How file-sharing achieves Anonymity@footnote{about.md#how-file-sharing-achieves-anonymity},
and see Deniability@footnote{about.md#deniability}.

@menu
* Peer Identities:: 
* Egos:: 

@end menu

@node Peer Identities,Egos,,Cryptography
@anchor{about peer-identities}@anchor{b}
@subsubsection Peer Identities


In GNUnet, the identity of a host is its public key called `Peer Identity'.
For that reason, man-in-the-middle attacks will not break the authentication or
accounting goals. Essentially, for GNUnet, the IP of the host has
nothing to do with the identity of the host. As the public key is the
only thing that truly matters, faking an IP, a port or any other
property of the underlying transport protocol is irrelevant. In fact,
GNUnet peers can use multiple IPs (IPv4 and IPv6) on multiple ports — or
even not use the IP protocol at all (by running directly on layer 2).

Peer identities are used to identify peers in the network and are unique
for each peer. The identity for a peer is simply its public key, which
is generated along with a private key when the peer is started for the
first time. While the identity is binary data, it is often expressed as
an ASCII string. For example, the following is a peer identity as you
might see it in various places:

@example
UAT1S6PMPITLBKSJ2DGV341JI6KF7B66AC4JVCN9811NNEGQLUN0
@end example

You can find your peer identity by running @code{gnunet-core}.

Almost all peer-to-peer communications in GNUnet are between mutually
authenticated peers. The authentication works by using ECDHE, that is a
DH (Diffie—Hellman) key exchange using ephemeral elliptic curve
cryptography. The ephemeral ECC (Elliptic Curve Cryptography) keys are
signed using `EdDSA'. The shared secret from ECDHE is used to create a
pair of session keys (using HKDF) which are then used to encrypt the
communication between the two peers using both `256-bit AES'
and `256-bit Twofish' (with independently derived
secret keys). As only the two participating hosts know the shared
secret, this authenticates each packet without requiring signatures each
time. GNUnet mostly uses the `SHA-512' hash algorithm.

GNUnet uses a special type of message to communicate a binding between
public (ECC) keys to their current network address. These messages are
commonly called `HELLOs' or peer advertisements. They contain the public
key of the peer and its current network addresses for various transport
services. A transport service is a special kind of shared library that
provides (possibly unreliable, out-of-order) message delivery between
peers. For the UDP and TCP transport services, a network address is an
IP and a port. GNUnet can also use other transports (HTTP, HTTPS, WLAN,
etc.) which use various other forms of addresses. Note that any node can
have many different active transport services at the same time, and each
of these can have a different addresses. Binding messages expire after
at most a week (the timeout can be shorter if the user configures the
node appropriately). This expiration ensures that the network will
eventually get rid of outdated advertisements.

For more information, refer to the following paper:

Ronaldo A. Ferreira, Christian Grothoff, and Paul Ruth. A Transport
Layer Abstraction for Peer-to-Peer Networks Proceedings of the 3rd
International Symposium on Cluster Computing and the Grid (GRID 2003),
2003. (@indicateurl{https://git.gnunet.org/bibliography.git/plain/docs/transport.pdf})

@node Egos,,Peer Identities,Cryptography
@anchor{about egos}@anchor{c}
@subsubsection Egos


`Egos' are your “identities” in GNUnet. Any user can assume multiple
identities, for example to separate their activities online. Egos can
correspond to “pseudonyms” or “real-world identities”. Technically an
ego is first of all a key pair of a public- and private-key.
The current primary use for Egos are in the GNU Name System as zone keys.

@menu
* Zones in the GNU Name System:: 

@end menu

@node Zones in the GNU Name System,,,Egos
@anchor{about zones-in-the-gnu-name-system}@anchor{d}
@subsubsection Zones in the GNU Name System


Egos are used as `GNS zones'.

GNS zones are similar to those of DNS zones, but instead of a hierarchy
of authorities to governing their use, GNS zones are controlled by a
private key. When you create a record in a DNS zone, that information is
stored in your nameserver. Anyone trying to resolve your domain then
gets pointed (hopefully) by the centralised authority to your
nameserver. Whereas GNS, being fully decentralized by design, stores
that information in DHT. The validity of the records is assured
cryptographically, by signing them with the private key of the
respective zone.

Anyone trying to resolve records in a zone of your domain can then
verify the signature of the records they get from the DHT and be assured
that they are indeed from the respective zone. To make this work, there
is a 1:1 correspondence between zones and their public-private key
pairs. So when we talk about the owner of a GNS zone, that’s really the
owner of the private key. And a user accessing a zone needs to somehow
specify the corresponding public key first.

For more information, refer to RFC 9498.

@node Anonymity,Deniability,Cryptography,Key Concepts
@anchor{about anonymity}@anchor{e}
@subsection Anonymity


Providing anonymity for users is the central goal for the anonymous
file-sharing application. Many other design decisions follow in the
footsteps of this requirement. Anonymity is never absolute. While there
are various scientific metrics (Claudia Díaz, Stefaan Seys, Joris
Claessens, and Bart Preneel. Towards measuring anonymity. 2002.
(@indicateurl{https://git.gnunet.org/bibliography.git/plain/docs/article-89.pdf}))
that can help quantify the level of anonymity that a given mechanism
provides, there is no such thing as “complete anonymity”.

GNUnet’s file-sharing implementation allows users to select for each
operation (publish, search, download) the desired level of anonymity.
The metric used is based on the amount of cover traffic needed to hide
the request.

While there is no clear way to relate the amount of available cover
traffic to traditional scientific metrics such as the anonymity set or
information leakage, it is probably the best metric available to a peer
with a purely local view of the world, in that it does not rely on
unreliable external information or a particular adversary model.

The default anonymity level is 1, which uses anonymous routing but
imposes no minimal requirements on cover traffic. It is possible to
forego anonymity when this is not required. The anonymity level of 0
allows GNUnet to use more efficient, non-anonymous routing.

@menu
* How file-sharing achieves Anonymity:: 
* How messaging provided Anonymity:: 

@end menu

@node How file-sharing achieves Anonymity,How messaging provided Anonymity,,Anonymity
@anchor{about how-file-sharing-achieves-anonymity}@anchor{f}
@subsubsection How file-sharing achieves Anonymity


Contrary to other designs, we do not believe that users achieve strong
anonymity just because their requests are obfuscated by a couple of
indirections. This is not sufficient if the adversary uses traffic
analysis. The threat model used for anonymous file sharing in GNUnet
assumes that the adversary is quite powerful. In particular, we assume
that the adversary can see all the traffic on the Internet. And while we
assume that the adversary can not break our encryption, we assume that
the adversary has many participating nodes in the network and that it
can thus see many of the node-to-node interactions since it controls
some of the nodes.

The system tries to achieve anonymity based on the idea that users can
be anonymous if they can hide their actions in the traffic created by
other users. Hiding actions in the traffic of other users requires
participating in the traffic, bringing back the traditional technique of
using indirection and source rewriting. Source rewriting is required to
gain anonymity since otherwise an adversary could tell if a message
originated from a host by looking at the source address. If all packets
look like they originate from one node, the adversary can not tell which
ones originate from that node and which ones were routed. Note that in
this mindset, any node can decide to break the source-rewriting paradigm
without violating the protocol, as this only reduces the amount of
traffic that a node can hide its own traffic in.

If we want to hide our actions in the traffic of other nodes, we must
make our traffic indistinguishable from the traffic that we route for
others. As our queries must have us as the receiver of the reply
(otherwise they would be useless), we must put ourselves as the receiver
of replies that actually go to other hosts; in other words, we must
indirect replies. Unlike other systems, in anonymous file-sharing as
implemented on top of GNUnet we do not have to indirect the replies if
we don’t think we need more traffic to hide our own actions.

This increases the efficiency of the network as we can indirect less
under higher load. Refer to the following paper for more: Krista Bennett
and Christian Grothoff. GAP — practical anonymous networking. In
Proceedings of Designing Privacy Enhancing Technologies, 2003.
(@indicateurl{https://git.gnunet.org/bibliography.git/plain/docs/aff.pdf})

@node How messaging provided Anonymity,,How file-sharing achieves Anonymity,Anonymity
@anchor{about how-messaging-provided-anonymity}@anchor{10}
@subsubsection How messaging provided Anonymity


While the file-sharing tries to achieve anonymity through hiding actions
in other traffic, the messaging service provides a weaker form of
protection against identification.

The messaging service allows the use of an anonymous ego for the signing
and verification process of messages instead of a unique ego. This
anonymous ego is a publicly known key pair which is shared between all
peers in GNUnet.

Using this ego only ensures that individual messages alone can’t
identify its sender inside of a messenger room. It should be clarified
that the route of the traffic for each message can still be tracked to
identify the senders peer inside of a messenger room if the threat agent
controls certain peers hosting the room.

Also opening a room in the messenger service will potentially match your
peer identity with the internal member identity from the messenger
service. So despite using the anonymous ego you can reveal your peer
identity. This means to decrease the chance of being identified, it is
recommended to enter rooms but you should not open them for others.

@node Deniability,Accounting to Encourage Resource Sharing,Anonymity,Key Concepts
@anchor{about deniability}@anchor{11}
@subsection Deniability


Even if the user that downloads data and the server that provides data
are anonymous, the intermediaries may still be targets. In particular,
if the intermediaries can find out which queries or which content they
are processing, a strong adversary could try to force them to censor
certain materials.

With the file-encoding used by GNUnet’s anonymous file-sharing, this
problem does not arise. The reason is that queries and replies are
transmitted in an encrypted format such that intermediaries cannot tell
what the query is for or what the content is about. Mind that this is
not the same encryption as the link-encryption between the nodes. GNUnet
has encryption on the network layer (link encryption, confidentiality,
authentication) and again on the application layer (provided by
gnunet-publish, gnunet-download, gnunet-search and gnunet-fs-gtk).

Refer to the following paper for more: Christian Grothoff, Krista
Grothoff, Tzvetan Horozov, and Jussi T. Lindgren. An Encoding for
Censorship-Resistant Sharing. 2009.
(@indicateurl{https://git.gnunet.org/bibliography.git/plain/docs/ecrs.pdf})

@node Accounting to Encourage Resource Sharing,,Deniability,Key Concepts
@anchor{about accounting-to-encourage-resource-sharing}@anchor{12}
@subsection Accounting to Encourage Resource Sharing


Most distributed P2P networks suffer from a lack of defenses or
precautions against attacks in the form of freeloading. While the
intentions of an attacker and a freeloader are different, their effect
on the network is the same; they both render it useless. Most simple
attacks on networks such as Gnutella involve flooding the network with
traffic, particularly with queries that are, in the worst case,
multiplied by the network.

In order to ensure that freeloaders or attackers have a minimal impact
on the network, GNUnet’s file-sharing implementation (FS) tries to
distinguish good (contributing) nodes from malicious (freeloading)
nodes. In GNUnet, every file-sharing node keeps track of the behavior of
every other node it has been in contact with. Many requests (depending
on the application) are transmitted with a priority (or importance)
level. That priority is used to establish how important the sender
believes this request is. If a peer responds to an important request,
the recipient will increase its trust in the responder: the responder
contributed resources. If a peer is too busy to answer all requests, it
needs to prioritize. For that, peers do not take the priorities of the
requests received at face value. First, they check how much they trust
the sender, and depending on that amount of trust they assign the
request a (possibly lower) effective priority. Then, they drop the
requests with the lowest effective priority to satisfy their resource
constraints. This way, GNUnet’s economic model ensures that nodes that
are not currently considered to have a surplus in contributions will not
be served if the network load is high.

For more information, refer to the following paper: Christian Grothoff.
An Excess-Based Economic Model for Resource Allocation in Peer-to-Peer
Networks. Wirtschaftsinformatik, June 2003.
(@indicateurl{https://git.gnunet.org/bibliography.git/plain/docs/ebe.pdf})

@node Installation,User Manual,About GNUnet,Top
@anchor{installing doc}@anchor{13}@anchor{installing installation}@anchor{14}
@chapter Installation


This guide is intended for those who want to install GNUnet from source.
For instructions on how to install GNUnet as a binary package please
refer to the official documentation of your operating system or package
manager.

For understanding this guide properly it is important to know that there
are two different ways of running GNUnet:


@itemize -

@item 
the single-user setup

@item 
the multi-user setup
@end itemize

The latter variant has a better security model and requires extra
preparation before running make install and a different configuration.
Beginners who want to quickly try out GNUnet can use the single-user
setup.

@menu
* Dependencies:: 
* Getting the Source Code:: 
* Create user and groups for the system services:: 
* Preparing and Compiling the Source Code:: 
* Minimal configuration:: 

@end menu

@node Dependencies,Getting the Source Code,,Installation
@anchor{installing dependencies}@anchor{15}
@section Dependencies


GNUnet needs few libraries and applications for being able to run and
another few optional ones for using certain features. Preferably they
should be installed with a package manager.

The mandatory libraries and applications are


@itemize -

@item 
meson 1.0 or above

@item 
recutils 1.0 or above (when building from git)

@item 
gettext

@item 
glibc (read below, other libcs work)

@item 
GnuTLS 3.2.12 or above, recommended to be linked against libunbound

@item 
GNU make 4.0 or higher (other make implementations do work)

@item 
iptables (on Linux systems)

@item 
libtool 2.2 or above

@item 
libltdl (part of libtool)

@item 
libgcrypt 1.6 or above

@item 
libidn2 or libidn

@item 
libmicrohttpd 0.9.63 or above

@item 
libunistring

@item 
libjansson

@item 
libjose (optional, for reclaimID)

@item 
libgmp

@item 
libcurl (ideally linked to GnuTLS) 7.35.0 or
above

@item 
Texinfo 5.2 or above (for building the documentation)

@item 
Texlive 2012 or above (for building the documentation, and for
gnunet-bcd)

@item 
makeinfo 4.8 or above

@item 
pkgconf (or pkg-config)

@item 
zlib
@end itemize

Glibc is required for certain NSS features:

@example
One mechanism of integrating GNS with legacy applications via NSS is
not available if this is disabled. But applications that don't use the
glibc for NS resolution won't work anyway with this, so little is lost
on *BSD systems.
GNS via direct use or via the HTTP or DNS proxies is unaffected.
@end example

Other libcs should work, the resulting builds just don’t include the
glibc NSS specific code. One example is the build against NetBSD’s libc
as detailed in @indicateurl{https://bugs.gnunet.org/view.php?id=5605}.

In addition GNUnet needs at least one of these three databases (at the
minimum sqlite3)


@itemize -

@item 
sqlite + libsqlite 3.8 or above (the default, requires no further
configuration)

@item 
postgres + libpq

@item 
mysql + libmysqlclient
@end itemize

These are the dependencies only required for certain features


@itemize -

@item 
miniupnpc (for traversing NAT boxes more reliably)

@item 
libnss

@item 
libopus (for running the GNUnet conversation telephony application)

@item 
libogg (for running the GNUnet conversation telephony application)

@item 
gstreamer OR libpulse (for running the GNUnet conversation telephony
application)

@item 
bluez (for bluetooth support)

@item 
libextractor (optional but highly recommended, read below)

@item 
texi2mdoc (for automatic mdoc generation)

@item 
perl5 for some utilities (which are not installed)
@end itemize

About libextractor being optional:

@example
While libextractor ("LE") is optional, it is recommended to build gnunet
against it. If you install it later, you won't benefit from libextractor.
If you are a distributor, we recommend to split LE into basis + plugins
rather than making LE an option as an afterthought by the user.  LE
itself is very small, but its dependency chain on first, second, third
etc level can be big.  There is a small effect on privacy if your LE
build differs from one which includes all plugins (plugins are build as
shared objects): if users publish a directory with a mixture of file
types (for example mpeg, jpeg, png, gif) the configuration of LE could
leak which plugins are installed for which filetypes are not providing
more details.  However, this leak is just a minor concern.
@end example

These are the test-suite requirements:


@itemize -

@item 
python3.6 or higher

@item 
gnunet (installation first)

@item 
some core-utils: which(1), bc(1), curl(1), sed(1), awk(1), etc.

@item 
a shell (very few Bash scripts, the majority are POSIX sh scripts)
@end itemize

These are runtime requirements:


@itemize -

@item 
nss (the certutil binary, for gnunet-gns-proxy-setup-ca)

@item 
openssl (openssl binary, for gnunet-gns-proxy-setup-ca)
@end itemize

@node Getting the Source Code,Create user and groups for the system services,Dependencies,Installation
@anchor{installing getting-the-source-code}@anchor{16}
@section Getting the Source Code


You can either download the source code using git (you obviously need
git installed) or as an archive.

Using git type

@example
$ git clone git://git.gnunet.org/gnunet.git
@end example

The release archive can be found at @indicateurl{https://ftpmirror.gnu.org/gnu/gnunet/}.
You can find nightly builds at @indicateurl{https://buildbot.gnunet.org/artifacts/}.
Extract it using a graphical archive tool or tar:

@example
tar xf gnunet-<VERSION> .tar.gz
@end example

In the next chapter we will assume that the source code is available in
the home directory at ~/gnunet.

@node Create user and groups for the system services,Preparing and Compiling the Source Code,Getting the Source Code,Installation
@anchor{installing create-user-and-groups-for-the-system-services}@anchor{17}
@section Create user and groups for the system services


`For single-user setup this section can be skipped'

The multi-user setup means that there are system services, which are run
once per machine as a dedicated system user (called gnunet) and user
services which can be started by every user who wants to use GNUnet
applications. The user services communicate with the system services
over unix domain sockets. To gain permissions to read and write those
sockets the users running GNUnet applications will need to be in the
gnunet group. In addition the group gnunetdns may be needed (see below).

Create user gnunet who is member of the group gnunet (automatically
created) and specify a home directory where the GNUnet services will
store persistent data such as information about peers.

@example
$ sudo useradd --system --home-dir /var/lib/gnunet --create-home gnunet
@end example

Now add your own user to the gnunet group:

@example
$ sudo usermod -aG gnunet <user>
@end example

Create a group gnunetdns. This allows using setgid in a way that only
the DNS service can run the gnunet-helper-dns binary. This is only
needed if system-wide DNS interception will be used. For more
information see Configuring system-wide DNS interception@footnote{installing#configuring%20system-wide-dns-interception}.

@example
$ sudo groupadd gnunetdns
@end example

@node Preparing and Compiling the Source Code,Minimal configuration,Create user and groups for the system services,Installation
@anchor{installing preparing-and-compiling-the-source-code}@anchor{18}
@section Preparing and Compiling the Source Code


Get the source code from git or a source tarball. When running
@cite{meson setup}, options can be specified to customize the compilation and
installation process. For details execute:

@example
$ meson setup --help
@end example

The following example configures the installation prefix @cite{/usr/local}
and creates the out-of-tree build folder @cite{build}.

@example
$ ./bootstrap (only when installing from git)
$ meson setup -Dprefix=/usr/local build
@end example

After running the bootstrap script and setup successfully the source
code can be compiled and the compiled binaries can be installed using:

@example
$ meson compile -C build
$ meson install -C build
@end example

The latter command may need to be run as root (or with sudo) because
some binaries need the suid bit set. Without that some features
(e.g. the VPN service, system-wide DNS interception, NAT traversal using
ICMP) will not work.

@menu
* NSS plugin (optional): NSS plugin optional. 
* Installing the GNS Certificate Authority (Optional): Installing the GNS Certificate Authority Optional. 

@end menu

@node NSS plugin optional,Installing the GNS Certificate Authority Optional,,Preparing and Compiling the Source Code
@anchor{installing nss-plugin-optional}@anchor{19}
@subsection NSS plugin (optional)


`NOTE: The installation of the NSS plugin is only necessary if GNS
resolution shall be used with legacy applications (that only support
DNS) and if you cannot do not want to use the DNS2GNS service.'

One important library is the GNS plugin for NSS (the name services
switch) which allows using GNS (the GNU name system) in the normal DNS
resolution process. Unfortunately NSS expects it in a specific location
(probably @code{/lib}) which may differ from the installation prefix (see
@code{--prefix} option in the previous section). This is why the plugin has
to be installed manually.

Find the directory where nss plugins are installed on your system, e.g.:

@example
$ ls -l /lib/libnss_*
/lib/libnss_mymachines.so.2
/lib/libnss_resolve.so.2
/lib/libnss_myhostname.so.2
/lib/libnss_systemd.so.2
@end example

Copy the GNS NSS plugin to that directory:

@example
cp ~/gnunet/src/gns/nss/.libs/libnss_gns.so.2 /lib
@end example

@node Installing the GNS Certificate Authority Optional,,NSS plugin optional,Preparing and Compiling the Source Code
@anchor{installing installing-the-gns-certificate-authority-optional}@anchor{1a}
@subsection Installing the GNS Certificate Authority (Optional)


`NOTE: Installing the GNS certificate authority is only necessary if
GNS shall be used in a browser and if you cannot or do not want to use
the DNS2GNS service.'

The GNS Certificate authority can provide TLS certificates for GNS names
while downloading webpages from legacy webservers. This allows browsers
to use HTTPS in combinations with GNS name resolution.

To install it execute the GNS CA-setup script. So far Firefox and
Chromium are supported.

@example
$ gnunet-gns-proxy-setup-ca
@end example

A local proxy server, that takes care of the name resolution and
provides certificates on-the-fly needs to be started:

@example
$ /usr/lib/gnunet/libexec/gnunet-gns-proxy
@end example

Now GNS should work in browsers that are configured to use a SOCKS proxy
on localhost:7777.

@node Minimal configuration,,Preparing and Compiling the Source Code,Installation
@anchor{installing id1}@anchor{1b}@anchor{installing minimal-configuration}@anchor{1c}
@section Minimal configuration


GNUnet needs a configuration file to start (see @ref{1d,,Config file format}). For the `single-user setup' an empty
file is sufficient:

@example
$ touch ~/.config/gnunet.conf
@end example

For the `multi-user setup' we need an extra config file for the system
services. The default location is @code{/etc/gnunet.conf}. The minimal
content of that file which activates the system services roll is:

@example
[arm]
START_SYSTEM_SERVICES = YES
START_USER_SERVICES = NO
@end example

The config file for the user services (@code{~/.config/gnunet.conf}) needs
the opposite configuration to activate the user services roll:

@example
[arm]
START_SYSTEM_SERVICES = NO
START_USER_SERVICES = YES
@end example

@node User Manual,Developer Handbook,Installation,Top
@anchor{users/index doc}@anchor{1e}@anchor{users/index user-manual}@anchor{1f}
@chapter User Manual


This tutorial is supposed to give a first introduction for users trying
to do something real with GNUnet. Installation and configuration are
specifically outside of the scope of this tutorial. Instead, we start by
briefly checking that the installation works, and then dive into
uncomplicated, concrete practical things that can be done with the
framework provided by GNUnet.

In short, this chapter of the “GNUnet Reference Documentation” will show
you how to use the various peer-to-peer applications of the GNUnet
system. As GNUnet evolves, we will add new sections for the various
applications that are being created.

Comments on the content of this chapter, and extensions of it are always
welcome.

@menu
* Getting started:: 
* Subsystems:: 
* The GNU Name System:: 
* re;claimID: re claimID. 
* File-sharing:: 
* Virtual Public Network:: 
* Messenger:: 
* Advanced Configuration:: 

@end menu

@node Getting started,Subsystems,,User Manual
@anchor{users/start doc}@anchor{20}@anchor{users/start getting-started}@anchor{21}
@section Getting started


Prior to using any GNUnet-based application, one has to start a node:

@example
$ gnunet-arm -s
@end example

To stop GNUnet:

@example
$ gnunet-arm -e
@end example

You can usually find the logs under @code{~/.cache/gnunet} and all files
such as databases and private keys in @code{~/.local/share/gnunet}.

The list of running services can be displayed using the @code{-I} option.
It should look similar to this example:

@example
$ gnunet-arm -I
Running services:
topology (gnunet-daemon-topology)
nat (gnunet-service-nat)
vpn (gnunet-service-vpn)
gns (gnunet-service-gns)
cadet (gnunet-service-cadet)
namecache (gnunet-service-namecache)
hostlist (gnunet-daemon-hostlist)
revocation (gnunet-service-revocation)
zonemaster (gnunet-service-zonemaster)
zonemaster-monitor (gnunet-service-zonemaster-monitor)
dht (gnunet-service-dht)
namestore (gnunet-service-namestore)
set (gnunet-service-set)
statistics (gnunet-service-statistics)
nse (gnunet-service-nse)
fs (gnunet-service-fs)
peerstore (gnunet-service-peerstore)
core (gnunet-service-core)
rest (gnunet-rest-server)
transport (gnunet-service-transport)
datastore (gnunet-service-datastore)
@end example

For the `multi-user' setup first the system services need to be
started as the system user, i.e. the user gnunet needs to execute
@code{gnunet-arm -s}. This should be done by the system’s init system. Then
the user who wants to start GNUnet applications has to run
@code{gnunet-arm -s}, too. It is recommended to automate this, e.g. using
the user’s crontab.

You can check directly connected peers with:

@example
$ gnunet-core --connection-status
@end example

This should return (at least) one established connection peer.
Otherwise, again, there is likely a problem with your network
configuration.

You can display your own current peer identity with:

@example
$ gnunet-core --show-identity
@end example

@node Subsystems,The GNU Name System,Getting started,User Manual
@anchor{users/subsystems doc}@anchor{22}@anchor{users/subsystems subsystems}@anchor{23}
@section Subsystems


This section consists brief description of the subsystems that make up
GNUnet.
This image is giving an overview over system dependencies and interactions.

@image{gnunet-figures/gnunet-arch-full,,,,svg}

@menu
* CADET - Decentralized End-to-end Transport:: 
* CORE - GNUnet link layer:: 
* DHT - Distributed Hash Table:: 
* FS — File sharing over GNUnet:: 
* NAMECACHE — DHT caching of GNS results:: 
* NAMESTORE — Storage of local GNS zones:: 
* HOSTLIST — HELLO bootstrapping and gossip:: 
* IDENTITY — Ego management:: 
* MESSENGER — Room-based end-to-end messaging:: 
* NSE — Network size estimation:: 
* PEERINFO — Persistent HELLO storage:: 
* PEERSTORE — Extensible local persistent data storage:: 
* REGEX — Service discovery using regular expressions:: 
* REST — RESTful GNUnet Web APIs:: 
* REVOCATION — Ego key revocation:: 
* RPS — Random peer sampling:: 
* STATISTICS — Runtime statistics publication:: 
* TRANSPORT-NG — Next-generation transport management:: 
* TRANSPORT — Overlay transport management:: 
* SET — Peer to peer set operations (Deprecated): SET — Peer to peer set operations Deprecated. 
* SETI — Peer to peer set intersections:: 
* SETU — Peer to peer set unions:: 

@end menu

@node CADET - Decentralized End-to-end Transport,CORE - GNUnet link layer,,Subsystems
@anchor{users/subsystems cadet-decentralized-end-to-end-transport}@anchor{24}
@subsection CADET - Decentralized End-to-end Transport


The Confidential Ad-hoc Decentralized End-to-end Transport (CADET) subsystem
in GNUnet is responsible for secure end-to-end
communications between nodes in the GNUnet overlay network. CADET builds
on the CORE subsystem, which provides for the link-layer communication,
by adding routing, forwarding, and additional security to the
connections. CADET offers the same cryptographic services as CORE, but
on an end-to-end level. This is done so peers retransmitting traffic on
behalf of other peers cannot access the payload data.


@itemize -

@item 
CADET provides confidentiality with so-called perfect forward
secrecy; we use ECDHE powered by Curve25519 for the key exchange and
then use symmetric encryption, encrypting with both AES-256 and
Twofish

@item 
authentication is achieved by signing the ephemeral keys using
Ed25519, a deterministic variant of ECDSA

@item 
integrity protection (using SHA-512 to do encrypt-then-MAC, although
only 256 bits are sent to reduce overhead)

@item 
replay protection (using nonces, timestamps, challenge-response,
message counters and ephemeral keys)

@item 
liveness (keep-alive messages, timeout)
@end itemize

Additional to the CORE-like security benefits, CADET offers other
properties that make it a more universal service than CORE.


@itemize -

@item 
CADET can establish channels to arbitrary peers in GNUnet. If a peer
is not immediately reachable, CADET will find a path through the
network and ask other peers to retransmit the traffic on its behalf.

@item 
CADET offers (optional) reliability mechanisms. In a reliable channel
traffic is guaranteed to arrive complete, unchanged and in-order.

@item 
CADET takes care of flow and congestion control mechanisms, not
allowing the sender to send more traffic than the receiver or the
network are able to process.
@end itemize

@geindex CORE; subsystem
@anchor{users/subsystems core-subsystem}@anchor{25}
@node CORE - GNUnet link layer,DHT - Distributed Hash Table,CADET - Decentralized End-to-end Transport,Subsystems
@anchor{users/subsystems core-gnunet-link-layer}@anchor{26}
@subsection CORE - GNUnet link layer


The CORE subsystem in GNUnet is responsible for securing link-layer
communications between nodes in the GNUnet overlay network. CORE builds
on the TRANSPORT subsystem which provides for the actual, insecure,
unreliable link-layer communication (for example, via UDP or WLAN), and
then adds fundamental security to the connections:


@itemize -

@item 
confidentiality with so-called perfect forward secrecy; we use ECDHE
(Elliptic-curve Diffie—Hellman@footnote{http://en.wikipedia.org/wiki/Elliptic_curve_Diffie%E2%80%93Hellman})
powered by Curve25519 (Curve25519@footnote{http://cr.yp.to/ecdh.html})
for the key exchange and then use symmetric encryption, encrypting
with both AES-256
(AES-256@footnote{http://en.wikipedia.org/wiki/Rijndael}) and Twofish
(Twofish@footnote{http://en.wikipedia.org/wiki/Twofish})

@item 
authentication@footnote{http://en.wikipedia.org/wiki/Authentication} is
achieved by signing the ephemeral keys using Ed25519
(Ed25519@footnote{http://ed25519.cr.yp.to/}), a deterministic variant of
ECDSA (ECDSA@footnote{http://en.wikipedia.org/wiki/ECDSA})

@item 
integrity protection (using SHA-512
(SHA-512@footnote{http://en.wikipedia.org/wiki/SHA-2}) to do
encrypt-then-MAC
(encrypt-then-MAC@footnote{http://en.wikipedia.org/wiki/Authenticated_encryption}))

@item 
Replay (replay@footnote{http://en.wikipedia.org/wiki/Replay_attack})
protection (using nonces, timestamps, challenge-response, message
counters and ephemeral keys)

@item 
liveness (keep-alive messages, timeout)
@end itemize
@anchor{users/subsystems limitations}@anchor{27}
@geindex CORE; limitations
Limitations
Limitations
-----------

CORE does not perform
routing@footnote{http://en.wikipedia.org/wiki/Routing}; using CORE it is
only possible to communicate with peers that happen to already be
"directly" connected with each other. CORE also does not have an API
to allow applications to establish such "direct" connections --- for
this, applications can ask TRANSPORT, but TRANSPORT might not be able to
establish a "direct" connection. The TOPOLOGY subsystem is responsible
for trying to keep a few "direct" connections open at all times.
Applications that need to talk to particular peers should use the CADET
subsystem, as it can establish arbitrary "indirect" connections.

Because CORE does not perform routing, CORE must only be used directly
by applications that either perform their own routing logic (such as
anonymous file-sharing) or that do not require routing, for example
because they are based on flooding the network. CORE communication is
unreliable and delivery is possibly out-of-order. Applications that
require reliable communication should use the CADET service. Each
application can only queue one message per target peer with the CORE
service at any time; messages cannot be larger than approximately 63
kilobytes. If messages are small, CORE may group multiple messages
(possibly from different applications) prior to encryption. If permitted
by the application (using the cork@footnote{http://baus.net/on-tcp_cork/}
option), CORE may delay transmissions to facilitate grouping of multiple
small messages. If cork is not enabled, CORE will transmit the message
as soon as TRANSPORT allows it (TRANSPORT is responsible for limiting
bandwidth and congestion control). CORE does not allow flow control;
applications are expected to process messages at line-speed. If flow
control is needed, applications should use the CADET service.

@c when is a peer connected

@menu
* When is a peer "connected"?:: 

@end menu

@node When is a peer "connected"?,,,CORE - GNUnet link layer
@anchor{users/subsystems when-is-a-peer-0022connected-0022-003f}@anchor{28}@anchor{users/subsystems when-is-a-peer-connected}@anchor{29}
@subsubsection When is a peer "connected"?


In addition to the security features mentioned above, CORE also provides
one additional key feature to applications using it, and that is a
limited form of protocol-compatibility checking. CORE distinguishes
between TRANSPORT-level connections (which enable communication with
other peers) and application-level connections. Applications using the
CORE API will (typically) learn about application-level connections from
CORE, and not about TRANSPORT-level connections. When a typical
application uses CORE, it will specify a set of message types (from
@code{gnunet_protocols.h}) that it understands. CORE will then notify the
application about connections it has with other peers if and only if
those applications registered an intersecting set of message types with
their CORE service. Thus, it is quite possible that CORE only exposes a
subset of the established direct connections to a particular application
--- and different applications running above CORE might see different
sets of connections at the same time.

A special case are applications that do not register a handler for any
message type. CORE assumes that these applications merely want to
monitor connections (or "all" messages via other callbacks) and will
notify those applications about all connections. This is used, for
example, by the @code{gnunet-core} command-line tool to display the active
connections. Note that it is also possible that the TRANSPORT service
has more active connections than the CORE service, as the CORE service
first has to perform a key exchange with connecting peers before
exchanging information about supported message types and notifying
applications about the new connection.
.. _Distributed-Hash-Table-_0028DHT_0029:

@geindex Distributed hash table; subsystem
@geindex DHT; Distributed hash table

@node DHT - Distributed Hash Table,FS — File sharing over GNUnet,CORE - GNUnet link layer,Subsystems
@anchor{users/subsystems dht-distributed-hash-table}@anchor{2a}
@subsection DHT - Distributed Hash Table


GNUnet includes a generic distributed hash table that can be used by
developers building P2P applications in the framework. This section
documents high-level features and how developers are expected to use the
DHT. We have a research paper detailing how the DHT works. Also, Nate's
thesis includes a detailed description and performance analysis (in
chapter 6). @ref{2b,,[R5N2011]} @ref{2c,,[EVANS2011]}

Key features of GNUnet's DHT include:


@itemize -

@item 
stores key-value pairs with values up to (approximately) 63k in size

@item 
works with many underlay network topologies (small-world, random
graph), underlay does not need to be a full mesh / clique

@item 
support for extended queries (more than just a simple 'key'),
filtering duplicate replies within the network (bloomfilter) and
content validation (for details, please read the subsection on the
block library)

@item 
can (optionally) return paths taken by the PUT and GET operations to
the application

@item 
provides content replication to handle churn
@end itemize

GNUnet's DHT is randomized and unreliable. Unreliable means that there
is no strict guarantee that a value stored in the DHT is always found
— values are only found with high probability. While this is somewhat
true in all P2P DHTs, GNUnet developers should be particularly wary of
this fact (this will help you write secure, fault-tolerant code). Thus,
when writing any application using the DHT, you should always consider
the possibility that a value stored in the DHT by you or some other peer
might simply not be returned, or returned with a significant delay. Your
application logic must be written to tolerate this (naturally, some loss
of performance or quality of service is expected in this case).

@menu
* Block library and plugins:: 

@end menu

@node Block library and plugins,,,DHT - Distributed Hash Table
@anchor{users/subsystems block-library-and-plugins}@anchor{2d}@anchor{users/subsystems id3}@anchor{2e}
@subsubsection Block library and plugins


@menu
* What is a Block?:: 

@end menu

@node What is a Block?,,,Block library and plugins
@anchor{users/subsystems what-is-a-block}@anchor{2f}@anchor{users/subsystems what-is-a-block-003f}@anchor{30}
@subsubsection What is a Block?


Blocks are small (< 63k) pieces of data stored under a key (struct
GNUNET_HashCode). Blocks have a type (enum GNUNET_BlockType) which
defines their data format. Blocks are used in GNUnet as units of static
data exchanged between peers and stored (or cached) locally. Uses of
blocks include file-sharing (the files are broken up into blocks), the
VPN (DNS information is stored in blocks) and the DHT (all information
in the DHT and meta-information for the maintenance of the DHT are both
stored using blocks). The block subsystem provides a few common
functions that must be available for any type of block.

@anchor{users/subsystems r5n2011}@anchor{2b}@w{(R5N2011)} 
@indicateurl{https://bib.gnunet.org/date.html#R5N}

@anchor{users/subsystems evans2011}@anchor{2c}@w{(EVANS2011)} 
@indicateurl{https://d-nb.info/1015129951}

@geindex File sharing; subsystem
@geindex FS; File sharing

@node FS — File sharing over GNUnet,NAMECACHE — DHT caching of GNS results,DHT - Distributed Hash Table,Subsystems
@anchor{users/subsystems file-002dsharing-0028fs-0029-subsystem}@anchor{31}@anchor{users/subsystems fs-file-sharing-over-gnunet}@anchor{32}
@subsection FS — File sharing over GNUnet


This chapter describes the details of how the file-sharing service
works. As with all services, it is split into an API (libgnunetfs), the
service process (gnunet-service-fs) and user interface(s). The
file-sharing service uses the datastore service to store blocks and the
DHT (and indirectly datacache) for lookups for non-anonymous
file-sharing. Furthermore, the file-sharing service uses the block
library (and the block fs plugin) for validation of DHT operations.

In contrast to many other services, libgnunetfs is rather complex since
the client library includes a large number of high-level abstractions;
this is necessary since the FS service itself largely only operates on
the block level. The FS library is responsible for providing a
file-based abstraction to applications, including directories, meta
data, keyword search, verification, and so on.

The method used by GNUnet to break large files into blocks and to use
keyword search is called the "Encoding for Censorship Resistant
Sharing" (ECRS). ECRS is largely implemented in the fs library; block
validation is also reflected in the block FS plugin and the FS service.
ECRS on-demand encoding is implemented in the FS service.

@cartouche
@quotation Note 
The documentation in this chapter is quite incomplete.
@end quotation
@end cartouche

@geindex Encoding for Censorship-Resistant Sharing; ECRS
@anchor{users/subsystems encoding-for-censorship-002dresistant-sharing-0028ecrs-0029}@anchor{33}
@geindex ECRS
ECRS — Encoding for Censorship-Resistant Sharing
ECRS — Encoding for Censorship-Resistant Sharing
------------------------------------------------

When GNUnet shares files, it uses a content encoding that is called
ECRS, the Encoding for Censorship-Resistant Sharing. Most of ECRS is
described in the (so far unpublished) research paper attached to this
page. ECRS obsoletes the previous ESED and ESED II encodings which were
used in GNUnet before version 0.7.0. The rest of this page assumes that
the reader is familiar with the attached paper. What follows is a
description of some minor extensions that GNUnet makes over what is
described in the paper. The reason why these extensions are not in the
paper is that we felt that they were obvious or trivial extensions to
the original scheme and thus did not warrant space in the research
report.

@cartouche
@quotation Todo 
Find missing link to file system paper.
@end quotation
@end cartouche

@geindex GNU Name System; subsystem
@geindex GNS; GNU Name System

@menu
* GNS - the GNU Name system:: 

@end menu

@node GNS - the GNU Name system,,,FS — File sharing over GNUnet
@anchor{users/subsystems gns-the-gnu-name-system}@anchor{34}@anchor{users/subsystems gnu-name-system-0028gns-0029}@anchor{35}
@subsubsection GNS - the GNU Name system


The GNU Name System (GNS) is a decentralized database that enables users
to securely resolve names to values. Names can be used to identify other
users (for example, in social networking), or network services (for
example, VPN services running at a peer in GNUnet, or purely IP-based
services on the Internet). Users interact with GNS by typing in a
hostname that ends in a top-level domain that is configured in the "GNS"
section, matches an identity of the user or ends in a Base32-encoded
public key.

Videos giving an overview of most of the GNS and the motivations behind
it is available here and here. The remainder of this chapter targets
developers that are familiar with high level concepts of GNS as
presented in these talks.

@cartouche
@quotation Todo 
Link to videos and GNS talks?
@end quotation
@end cartouche

GNS-aware applications should use the GNS resolver to obtain the
respective records that are stored under that name in GNS. Each record
consists of a type, value, expiration time and flags.

The type specifies the format of the value. Types below 65536 correspond
to DNS record types, larger values are used for GNS-specific records.
Applications can define new GNS record types by reserving a number and
implementing a plugin (which mostly needs to convert the binary value
representation to a human-readable text format and vice-versa). The
expiration time specifies how long the record is to be valid. The GNS
API ensures that applications are only given non-expired values. The
flags are typically irrelevant for applications, as GNS uses them
internally to control visibility and validity of records.

Records are stored along with a signature. The signature is generated
using the private key of the authoritative zone. This allows any GNS
resolver to verify the correctness of a name-value mapping.

Internally, GNS uses the NAMECACHE to cache information obtained from
other users, the NAMESTORE to store information specific to the local
users, and the DHT to exchange data between users. A plugin API is used
to enable applications to define new GNS record types.

@geindex GNS; name cache
@geindex subsystem; NAMECACHE

@node NAMECACHE — DHT caching of GNS results,NAMESTORE — Storage of local GNS zones,FS — File sharing over GNUnet,Subsystems
@anchor{users/subsystems gns-namecache}@anchor{36}@anchor{users/subsystems namecache-dht-caching-of-gns-results}@anchor{37}
@subsection NAMECACHE — DHT caching of GNS results


The NAMECACHE subsystem is responsible for caching (encrypted)
resolution results of the GNU Name System (GNS). GNS makes zone
information available to other users via the DHT. However, as accessing
the DHT for every lookup is expensive (and as the DHT's local cache is
lost whenever the peer is restarted), GNS uses the NAMECACHE as a more
persistent cache for DHT lookups. Thus, instead of always looking up
every name in the DHT, GNS first checks if the result is already
available locally in the NAMECACHE. Only if there is no result in the
NAMECACHE, GNS queries the DHT. The NAMECACHE stores data in the same
(encrypted) format as the DHT. It thus makes no sense to iterate over
all items in the NAMECACHE – the NAMECACHE does not have a way to
provide the keys required to decrypt the entries.

Blocks in the NAMECACHE share the same expiration mechanism as blocks in
the DHT – the block expires wheneever any of the records in the
(encrypted) block expires. The expiration time of the block is the only
information stored in plaintext. The NAMECACHE service internally
performs all of the required work to expire blocks, clients do not have
to worry about this. Also, given that NAMECACHE stores only GNS blocks
that local users requested, there is no configuration option to limit
the size of the NAMECACHE. It is assumed to be always small enough (a
few MB) to fit on the drive.

The NAMECACHE supports the use of different database backends via a
plugin API.

@geindex subsystem; NAMESTORE

@node NAMESTORE — Storage of local GNS zones,HOSTLIST — HELLO bootstrapping and gossip,NAMECACHE — DHT caching of GNS results,Subsystems
@anchor{users/subsystems namestore-storage-of-local-gns-zones}@anchor{38}@anchor{users/subsystems namestore-subsystem}@anchor{39}
@subsection NAMESTORE — Storage of local GNS zones


The NAMESTORE subsystem provides persistent storage for local GNS zone
information. All local GNS zone information are managed by NAMESTORE. It
provides both the functionality to administer local GNS information
(e.g. delete and add records) as well as to retrieve GNS information
(e.g to list name information in a client). NAMESTORE does only manage
the persistent storage of zone information belonging to the user running
the service: GNS information from other users obtained from the DHT are
stored by the NAMECACHE subsystem.

NAMESTORE uses a plugin-based database backend to store GNS information
with good performance. Here sqlite and PostgreSQL are supported
database backends. NAMESTORE clients interact with the IDENTITY
subsystem to obtain cryptographic information about zones based on egos
as described with the IDENTITY subsystem, but internally NAMESTORE
refers to zones using the respective private key.

NAMESTORE is queried and monitored by the ZONEMASTER service which periodically
publishes public records of GNS zones. ZONEMASTER also
collaborates with the NAMECACHE subsystem and stores zone information
when local information are modified in the NAMECACHE cache to increase look-up
performance for local information and to enable local access to private records
in zones through GNS.

NAMESTORE provides functionality to look-up and store records, to
iterate over a specific or all zones and to monitor zones for changes.
NAMESTORE functionality can be accessed using the NAMESTORE C API, the NAMESTORE
REST API, or the NAMESTORE command line tool.

@geindex HOSTLIST; subsystem

@node HOSTLIST — HELLO bootstrapping and gossip,IDENTITY — Ego management,NAMESTORE — Storage of local GNS zones,Subsystems
@anchor{users/subsystems hostlist-hello-bootstrapping-and-gossip}@anchor{3a}@anchor{users/subsystems hostlist-subsystem}@anchor{3b}
@subsection HOSTLIST — HELLO bootstrapping and gossip


Peers in the GNUnet overlay network need address information so that
they can connect with other peers. GNUnet uses so called HELLO messages
to store and exchange peer addresses. GNUnet provides several methods
for peers to obtain this information:


@itemize -

@item 
out-of-band exchange of HELLO messages (manually, using for example
gnunet-core)

@item 
HELLO messages shipped with GNUnet (automatic with distribution)

@item 
UDP neighbor discovery in LAN (IPv4 broadcast, IPv6 multicast)

@item 
topology gossiping (learning from other peers we already connected
to), and

@item 
the HOSTLIST daemon covered in this section, which is particularly
relevant for bootstrapping new peers.
@end itemize

New peers have no existing connections (and thus cannot learn from
gossip among peers), may not have other peers in their LAN and might be
started with an outdated set of HELLO messages from the distribution. In
this case, getting new peers to connect to the network requires either
manual effort or the use of a HOSTLIST to obtain HELLOs.

@menu
* HELLOs:: 
* Overview for the HOSTLIST subsystem:: 
* Interacting with the HOSTLIST daemon:: 
* Hostlist security address validation:: 
* The HOSTLIST client:: 
* Usage:: 

@end menu

@node HELLOs,Overview for the HOSTLIST subsystem,,HOSTLIST — HELLO bootstrapping and gossip
@anchor{users/subsystems hellos}@anchor{3c}@anchor{users/subsystems id6}@anchor{3d}
@subsubsection HELLOs


The basic information peers require to connect to other peers are
contained in so called HELLO messages you can think of as a business
card. Besides the identity of the peer (based on the cryptographic
public key) a HELLO message may contain address information that
specifies ways to contact a peer. By obtaining HELLO messages, a peer
can learn how to contact other peers.

@node Overview for the HOSTLIST subsystem,Interacting with the HOSTLIST daemon,HELLOs,HOSTLIST — HELLO bootstrapping and gossip
@anchor{users/subsystems id7}@anchor{3e}@anchor{users/subsystems overview-for-the-hostlist-subsystem}@anchor{3f}
@subsubsection Overview for the HOSTLIST subsystem


The HOSTLIST subsystem provides a way to distribute and obtain contact
information to connect to other peers using a simple HTTP GET request.
Its implementation is split in three parts, the main file for the
daemon itself (@code{gnunet-daemon-hostlist.c}), the HTTP client used to
download peer information (@code{hostlist-client.c}) and the server
component used to provide this information to other peers
(@code{hostlist-server.c}). The server is basically a small HTTP web server
(based on GNU libmicrohttpd) which provides a list of HELLOs known to
the local peer for download. The client component is basically a HTTP
client (based on libcurl) which can download hostlists from one or more
websites. The hostlist format is a binary blob containing a sequence of
HELLO messages. Note that any HTTP server can theoretically serve a
hostlist, the built-in hostlist server makes it simply convenient to
offer this service.

@menu
* Features:: 
* HOSTLIST - Limitations:: 

@end menu

@node Features,HOSTLIST - Limitations,,Overview for the HOSTLIST subsystem
@anchor{users/subsystems features}@anchor{40}@anchor{users/subsystems id8}@anchor{41}
@subsubsection Features


The HOSTLIST daemon can:


@itemize -

@item 
provide HELLO messages with validated addresses obtained from
PEERINFO to download for other peers

@item 
download HELLO messages and forward these message to the TRANSPORT
subsystem for validation

@item 
advertises the URL of this peer's hostlist address to other peers via
gossip

@item 
automatically learn about hostlist servers from the gossip of other
peers
@end itemize

@node HOSTLIST - Limitations,,Features,Overview for the HOSTLIST subsystem
@anchor{users/subsystems hostlist-002d-limitations}@anchor{42}@anchor{users/subsystems hostlist-limitations}@anchor{43}
@subsubsection HOSTLIST - Limitations


The HOSTLIST daemon does not:


@itemize -

@item 
verify the cryptographic information in the HELLO messages

@item 
verify the address information in the HELLO messages
@end itemize

@node Interacting with the HOSTLIST daemon,Hostlist security address validation,Overview for the HOSTLIST subsystem,HOSTLIST — HELLO bootstrapping and gossip
@anchor{users/subsystems id9}@anchor{44}@anchor{users/subsystems interacting-with-the-hostlist-daemon}@anchor{45}
@subsubsection Interacting with the HOSTLIST daemon


The HOSTLIST subsystem is currently implemented as a daemon, so there is
no need for the user to interact with it and therefore there is no
command line tool and no API to communicate with the daemon. In the
future, we can envision changing this to allow users to manually trigger
the download of a hostlist.

Since there is no command line interface to interact with HOSTLIST, the
only way to interact with the hostlist is to use STATISTICS to obtain or
modify information about the status of HOSTLIST:

@example
$ gnunet-statistics -s hostlist
@end example

In particular, HOSTLIST includes a `persistent' value in statistics
that specifies when the hostlist server might be queried next. As this
value is exponentially increasing during runtime, developers may want to
reset or manually adjust it. Note that HOSTLIST (but not STATISTICS)
needs to be shutdown if changes to this value are to have any effect on
the daemon (as HOSTLIST does not monitor STATISTICS for changes to the
download frequency).

@node Hostlist security address validation,The HOSTLIST client,Interacting with the HOSTLIST daemon,HOSTLIST — HELLO bootstrapping and gossip
@anchor{users/subsystems hostlist-security-address-validation}@anchor{46}@anchor{users/subsystems id10}@anchor{47}
@subsubsection Hostlist security address validation


Since information obtained from other parties cannot be trusted without
validation, we have to distinguish between `validated' and `not
validated' addresses. Before using (and so trusting) information from
other parties, this information has to be double-checked (validated).
Address validation is not done by HOSTLIST but by the TRANSPORT service.

The HOSTLIST component is functionally located between the PEERINFO and
the TRANSPORT subsystem. When acting as a server, the daemon obtains
valid (`validated') peer information (HELLO messages) from the PEERINFO
service and provides it to other peers. When acting as a client, it
contacts the HOSTLIST servers specified in the configuration, downloads
the (unvalidated) list of HELLO messages and forwards these information
to the TRANSPORT server to validate the addresses.
@anchor{users/subsystems the-hostlist-daemon}@anchor{48}
@geindex daemon; HOSTLIST
The HOSTLIST daemon
The HOSTLIST daemon
-------------------

The hostlist daemon is the main component of the HOSTLIST subsystem. It
is started by the ARM service and (if configured) starts the HOSTLIST
client and server components.

GNUNET_MESSAGE_TYPE_HOSTLIST_ADVERTISEMENT
If the daemon provides a hostlist itself it can advertise it's own
hostlist to other peers. To do so it sends a
@code{GNUNET_MESSAGE_TYPE_HOSTLIST_ADVERTISEMENT} message to other peers
when they connect to this peer on the CORE level. This hostlist
advertisement message contains the URL to access the HOSTLIST HTTP
server of the sender. The daemon may also subscribe to this type of
message from CORE service, and then forward these kind of message to the
HOSTLIST client. The client then uses all available URLs to download
peer information when necessary.

When starting, the HOSTLIST daemon first connects to the CORE subsystem
and if hostlist learning is enabled, registers a CORE handler to receive
this kind of messages. Next it starts (if configured) the client and
server. It passes pointers to CORE connect and disconnect and receive
handlers where the client and server store their functions, so the
daemon can notify them about CORE events.

To clean up on shutdown, the daemon has a cleaning task, shutting down
all subsystems and disconnecting from CORE.
@anchor{users/subsystems the-hostlist-server}@anchor{49}
@geindex HOSTLIST; server
The HOSTLIST server
The HOSTLIST server
-------------------

The server provides a way for other peers to obtain HELLOs. Basically it
is a small web server other peers can connect to and download a list of
HELLOs using standard HTTP; it may also advertise the URL of the
hostlist to other peers connecting on CORE level.

@menu
* The HTTP Server:: 
* Advertising the URL:: 

@end menu

@node The HTTP Server,Advertising the URL,,Hostlist security address validation
@anchor{users/subsystems id11}@anchor{4a}@anchor{users/subsystems the-http-server}@anchor{4b}
@subsubsection The HTTP Server


During startup, the server starts a web server listening on the port
specified with the HTTPPORT value (default 8080). In addition it
connects to the PEERINFO service to obtain peer information. The
HOSTLIST server uses the GNUNET_PEERINFO_iterate function to request
HELLO information for all peers and adds their information to a new
hostlist if they are suitable (expired addresses and HELLOs without
addresses are both not suitable) and the maximum size for a hostlist is
not exceeded (MAX_BYTES_PER_HOSTLISTS = 500000). When PEERINFO finishes
(with a last NULL callback), the server destroys the previous hostlist
response available for download on the web server and replaces it with
the updated hostlist. The hostlist format is basically a sequence of
HELLO messages (as obtained from PEERINFO) without any special
tokenization. Since each HELLO message contains a size field, the
response can easily be split into separate HELLO messages by the client.

A HOSTLIST client connecting to the HOSTLIST server will receive the
hostlist as an HTTP response and the server will terminate the
connection with the result code @code{HTTP 200 OK}. The connection will be
closed immediately if no hostlist is available.

@node Advertising the URL,,The HTTP Server,Hostlist security address validation
@anchor{users/subsystems advertising-the-url}@anchor{4c}@anchor{users/subsystems id12}@anchor{4d}
@subsubsection Advertising the URL


The server also advertises the URL to download the hostlist to other
peers if hostlist advertisement is enabled. When a new peer connects and
has hostlist learning enabled, the server sends a
@code{GNUNET_MESSAGE_TYPE_HOSTLIST_ADVERTISEMENT} message to this peer
using the CORE service.

HOSTLIST client
.. _The-HOSTLIST-client:

@node The HOSTLIST client,Usage,Hostlist security address validation,HOSTLIST — HELLO bootstrapping and gossip
@anchor{users/subsystems the-hostlist-client}@anchor{4e}
@subsubsection The HOSTLIST client


The client provides the functionality to download the list of HELLOs
from a set of URLs. It performs a standard HTTP request to the URLs
configured and learned from advertisement messages received from other
peers. When a HELLO is downloaded, the HOSTLIST client forwards the
HELLO to the TRANSPORT service for validation.

The client supports two modes of operation:


@itemize -

@item 
download of HELLOs (bootstrapping)

@item 
learning of URLs
@end itemize

@menu
* Bootstrapping:: 

@end menu

@node Bootstrapping,,,The HOSTLIST client
@anchor{users/subsystems bootstrapping}@anchor{4f}@anchor{users/subsystems id13}@anchor{50}
@subsubsection Bootstrapping


For bootstrapping, it schedules a task to download the hostlist from the
set of known URLs. The downloads are only performed if the number of
current connections is smaller than a minimum number of connections (at
the moment 4). The interval between downloads increases exponentially;
however, the exponential growth is limited if it becomes longer than an
hour. At that point, the frequency growth is capped at (#number of
connections * 1h).

Once the decision has been taken to download HELLOs, the daemon chooses
a random URL from the list of known URLs. URLs can be configured in the
configuration or be learned from advertisement messages. The client uses
a HTTP client library (libcurl) to initiate the download using the
libcurl multi interface. Libcurl passes the data to the
callback_download function which stores the data in a buffer if space is
available and the maximum size for a hostlist download is not exceeded
(MAX_BYTES_PER_HOSTLISTS = 500000). When a full HELLO was downloaded,
the HOSTLIST client offers this HELLO message to the TRANSPORT service
for validation. When the download is finished or failed, statistical
information about the quality of this URL is updated.
@anchor{users/subsystems learning}@anchor{51}
@geindex HOSTLIST; learning
Learning
Learning
^^^^^^^^

The client also manages hostlist advertisements from other peers. The
HOSTLIST daemon forwards @code{GNUNET_MESSAGE_TYPE_HOSTLIST_ADVERTISEMENT}
messages to the client subsystem, which extracts the URL from the
message. Next, a test of the newly obtained URL is performed by
triggering a download from the new URL. If the URL works correctly, it
is added to the list of working URLs.

The size of the list of URLs is restricted, so if an additional server
is added and the list is full, the URL with the worst quality ranking
(determined through successful downloads and number of HELLOs e.g.) is
discarded. During shutdown the list of URLs is saved to a file for
persistence and loaded on startup. URLs from the configuration file are
never discarded.

@node Usage,,The HOSTLIST client,HOSTLIST — HELLO bootstrapping and gossip
@anchor{users/subsystems id14}@anchor{52}@anchor{users/subsystems usage}@anchor{53}
@subsubsection Usage


To start HOSTLIST by default, it has to be added to the DEFAULTSERVICES
section for the ARM services. This is done in the default configuration.

For more information on how to configure the HOSTLIST subsystem see the
installation handbook: Configuring the hostlist to bootstrap Configuring
your peer to provide a hostlist

@geindex IDENTITY; subsystem

@node IDENTITY — Ego management,MESSENGER — Room-based end-to-end messaging,HOSTLIST — HELLO bootstrapping and gossip,Subsystems
@anchor{users/subsystems identity-ego-management}@anchor{54}@anchor{users/subsystems identity-subsystem}@anchor{55}
@subsection IDENTITY — Ego management


Identities of "users" in GNUnet are called egos. Egos can be used as
pseudonyms ("fake names") or be tied to an organization (for example,
"GNU") or even the actual identity of a human. GNUnet users are
expected to have many egos. They might have one tied to their real
identity, some for organizations they manage, and more for different
domains where they want to operate under a pseudonym.

The IDENTITY service allows users to manage their egos. The identity
service manages the private keys egos of the local user; it does not
manage identities of other users (public keys). Public keys for other
users need names to become manageable. GNUnet uses the GNU Name System
(GNS) to give names to other users and manage their public keys
securely. This chapter is about the IDENTITY service, which is about the
management of private keys.

On the network, an ego corresponds to an ECDSA key (over Curve25519,
using RFC 6979, as required by GNS). Thus, users can perform actions
under a particular ego by using (signing with) a particular private key.
Other users can then confirm that the action was really performed by
that ego by checking the signature against the respective public key.

The IDENTITY service allows users to associate a human-readable name
with each ego. This way, users can use names that will remind them of
the purpose of a particular ego. The IDENTITY service will store the
respective private keys and allows applications to access key
information by name. Users can change the name that is locally (!)
associated with an ego. Egos can also be deleted, which means that the
private key will be removed and it thus will not be possible to perform
actions with that ego in the future.

Additionally, the IDENTITY subsystem can associate service functions
with egos. For example, GNS requires the ego that should be used for the
shorten zone. GNS will ask IDENTITY for an ego for the "gns-short"
service. The IDENTITY service has a mapping of such service strings to
the name of the ego that the user wants to use for this service, for
example "my-short-zone-ego".

Finally, the IDENTITY API provides access to a special ego, the
anonymous ego. The anonymous ego is special in that its private key is
not really private, but fixed and known to everyone. Thus, anyone can
perform actions as anonymous. This can be useful as with this trick,
code does not have to contain a special case to distinguish between
anonymous and pseudonymous egos.

@geindex subsystem; MESSENGER

@node MESSENGER — Room-based end-to-end messaging,NSE — Network size estimation,IDENTITY — Ego management,Subsystems
@anchor{users/subsystems messenger-room-based-end-to-end-messaging}@anchor{56}@anchor{users/subsystems messenger-subsystem}@anchor{57}
@subsection MESSENGER — Room-based end-to-end messaging


The MESSENGER subsystem is responsible for secure end-to-end
communication in groups of nodes in the GNUnet overlay network.
MESSENGER builds on the CADET subsystem which provides a reliable and
secure end-to-end communication between the nodes inside of these
groups.

Additionally to the CADET security benefits, MESSENGER provides
following properties designed for application level usage:


@itemize -

@item 
MESSENGER provides integrity by signing the messages with the users
provided ego

@item 
MESSENGER adds (optional) forward secrecy by replacing the key pair
of the used ego and signing the propagation of the new one with old
one (chaining egos)

@item 
MESSENGER provides verification of a original sender by checking
against all used egos from a member which are currently in active use
(active use depends on the state of a member session)

@item 
MESSENGER offsers (optional) decentralized message forwarding between
all nodes in a group to improve availability and prevent MITM-attacks

@item 
MESSENGER handles new connections and disconnections from nodes in
the group by reconnecting them preserving an efficient structure for
message distribution (ensuring availability and accountablity)

@item 
MESSENGER provides replay protection (messages can be uniquely
identified via SHA-512, include a timestamp and the hash of the last
message)

@item 
MESSENGER allows detection for dropped messages by chaining them
(messages refer to the last message by their hash) improving
accountability

@item 
MESSENGER allows requesting messages from other peers explicitly to
ensure availability

@item 
MESSENGER provides confidentiality by padding messages to few
different sizes (512 bytes, 4096 bytes, 32768 bytes and maximal
message size from CADET)

@item 
MESSENGER adds (optional) confidentiality with ECDHE to exchange and
use symmetric encryption, encrypting with both AES-256 and Twofish
but allowing only selected members to decrypt (using the receivers
ego for ECDHE)
@end itemize

Also MESSENGER provides multiple features with privacy in mind:


@itemize -

@item 
MESSENGER allows deleting messages from all peers in the group by the
original sender (uses the MESSENGER provided verification)

@item 
MESSENGER allows using the publicly known anonymous ego instead of
any unique identifying ego

@item 
MESSENGER allows your node to decide between acting as host of the
used messaging room (sharing your peer's identity with all nodes in
the group) or acting as guest (sharing your peer's identity only with
the nodes you explicitly open a connection to)

@item 
MESSENGER handles members independently of the peer's identity making
forwarded messages indistinguishable from directly received ones (
complicating the tracking of messages and identifying its origin)

@item 
MESSENGER allows names of members being not unique (also names are
optional)

@item 
MESSENGER does not include information about the selected receiver of
an explicitly encrypted message in its header, complicating it for
other members to draw conclusions from communication partners
@end itemize

@geindex subsystem; Network size estimation
@geindex NSE; Network size estimation

@node NSE — Network size estimation,PEERINFO — Persistent HELLO storage,MESSENGER — Room-based end-to-end messaging,Subsystems
@anchor{users/subsystems nse-network-size-estimation}@anchor{58}@anchor{users/subsystems nse-subsystem}@anchor{59}
@subsection NSE — Network size estimation


NSE stands for Network Size Estimation. The NSE subsystem provides other
subsystems and users with a rough estimate of the number of peers
currently participating in the GNUnet overlay. The computed value is not
a precise number as producing a precise number in a decentralized,
efficient and secure way is impossible. While NSE's estimate is
inherently imprecise, NSE also gives the expected range. For a peer that
has been running in a stable network for a while, the real network size
will typically (99.7% of the time) be in the range of [2/3 estimate, 3/2
estimate]. We will now give an overview of the algorithm used to
calculate the estimate; all of the details can be found in this
technical report.

@cartouche
@quotation Todo 
link to the report.
@end quotation
@end cartouche

@menu
* Motivation:: 

@end menu

@node Motivation,,,NSE — Network size estimation
@anchor{users/subsystems id16}@anchor{5a}@anchor{users/subsystems motivation}@anchor{5b}
@subsubsection Motivation


Some subsystems, like DHT, need to know the size of the GNUnet network
to optimize some parameters of their own protocol. The decentralized
nature of GNUnet makes efficient and securely counting the exact number
of peers infeasible. Although there are several decentralized algorithms
to count the number of peers in a system, so far there is none to do so
securely. Other protocols may allow any malicious peer to manipulate the
final result or to take advantage of the system to perform Denial of
Service (DoS) attacks against the network. GNUnet's NSE protocol avoids
these drawbacks.

NSE security
.. _Security:

@geindex NSE; security
Security
Security
^^^^^^^^

The NSE subsystem is designed to be resilient against these attacks. It
uses proofs of work@footnote{http://en.wikipedia.org/wiki/Proof-of-work_system} to prevent
one peer from impersonating a large number of participants, which would
otherwise allow an adversary to artificially inflate the estimate. The
DoS protection comes from the time-based nature of the protocol: the
estimates are calculated periodically and out-of-time traffic is either
ignored or stored for later retransmission by benign peers. In
particular, peers cannot trigger global network communication at will.
@anchor{users/subsystems principle}@anchor{5c}
@geindex NSE; principle of operation
Principle
Principle
---------

The algorithm calculates the estimate by finding the globally closest
peer ID to a random, time-based value.

The idea is that the closer the ID is to the random value, the more
"densely packed" the ID space is, and therefore, more peers are in the
network.

@menu
* Example:: 
* Algorithm:: 
* Target value:: 
* Timing:: 
* Controlled Flooding:: 
* Calculating the estimate:: 

@end menu

@node Example,Algorithm,,Motivation
@anchor{users/subsystems example}@anchor{5d}@anchor{users/subsystems id17}@anchor{5e}
@subsubsection Example


Suppose all peers have IDs between 0 and 100 (our ID space), and the
random value is 42. If the closest peer has the ID 70 we can imagine
that the average "distance" between peers is around 30 and therefore
the are around 3 peers in the whole ID space. On the other hand, if the
closest peer has the ID 44, we can imagine that the space is rather
packed with peers, maybe as much as 50 of them. Naturally, we could have
been rather unlucky, and there is only one peer and happens to have the
ID 44. Thus, the current estimate is calculated as the average over
multiple rounds, and not just a single sample.

@node Algorithm,Target value,Example,Motivation
@anchor{users/subsystems algorithm}@anchor{5f}@anchor{users/subsystems id18}@anchor{60}
@subsubsection Algorithm


Given that example, one can imagine that the job of the subsystem is to
efficiently communicate the ID of the closest peer to the target value
to all the other peers, who will calculate the estimate from it.

@node Target value,Timing,Algorithm,Motivation
@anchor{users/subsystems id19}@anchor{61}@anchor{users/subsystems target-value}@anchor{62}
@subsubsection Target value


The target value itself is generated by hashing the current time,
rounded down to an agreed value. If the rounding amount is 1h (default)
and the time is 12:34:56, the time to hash would be 12:00:00. The
process is repeated each rounding amount (in this example would be every
hour). Every repetition is called a round.

@node Timing,Controlled Flooding,Target value,Motivation
@anchor{users/subsystems id20}@anchor{63}@anchor{users/subsystems timing}@anchor{64}
@subsubsection Timing


The NSE subsystem has some timing control to avoid everybody
broadcasting its ID all at one. Once each peer has the target random
value, it compares its own ID to the target and calculates the
hypothetical size of the network if that peer were to be the closest.
Then it compares the hypothetical size with the estimate from the
previous rounds. For each value there is an associated point in the
period, let's call it "broadcast time". If its own hypothetical
estimate is the same as the previous global estimate, its "broadcast
time" will be in the middle of the round. If its bigger it will be
earlier and if its smaller (the most likely case) it will be later. This
ensures that the peers closest to the target value start broadcasting
their ID the first.

@node Controlled Flooding,Calculating the estimate,Timing,Motivation
@anchor{users/subsystems controlled-flooding}@anchor{65}@anchor{users/subsystems id21}@anchor{66}
@subsubsection Controlled Flooding


When a peer receives a value, first it verifies that it is closer than
the closest value it had so far, otherwise it answers the incoming
message with a message containing the better value. Then it checks a
proof of work that must be included in the incoming message, to ensure
that the other peer's ID is not made up (otherwise a malicious peer
could claim to have an ID of exactly the target value every round). Once
validated, it compares the broadcast time of the received value with the
current time and if it's not too early, sends the received value to its
neighbors. Otherwise it stores the value until the correct broadcast
time comes. This prevents unnecessary traffic of sub-optimal values,
since a better value can come before the broadcast time, rendering the
previous one obsolete and saving the traffic that would have been used
to broadcast it to the neighbors.

@node Calculating the estimate,,Controlled Flooding,Motivation
@anchor{users/subsystems calculating-the-estimate}@anchor{67}@anchor{users/subsystems id22}@anchor{68}
@subsubsection Calculating the estimate


Once the closest ID has been spread across the network each peer gets
the exact distance between this ID and the target value of the round and
calculates the estimate with a mathematical formula described in the
tech report. The estimate generated with this method for a single round
is not very precise. Remember the case of the example, where the only
peer is the ID 44 and we happen to generate the target value 42,
thinking there are 50 peers in the network. Therefore, the NSE subsystem
remembers the last 64 estimates and calculates an average over them,
giving a result of which usually has one bit of uncertainty (the real
size could be half of the estimate or twice as much). Note that the
actual network size is calculated in powers of two of the raw input,
thus one bit of uncertainty means a factor of two in the size estimate.

@geindex subsystem; PEERINFO

@node PEERINFO — Persistent HELLO storage,PEERSTORE — Extensible local persistent data storage,NSE — Network size estimation,Subsystems
@anchor{users/subsystems peerinfo-persistent-hello-storage}@anchor{69}@anchor{users/subsystems peerinfo-subsystem}@anchor{6a}
@subsection PEERINFO — Persistent HELLO storage


The PEERINFO subsystem is used to store verified (validated) information
about known peers in a persistent way. It obtains these addresses for
example from TRANSPORT service which is in charge of address validation.
Validation means that the information in the HELLO message are checked
by connecting to the addresses and performing a cryptographic handshake
to authenticate the peer instance stating to be reachable with these
addresses. Peerinfo does not validate the HELLO messages itself but only
stores them and gives them to interested clients.

As future work, we think about moving from storing just HELLO messages
to providing a generic persistent per-peer information store. More and
more subsystems tend to need to store per-peer information in persistent
way. To not duplicate this functionality we plan to provide a PEERSTORE
service providing this functionality.

@menu
* PEERINFO - Features:: 
* PEERINFO - Limitations:: 
* DeveloperPeer Information:: 
* Startup:: 
* Managing Information:: 
* Obtaining Information:: 

@end menu

@node PEERINFO - Features,PEERINFO - Limitations,,PEERINFO — Persistent HELLO storage
@anchor{users/subsystems peerinfo-002d-features}@anchor{6b}@anchor{users/subsystems peerinfo-features}@anchor{6c}
@subsubsection PEERINFO - Features



@itemize -

@item 
Persistent storage

@item 
Client notification mechanism on update

@item 
Periodic clean up for expired information

@item 
Differentiation between public and friend-only HELLO
@end itemize

@node PEERINFO - Limitations,DeveloperPeer Information,PEERINFO - Features,PEERINFO — Persistent HELLO storage
@anchor{users/subsystems peerinfo-002d-limitations}@anchor{6d}@anchor{users/subsystems peerinfo-limitations}@anchor{6e}
@subsubsection PEERINFO - Limitations



@itemize -

@item 
Does not perform HELLO validation
@end itemize

@node DeveloperPeer Information,Startup,PEERINFO - Limitations,PEERINFO — Persistent HELLO storage
@anchor{users/subsystems developerpeer-information}@anchor{6f}@anchor{users/subsystems id23}@anchor{70}
@subsubsection DeveloperPeer Information


The PEERINFO subsystem stores these information in the form of HELLO
messages you can think of as business cards. These HELLO messages
contain the public key of a peer and the addresses a peer can be reached
under. The addresses include an expiration date describing how long they
are valid. This information is updated regularly by the TRANSPORT
service by revalidating the address. If an address is expired and not
renewed, it can be removed from the HELLO message.

Some peer do not want to have their HELLO messages distributed to other
peers, especially when GNUnet's friend-to-friend modus is enabled. To
prevent this undesired distribution. PEERINFO distinguishes between
`public' and `friend-only' HELLO messages. Public HELLO messages can be
freely distributed to other (possibly unknown) peers (for example using
the hostlist, gossiping, broadcasting), whereas friend-only HELLO
messages may not be distributed to other peers. Friend-only HELLO
messages have an additional flag @code{friend_only} set internally. For
public HELLO message this flag is not set. PEERINFO does and cannot not
check if a client is allowed to obtain a specific HELLO type.

The HELLO messages can be managed using the GNUnet HELLO library. Other
GNUnet systems can obtain these information from PEERINFO and use it for
their purposes. Clients are for example the HOSTLIST component providing
these information to other peers in form of a hostlist or the TRANSPORT
subsystem using these information to maintain connections to other
peers.

@node Startup,Managing Information,DeveloperPeer Information,PEERINFO — Persistent HELLO storage
@anchor{users/subsystems id24}@anchor{71}@anchor{users/subsystems startup}@anchor{72}
@subsubsection Startup


During startup the PEERINFO services loads persistent HELLOs from disk.
First PEERINFO parses the directory configured in the HOSTS value of the
@code{PEERINFO} configuration section to store PEERINFO information. For
all files found in this directory valid HELLO messages are extracted. In
addition it loads HELLO messages shipped with the GNUnet distribution.
These HELLOs are used to simplify network bootstrapping by providing
valid peer information with the distribution. The use of these HELLOs
can be prevented by setting the @code{USE_INCLUDED_HELLOS} in the
@code{PEERINFO} configuration section to @code{NO}. Files containing invalid
information are removed.

@node Managing Information,Obtaining Information,Startup,PEERINFO — Persistent HELLO storage
@anchor{users/subsystems id25}@anchor{73}@anchor{users/subsystems managing-information}@anchor{74}
@subsubsection Managing Information


The PEERINFO services stores information about known PEERS and a single
HELLO message for every peer. A peer does not need to have a HELLO if no
information are available. HELLO information from different sources, for
example a HELLO obtained from a remote HOSTLIST and a second HELLO
stored on disk, are combined and merged into one single HELLO message
per peer which will be given to clients. During this merge process the
HELLO is immediately written to disk to ensure persistence.

PEERINFO in addition periodically scans the directory where information
are stored for empty HELLO messages with expired TRANSPORT addresses.
This periodic task scans all files in the directory and recreates the
HELLO messages it finds. Expired TRANSPORT addresses are removed from
the HELLO and if the HELLO does not contain any valid addresses, it is
discarded and removed from the disk.

@node Obtaining Information,,Managing Information,PEERINFO — Persistent HELLO storage
@anchor{users/subsystems id26}@anchor{75}@anchor{users/subsystems obtaining-information}@anchor{76}
@subsubsection Obtaining Information


When a client requests information from PEERINFO, PEERINFO performs a
lookup for the respective peer or all peers if desired and transmits
this information to the client. The client can specify if friend-only
HELLOs have to be included or not and PEERINFO filters the respective
HELLO messages before transmitting information.

To notify clients about changes to PEERINFO information, PEERINFO
maintains a list of clients interested in this notifications. Such a
notification occurs if a HELLO for a peer was updated (due to a merge
for example) or a new peer was added.

@geindex subsystem; PEERSTORE

@node PEERSTORE — Extensible local persistent data storage,REGEX — Service discovery using regular expressions,PEERINFO — Persistent HELLO storage,Subsystems
@anchor{users/subsystems peerstore-extensible-local-persistent-data-storage}@anchor{77}@anchor{users/subsystems peerstore-subsystem}@anchor{78}
@subsection PEERSTORE — Extensible local persistent data storage


GNUnet's PEERSTORE subsystem offers persistent per-peer storage for
other GNUnet subsystems. GNUnet subsystems can use PEERSTORE to
persistently store and retrieve arbitrary data. Each data record stored
with PEERSTORE contains the following fields:


@itemize -

@item 
subsystem: Name of the subsystem responsible for the record.

@item 
peerid: Identity of the peer this record is related to.

@item 
key: a key string identifying the record.

@item 
value: binary record value.

@item 
expiry: record expiry date.
@end itemize

@menu
* Functionality:: 
* Architecture:: 

@end menu

@node Functionality,Architecture,,PEERSTORE — Extensible local persistent data storage
@anchor{users/subsystems functionality}@anchor{79}@anchor{users/subsystems id27}@anchor{7a}
@subsubsection Functionality


Subsystems can store any type of value under a (subsystem, peerid, key)
combination. A "replace" flag set during store operations forces the
PEERSTORE to replace any old values stored under the same (subsystem,
peerid, key) combination with the new value. Additionally, an expiry
date is set after which the record is *possibly* deleted by PEERSTORE.

Subsystems can iterate over all values stored under any of the following
combination of fields:


@itemize -

@item 
(subsystem)

@item 
(subsystem, peerid)

@item 
(subsystem, key)

@item 
(subsystem, peerid, key)
@end itemize

Subsystems can also request to be notified about any new values stored
under a (subsystem, peerid, key) combination by sending a "watch"
request to PEERSTORE.

@node Architecture,,Functionality,PEERSTORE — Extensible local persistent data storage
@anchor{users/subsystems architecture}@anchor{7b}@anchor{users/subsystems id28}@anchor{7c}
@subsubsection Architecture


PEERSTORE implements the following components:


@itemize -

@item 
PEERSTORE service: Handles store, iterate and watch operations.

@item 
PEERSTORE API: API to be used by other subsystems to communicate and
issue commands to the PEERSTORE service.

@item 
PEERSTORE plugins: Handles the persistent storage. At the moment,
only an "sqlite" plugin is implemented.
@end itemize

@geindex subsystem; REGEX

@node REGEX — Service discovery using regular expressions,REST — RESTful GNUnet Web APIs,PEERSTORE — Extensible local persistent data storage,Subsystems
@anchor{users/subsystems regex-service-discovery-using-regular-expressions}@anchor{7d}@anchor{users/subsystems regex-subsystem}@anchor{7e}
@subsection REGEX — Service discovery using regular expressions


Using the REGEX subsystem, you can discover peers that offer a
particular service using regular expressions. The peers that offer a
service specify it using a regular expressions. Peers that want to
patronize a service search using a string. The REGEX subsystem will then
use the DHT to return a set of matching offerers to the patrons.

For the technical details, we have Max's defense talk and Max's Master's
thesis.

@cartouche
@quotation Note 
An additional publication is under preparation and available
to team members (in Git).
@end quotation
@end cartouche

@cartouche
@quotation Todo 
Missing links to Max's talk and Master's thesis
@end quotation
@end cartouche

@menu
* How to run the regex profiler:: 

@end menu

@node How to run the regex profiler,,,REGEX — Service discovery using regular expressions
@anchor{users/subsystems how-to-run-the-regex-profiler}@anchor{7f}@anchor{users/subsystems id30}@anchor{80}
@subsubsection How to run the regex profiler


The gnunet-regex-profiler can be used to profile the usage of mesh/regex
for a given set of regular expressions and strings. Mesh/regex allows
you to announce your peer ID under a certain regex and search for peers
matching a particular regex using a string. See
szengel2012ms@footnote{https://bib.gnunet.org/full/date.html#2012_5f2} for a
full introduction.

First of all, the regex profiler uses GNUnet testbed, thus all the
implications for testbed also apply to the regex profiler (for example
you need password-less ssh login to the machines listed in your hosts
file).

`Configuration'

Moreover, an appropriate configuration file is needed. In the following
paragraph the important details are highlighted.

Announcing of the regular expressions is done by the
gnunet-daemon-regexprofiler, therefore you have to make sure it is
started, by adding it to the START_ON_DEMAND set of ARM:

@example
[regexprofiler]
START_ON_DEMAND = YES
@end example

Furthermore you have to specify the location of the binary:

@example
[regexprofiler]
# Location of the gnunet-daemon-regexprofiler binary.
BINARY = /home/szengel/gnunet/src/mesh/.libs/gnunet-daemon-regexprofiler
# Regex prefix that will be applied to all regular expressions and
# search string.
REGEX_PREFIX = "GNVPN-0001-PAD"
@end example

When running the profiler with a large scale deployment, you probably
want to reduce the workload of each peer. Use the following options to
do this.

@example
[dht]
# Force network size estimation
FORCE_NSE = 1

[dhtcache]
DATABASE = heap
# Disable RC-file for Bloom filter? (for benchmarking with limited IO
# availability)
DISABLE_BF_RC = YES
# Disable Bloom filter entirely
DISABLE_BF = YES

[nse]
# Minimize proof-of-work CPU consumption by NSE
WORKBITS = 1
@end example

`Options'

To finally run the profiler some options and the input data need to be
specified on the command line.

@example
gnunet-regex-profiler -c config-file -d log-file -n num-links \
-p path-compression-length -s search-delay -t matching-timeout \
-a num-search-strings hosts-file policy-dir search-strings-file
@end example

Where...


@itemize -

@item 
... @code{config-file} means the configuration file created earlier.

@item 
... @code{log-file} is the file where to write statistics output.

@item 
... @code{num-links} indicates the number of random links between
started peers.

@item 
... @code{path-compression-length} is the maximum path compression
length in the DFA.

@item 
... @code{search-delay} time to wait between peers finished linking and
starting to match strings.

@item 
... @code{matching-timeout} timeout after which to cancel the
searching.

@item 
... @code{num-search-strings} number of strings in the
search-strings-file.

@item 
... the @code{hosts-file} should contain a list of hosts for the
testbed, one per line in the following format:


@itemize -

@item 
@code{user@@host_ip:port}
@end itemize

@item 
... the @code{policy-dir} is a folder containing text files containing
one or more regular expressions. A peer is started for each file in
that folder and the regular expressions in the corresponding file are
announced by this peer.

@item 
... the @code{search-strings-file} is a text file containing search
strings, one in each line.
@end itemize

You can create regular expressions and search strings for every AS in
the Internet using the attached scripts. You need one of the CAIDA routeviews prefix2as@footnote{http://data.caida.org/datasets/routing/routeviews-prefix2as/}
data files for this. Run

@example
create_regex.py <filename> <output path>
@end example

to create the regular expressions and

@example
create_strings.py <input path> <outfile>
@end example

to create a search strings file from the previously created regular
expressions.

@geindex subsystem; REST

@node REST — RESTful GNUnet Web APIs,REVOCATION — Ego key revocation,REGEX — Service discovery using regular expressions,Subsystems
@anchor{users/subsystems rest-restful-gnunet-web-apis}@anchor{81}@anchor{users/subsystems rest-subsystem}@anchor{82}
@subsection REST — RESTful GNUnet Web APIs


@cartouche
@quotation Todo 
Define REST
@end quotation
@end cartouche

Using the REST subsystem, you can expose REST-based APIs or services.
The REST service is designed as a pluggable architecture.

`Configuration'

The REST service can be configured in various ways. The reference config
file can be found in @code{src/rest/rest.conf}:

@example
[rest]
REST_PORT=7776
REST_ALLOW_HEADERS=Authorization,Accept,Content-Type
REST_ALLOW_ORIGIN=*
REST_ALLOW_CREDENTIALS=true
@end example

The port as well as CORS (cross-origin resource sharing) headers
that are supposed to be advertised by the rest service are configurable.

@geindex subsystem; REVOCATION

@node REVOCATION — Ego key revocation,RPS — Random peer sampling,REST — RESTful GNUnet Web APIs,Subsystems
@anchor{users/subsystems revocation-ego-key-revocation}@anchor{83}@anchor{users/subsystems revocation-subsystem}@anchor{84}
@subsection REVOCATION — Ego key revocation


The REVOCATION subsystem is responsible for key revocation of Egos. If a
user learns that their private key has been compromised or has lost it,
they can use the REVOCATION system to inform all of the other users that
their private key is no longer valid. The subsystem thus includes ways
to query for the validity of keys and to propagate revocation messages.

@menu
* Dissemination:: 
* Revocation Message Design Requirements:: 

@end menu

@node Dissemination,Revocation Message Design Requirements,,REVOCATION — Ego key revocation
@anchor{users/subsystems dissemination}@anchor{85}@anchor{users/subsystems id32}@anchor{86}
@subsubsection Dissemination


When a revocation is performed, the revocation is first of all
disseminated by flooding the overlay network. The goal is to reach every
peer, so that when a peer needs to check if a key has been revoked, this
will be purely a local operation where the peer looks at its local
revocation list. Flooding the network is also the most robust form of
key revocation --- an adversary would have to control a separator of the
overlay graph to restrict the propagation of the revocation message.
Flooding is also very easy to implement --- peers that receive a
revocation message for a key that they have never seen before simply
pass the message to all of their neighbours.

Flooding can only distribute the revocation message to peers that are
online. In order to notify peers that join the network later, the
revocation service performs efficient set reconciliation over the sets
of known revocation messages whenever two peers (that both support
REVOCATION dissemination) connect. The SET service is used to perform
this operation efficiently.

@node Revocation Message Design Requirements,,Dissemination,REVOCATION — Ego key revocation
@anchor{users/subsystems id33}@anchor{87}@anchor{users/subsystems revocation-message-design-requirements}@anchor{88}
@subsubsection Revocation Message Design Requirements


However, flooding is also quite costly, creating O(|E|) messages on a
network with |E| edges. Thus, revocation messages are required to
contain a proof-of-work, the result of an expensive computation (which,
however, is cheap to verify). Only peers that have expended the CPU time
necessary to provide this proof will be able to flood the network with
the revocation message. This ensures that an attacker cannot simply
flood the network with millions of revocation messages. The
proof-of-work required by GNUnet is set to take days on a typical PC to
compute; if the ability to quickly revoke a key is needed, users have
the option to pre-compute revocation messages to store off-line and use
instantly after their key has expired.

Revocation messages must also be signed by the private key that is being
revoked. Thus, they can only be created while the private key is in the
possession of the respective user. This is another reason to create a
revocation message ahead of time and store it in a secure location.

@geindex subsystems; Random peer sampling
@geindex RPS; Random peer sampling

@node RPS — Random peer sampling,STATISTICS — Runtime statistics publication,REVOCATION — Ego key revocation,Subsystems
@anchor{users/subsystems rps-random-peer-sampling}@anchor{89}@anchor{users/subsystems rps-subsystem}@anchor{8a}
@subsection RPS — Random peer sampling


In literature, Random Peer Sampling (RPS) refers to the problem of
reliably @footnote{
"Reliable" in this context means having no bias, neither spatial,
nor temporal, nor through malicious activity.
} drawing random samples from an unstructured p2p network.

Doing so in a reliable manner is not only hard because of inherent
problems but also because of possible malicious peers that could try to
bias the selection.

It is useful for all kind of gossip protocols that require the selection
of random peers in the whole network like gathering statistics,
spreading and aggregating information in the network, load balancing and
overlay topology management.

The approach chosen in the RPS service implementation in GNUnet follows
the Brahms@footnote{https://bib.gnunet.org/full/date.html#2009_5f0} design.

The current state is "work in progress". There are a lot of things
that need to be done, primarily finishing the experimental evaluation
and a re-design of the API.

The abstract idea is to subscribe to connect to/start the RPS service
and request random peers that will be returned when they represent a
random selection from the whole network with high probability.

An additional feature to the original Brahms-design is the selection of
sub-groups: The GNUnet implementation of RPS enables clients to ask for
random peers from a group that is defined by a common shared secret.
(The secret could of course also be public, depending on the use-case.)

Another addition to the original protocol was made: The sampler
mechanism that was introduced in Brahms was slightly adapted and used to
actually sample the peers and returned to the client. This is necessary
as the original design only keeps peers connected to random other peers
in the network. In order to return random peers to client requests
independently random, they cannot be drawn from the connected peers. The
adapted sampler makes sure that each request for random peers is
independent from the others.

@menu
* Brahms:: 

@end menu

@node Brahms,,,RPS — Random peer sampling
@anchor{users/subsystems brahms}@anchor{8b}@anchor{users/subsystems id35}@anchor{8c}
@subsubsection Brahms


The high-level concept of Brahms is two-fold: Combining push-pull gossip
with locally fixing a assumed bias using cryptographic min-wise
permutations. The central data structure is the view - a peer's current
local sample. This view is used to select peers to push to and pull
from. This simple mechanism can be biased easily. For this reason Brahms
'fixes' the bias by using the so-called sampler. A data structure that
takes a list of elements as input and outputs a random one of them
independently of the frequency in the input set. Both an element that
was put into the sampler a single time and an element that was put into
it a million times have the same probability of being the output. This
is achieved with exploiting min-wise independent permutations. In the
RPS service we use HMACs: On the initialisation of a sampler element, a
key is chosen at random. On each input the HMAC with the random key is
computed. The sampler element keeps the element with the minimal HMAC.

In order to fix the bias in the view, a fraction of the elements in the
view are sampled through the sampler from the random stream of peer IDs.

According to the theoretical analysis of Bortnikov et al. this suffices
to keep the network connected and having random peers in the view.

@geindex STATISTICS; subsystem

@node STATISTICS — Runtime statistics publication,TRANSPORT-NG — Next-generation transport management,RPS — Random peer sampling,Subsystems
@anchor{users/subsystems statistics-runtime-statistics-publication}@anchor{8d}@anchor{users/subsystems statistics-subsystem}@anchor{8e}
@subsection STATISTICS — Runtime statistics publication


In GNUnet, the STATISTICS subsystem offers a central place for all
subsystems to publish unsigned 64-bit integer run-time statistics.
Keeping this information centrally means that there is a unified way for
the user to obtain data on all subsystems, and individual subsystems do
not have to always include a custom data export method for performance
metrics and other statistics. For example, the TRANSPORT system uses
STATISTICS to update information about the number of directly connected
peers and the bandwidth that has been consumed by the various plugins.
This information is valuable for diagnosing connectivity and performance
issues.

Following the GNUnet service architecture, the STATISTICS subsystem is
divided into an API which is exposed through the header
`gnunet_statistics_service.h' and the STATISTICS service
`gnunet-service-statistics'. The `gnunet-statistics' command-line
tool can be used to obtain (and change) information about the values
stored by the STATISTICS service. The STATISTICS service does not
communicate with other peers.

Data is stored in the STATISTICS service in the form of tuples
`(subsystem, name, value, persistence)'. The subsystem determines to
which other GNUnet's subsystem the data belongs. name is the name
through which value is associated. It uniquely identifies the record
from among other records belonging to the same subsystem. In some parts
of the code, the pair `(subsystem, name)' is called a `statistic' as
it identifies the values stored in the STATISTCS service.The persistence
flag determines if the record has to be preserved across service
restarts. A record is said to be persistent if this flag is set for it;
if not, the record is treated as a non-persistent record and it is lost
after service restart. Persistent records are written to and read from
the file `statistics.data' before shutdown and upon startup. The file
is located in the HOME directory of the peer.

An anomaly of the STATISTICS service is that it does not terminate
immediately upon receiving a shutdown signal if it has any clients
connected to it. It waits for all the clients that are not monitors to
close their connections before terminating itself. This is to prevent
the loss of data during peer shutdown — delaying the STATISTICS
service shutdown helps other services to store important data to
STATISTICS during shutdown.

@geindex TRANSPORT Next Generation; subsystem

@node TRANSPORT-NG — Next-generation transport management,TRANSPORT — Overlay transport management,STATISTICS — Runtime statistics publication,Subsystems
@anchor{users/subsystems transport-002dng-subsystem}@anchor{8f}@anchor{users/subsystems transport-ng-next-generation-transport-management}@anchor{90}
@subsection TRANSPORT-NG — Next-generation transport management


The current GNUnet TRANSPORT architecture is rooted in the GNUnet 0.4
design of using plugins for the actual transmission operations and the
ATS subsystem to select a plugin and allocate bandwidth. The following
key issues have been identified with this design:


@itemize -

@item 
Bugs in one plugin can affect the TRANSPORT service and other
plugins. There is at least one open bug that affects sockets, where
the origin is difficult to pinpoint due to the large code base.

@item 
Relevant operating system default configurations often impose a limit
of 1024 file descriptors per process. Thus, one plugin may impact
other plugin's connectivity choices.

@item 
Plugins are required to offer bi-directional connectivity. However,
firewalls (incl. NAT boxes) and physical environments sometimes only
allow uni-directional connectivity, which then currently cannot be
utilized at all.

@item 
Distance vector routing was implemented in 209 but shortly afterwards
broken and due to the complexity of implementing it as a plugin and
dealing with the resource allocation consequences was never useful.

@item 
Most existing plugins communicate completely using cleartext,
exposing metad data (message size) and making it easy to fingerprint
and possibly block GNUnet traffic.

@item 
Various NAT traversal methods are not supported.

@item 
The service logic is cluttered with "manipulation" support code for
TESTBED to enable faking network characteristics like lossy
connections or firewewalls.

@item 
Bandwidth allocation is done in ATS, requiring the duplication of
state and resulting in much delayed allocation decisions. As a
result, often available bandwidth goes unused. Users are expected to
manually configure bandwidth limits, instead of TRANSPORT using
congestion control to adapt automatically.

@item 
TRANSPORT is difficult to test and has bad test coverage.

@item 
HELLOs include an absolute expiration time. Nodes with unsynchronized
clocks cannot connect.

@item 
Displaying the contents of a HELLO requires the respective plugin as
the plugin-specific data is encoded in binary. This also complicates
logging.
@end itemize

@menu
* Design goals of TNG:: 
* HELLO-NG:: 
* Priorities and preferences:: 
* Communicators:: 

@end menu

@node Design goals of TNG,HELLO-NG,,TRANSPORT-NG — Next-generation transport management
@anchor{users/subsystems design-goals-of-tng}@anchor{91}@anchor{users/subsystems id37}@anchor{92}
@subsubsection Design goals of TNG


In order to address the above issues, we want to:


@itemize -

@item 
Move plugins into separate processes which we shall call
`communicators'. Communicators connect as clients to the transport
service.

@item 
TRANSPORT should be able to utilize any number of communicators to the
same peer at the same time.

@item 
TRANSPORT should be responsible for fragmentation, retransmission,
flow- and congestion-control. Users should no longer have to
configure bandwidth limits: TRANSPORT should detect what is available
and use it.

@item 
Communicators should be allowed to be uni-directional and
unreliable. TRANSPORT shall create bi-directional channels from this
whenever possible.

@item 
DV should no longer be a plugin, but part of TRANSPORT.

@item 
TRANSPORT should provide communicators help communicating, for
example in the case of uni-directional communicators or the need for
out-of-band signalling for NAT traversal. We call this functionality
`backchannels'.

@item 
Transport manipulation should be signalled to CORE on a per-message
basis instead of an approximate bandwidth.

@item 
CORE should signal performance requirements (reliability, latency,
etc.) on a per-message basis to TRANSPORT. If possible, TRANSPORT
should consider those options when scheduling messages for
transmission.

@item 
HELLOs should be in a human-readable format with monotonic time
expirations.
@end itemize

The new architecture is planned as follows:

@image{gnunet-figures/tng,,,,png}

TRANSPORT's main objective is to establish bi-directional virtual links
using a variety of possibly uni-directional communicators. Links undergo
the following steps:


@enumerate 

@item 
Communicator informs TRANSPORT A that a queue (direct neighbour) is
available, or equivalently TRANSPORT A discovers a (DV) path to a
target B.

@item 
TRANSPORT A sends a challenge to the target peer, trying to confirm
that the peer can receive. FIXME: This is not implemented properly
for DV. Here we should really take a validated DVH and send a
challenge exactly down that path!

@item 
The other TRANSPORT, TRANSPORT B, receives the challenge, and sends
back a response, possibly using a dierent path. If TRANSPORT B does
not yet have a virtual link to A, it must try to establish a virtual
link.

@item 
Upon receiving the response, TRANSPORT A creates the virtual link. If
the response included a challenge, TRANSPORT A must respond to this
challenge as well, eectively re-creating the TCP 3-way handshake
(just with longer challenge values).
@end enumerate

@node HELLO-NG,Priorities and preferences,Design goals of TNG,TRANSPORT-NG — Next-generation transport management
@anchor{users/subsystems hello-002dng}@anchor{93}@anchor{users/subsystems hello-ng}@anchor{94}
@subsubsection HELLO-NG


HELLOs change in three ways. First of all, communicators encode the
respective addresses in a human-readable URL-like string. This way, we
do no longer require the communicator to print the contents of a HELLO.
Second, HELLOs no longer contain an expiration time, only a creation
time. The receiver must only compare the respective absolute values. So
given a HELLO from the same sender with a larger creation time, then the
old one is no longer valid. This also obsoletes the need for the
gnunet-hello binary to set HELLO expiration times to never. Third, a
peer no longer generates one big HELLO that always contains all of the
addresses. Instead, each address is signed individually and shared only
over the address scopes where it makes sense to share the address. In
particular, care should be taken to not share MACs across the Internet
and confine their use to the LAN. As each address is signed separately,
having multiple addresses valid at the same time (given the new creation
time expiration logic) requires that those addresses must have exactly
the same creation time. Whenever that monotonic time is increased, all
addresses must be re-signed and re-distributed.

@node Priorities and preferences,Communicators,HELLO-NG,TRANSPORT-NG — Next-generation transport management
@anchor{users/subsystems id38}@anchor{95}@anchor{users/subsystems priorities-and-preferences}@anchor{96}
@subsubsection Priorities and preferences


In the new design, TRANSPORT adopts a feature (which was previously
already available in CORE) of the MQ API to allow applications to
specify priorities and preferences per message (or rather, per MQ
envelope). The (updated) MQ API allows applications to specify one of
four priority levels as well as desired preferences for transmission by
setting options on an envelope. These preferences currently are:


@itemize -

@item 
GNUNET_MQ_PREF_UNRELIABLE: Disables TRANSPORT waiting for ACKS on
unreliable channels like UDP. Now it is fire and forget. These
messages then cannot be used for RTT estimates either.

@item 
GNUNET_MQ_PREF_LOW_LATENCY: Directs TRANSPORT to select the
lowest-latency transmission choices possible.

@item 
GNUNET_MQ_PREF_CORK_ALLOWED: Allows TRANSPORT to delay transmission
to group the message with other messages into a larger batch to
reduce the number of packets sent.

@item 
GNUNET_MQ_PREF_GOODPUT: Directs TRANSPORT to select the highest
goodput channel available.

@item 
GNUNET_MQ_PREF_OUT_OF_ORDER: Allows TRANSPORT to reorder the messages
as it sees fit, otherwise TRANSPORT should attempt to preserve
transmission order.
@end itemize

Each MQ envelope is always able to store those options (and the
priority), and in the future this uniform API will be used by TRANSPORT,
CORE, CADET and possibly other subsystems that send messages (like
LAKE). When CORE sets preferences and priorities, it is supposed to
respect the preferences and priorities it is given from higher layers.
Similarly, CADET also simply passes on the preferences and priorities of
the layer above CADET. When a layer combines multiple smaller messages
into one larger transmission, the @code{GNUNET_MQ_env_combine_options()}
should be used to calculate options for the combined message. We note
that the exact semantics of the options may differ by layer. For
example, CADET will always strictly implement reliable and in-order
delivery of messages, while the same options are only advisory for
TRANSPORT and CORE: they should try (using ACKs on unreliable
communicators, not changing the message order themselves), but if
messages are lost anyway (e.g. because a TCP is dropped in the middle),
or if messages are reordered (e.g. because they took different paths
over the network and arrived in a different order) TRANSPORT and CORE do
not have to correct this. Whether a preference is strict or loose is
thus dened by the respective layer.

@node Communicators,,Priorities and preferences,TRANSPORT-NG — Next-generation transport management
@anchor{users/subsystems communicators}@anchor{97}@anchor{users/subsystems id39}@anchor{98}
@subsubsection Communicators


The API for communicators is defined in
@code{gnunet_transport_communication_service.h}. Each communicator must
specify its (global) communication characteristics, which for now only
say whether the communication is reliable (e.g. TCP, HTTPS) or
unreliable (e.g. UDP, WLAN). Each communicator must specify a unique
address prex, or NULL if the communicator cannot establish outgoing
connections (for example because it is only acting as a TCP server). A
communicator must tell TRANSPORT which addresses it is reachable under.
Addresses may be added or removed at any time. A communicator may have
zero addresses (transmission only). Addresses do not have to match the
address prefix.

TRANSPORT may ask a communicator to try to connect to another address.
TRANSPORT will only ask for connections where the address matches the
communicator's address prefix that was provided when the connection was
established. Communicators should then attempt to establish a
connection.
It is under the discretion of the communicator whether to honor this request.
Reasons for not honoring such a request may be that an existing connection exists
or resource limitations.
No response is provided to TRANSPORT service on failure.
The TRANSPORT service has to ask the communicator explicitly to retry.

If a communicator succeeds in establishing an outgoing connection for
transmission, or if a communicator receives an incoming bi-directional
connection, the communicator must inform the TRANSPORT service that a
message queue (MQ) for transmission is now available.
For that MQ, the communicator must provide the peer identity claimed by the other end.
It must also provide a human-readable address (for debugging) and a maximum transfer unit
(MTU). A MTU of zero means sending is not supported, SIZE_MAX should be
used for no MTU. The communicator should also tell TRANSPORT what
network type is used for the queue. The communicator may tell TRANSPORT
anytime that the queue was deleted and is no longer available.

The communicator API also provides for flow control. First,
communicators exhibit back-pressure on TRANSPORT: the number of messages
TRANSPORT may add to a queue for transmission will be limited. So by not
draining the transmission queue, back-pressure is provided to TRANSPORT.
In the other direction, communicators may allow TRANSPORT to give
back-pressure towards the communicator by providing a non-NULL
@code{GNUNET_TRANSPORT_MessageCompletedCallback} argument to the
@code{GNUNET_TRANSPORT_communicator_receive} function. In this case,
TRANSPORT will only invoke this function once it has processed the
message and is ready to receive more. Communicators should then limit
how much traffic they receive based on this backpressure. Note that
communicators do not have to provide a
@code{GNUNET_TRANSPORT_MessageCompletedCallback}; for example, UDP cannot
support back-pressure due to the nature of the UDP protocol. In this
case, TRANSPORT will implement its own TRANSPORT-to-TRANSPORT flow
control to reduce the sender's data rate to acceptable levels.

TRANSPORT may notify a communicator about backchannel messages TRANSPORT
received from other peers for this communicator. Similarly,
communicators can ask TRANSPORT to try to send a backchannel message to
other communicators of other peers. The semantics of the backchannel
message are up to the communicators which use them. TRANSPORT may fail
transmitting backchannel messages, and TRANSPORT will not attempt to
retransmit them.

@menu
* UDP communicator:: 
* TCP communicator:: 
* QUIC communicator:: 

@end menu

@node UDP communicator,TCP communicator,,Communicators
@anchor{users/subsystems udp-communicator}@anchor{99}
@subsubsection UDP communicator


The UDP communicator implements a basic encryption layer to protect from
metadata leakage.
The layer tries to establish a shared secret using an Elliptic-Curve Diffie-Hellman
key exchange in which the initiator of a packet creates an ephemeral key pair
to encrypt a message for the target peer identity.
The communicator always offers this kind of transmission queue to a (reachable)
peer in which messages are encrypted with dedicated keys.
The performance of this queue is not suitable for high volume data transfer.

If the UDP connection is bi-directional, or the TRANSPORT is able to offer a
backchannel connection, the resulting key can be re-used if the recieving peer
is able to ACK the reception.
This will cause the communicator to offer a new queue (with a higher priority
than the default queue) to TRANSPORT with a limited capacity.
The capacity is increased whenever the communicator receives an ACK for a
transmission.
This queue is suitable for high-volume data transfer and TRANSPORT will likely
prioritize this queue (if available).

Communicators that try to establish a connection to a target peer authenticate
their peer ID (public key) in the first packets by signing a monotonic time
stamp, its peer ID, and the target peerID and send this data as well as the signature
in one of the first packets.
Receivers should keep track (persist) of the monotonic time stamps for each
peer ID to reject possible replay attacks.

FIXME: Handshake wire format? KX, Flow.

@node TCP communicator,QUIC communicator,UDP communicator,Communicators
@anchor{users/subsystems tcp-communicator}@anchor{9a}
@subsubsection TCP communicator


FIXME: Handshake wire format? KX, Flow.

@node QUIC communicator,,TCP communicator,Communicators
@anchor{users/subsystems quic-communicator}@anchor{9b}
@subsubsection QUIC communicator


The QUIC communicator runs over a bi-directional UDP connection.
TLS layer with self-signed certificates (binding/signed with peer ID?).
Single, bi-directional stream?
FIXME: Handshake wire format? KX, Flow.

@geindex TRANSPORT; subsystem

@node TRANSPORT — Overlay transport management,SET — Peer to peer set operations Deprecated,TRANSPORT-NG — Next-generation transport management,Subsystems
@anchor{users/subsystems transport-overlay-transport-management}@anchor{9c}@anchor{users/subsystems transport-subsystem}@anchor{9d}
@subsection TRANSPORT — Overlay transport management


This chapter documents how the GNUnet transport subsystem works. The
GNUnet transport subsystem consists of three main components: the
transport API (the interface used by the rest of the system to access
the transport service), the transport service itself (most of the
interesting functions, such as choosing transports, happens here) and
the transport plugins. A transport plugin is a concrete implementation
for how two GNUnet peers communicate; many plugins exist, for example
for communication via TCP, UDP, HTTP, HTTPS and others. Finally, the
transport subsystem uses supporting code, especially the NAT/UPnP
library to help with tasks such as NAT traversal.

Key tasks of the transport service include:


@itemize -

@item 
Create our HELLO message, notify clients and neighbours if our HELLO
changes (using NAT library as necessary)

@item 
Validate HELLOs from other peers (send PING), allow other peers to
validate our HELLO's addresses (send PONG)

@item 
Upon request, establish connections to other peers (using address
selection from ATS subsystem) and maintain them (again using PINGs
and PONGs) as long as desired

@item 
Accept incoming connections, give ATS service the opportunity to
switch communication channels

@item 
Notify clients about peers that have connected to us or that have
been disconnected from us

@item 
If a (stateful) connection goes down unexpectedly (without explicit
DISCONNECT), quickly attempt to recover (without notifying clients)
but do notify clients quickly if reconnecting fails

@item 
Send (payload) messages arriving from clients to other peers via
transport plugins and receive messages from other peers, forwarding
those to clients

@item 
Enforce inbound traffic limits (using flow-control if it is
applicable); outbound traffic limits are enforced by CORE, not by us
(!)

@item 
Enforce restrictions on P2P connection as specified by the blacklist
configuration and blacklisting clients
@end itemize

Note that the term "clients" in the list above really refers to the
GNUnet-CORE service, as CORE is typically the only client of the
transport service.

@geindex subsystem; SET

@node SET — Peer to peer set operations Deprecated,SETI — Peer to peer set intersections,TRANSPORT — Overlay transport management,Subsystems
@anchor{users/subsystems set-peer-to-peer-set-operations-deprecated}@anchor{9e}@anchor{users/subsystems set-subsystem}@anchor{9f}
@subsection SET — Peer to peer set operations (Deprecated)


@cartouche
@quotation Note 
The SET subsystem is in process of being replaced by the SETU and SETI
subsystems, which provide basically the same functionality, just using
two different subsystems. SETI and SETU should be used for new code.
@end quotation
@end cartouche

The SET service implements efficient set operations between two peers
over a CADET tunnel. Currently, set union and set intersection are the
only supported operations. Elements of a set consist of an `element
type' and arbitrary binary `data'. The size of an element's data is
limited to around 62 KB.

@menu
* Local Sets:: 
* Set Modifications:: 
* Set Operations:: 
* Result Elements:: 

@end menu

@node Local Sets,Set Modifications,,SET — Peer to peer set operations Deprecated
@anchor{users/subsystems id40}@anchor{a0}@anchor{users/subsystems local-sets}@anchor{a1}
@subsubsection Local Sets


Sets created by a local client can be modified and reused for multiple
operations. As each set operation requires potentially expensive special
auxiliary data to be computed for each element of a set, a set can only
participate in one type of set operation (either union or intersection).
The type of a set is determined upon its creation. If a the elements of
a set are needed for an operation of a different type, all of the set's
element must be copied to a new set of appropriate type.

@node Set Modifications,Set Operations,Local Sets,SET — Peer to peer set operations Deprecated
@anchor{users/subsystems id41}@anchor{a2}@anchor{users/subsystems set-modifications}@anchor{a3}
@subsubsection Set Modifications


Even when set operations are active, one can add to and remove elements
from a set. However, these changes will only be visible to operations
that have been created after the changes have taken place. That is,
every set operation only sees a snapshot of the set from the time the
operation was started. This mechanism is `not' implemented by copying
the whole set, but by attaching `generation information' to each element
and operation.

@node Set Operations,Result Elements,Set Modifications,SET — Peer to peer set operations Deprecated
@anchor{users/subsystems id42}@anchor{a4}@anchor{users/subsystems set-operations}@anchor{a5}
@subsubsection Set Operations


Set operations can be started in two ways: Either by accepting an
operation request from a remote peer, or by requesting a set operation
from a remote peer. Set operations are uniquely identified by the
involved `peers', an `application id' and the `operation type'.

The client is notified of incoming set operations by `set listeners'. A
set listener listens for incoming operations of a specific operation
type and application id. Once notified of an incoming set request, the
client can accept the set request (providing a local set for the
operation) or reject it.

@node Result Elements,,Set Operations,SET — Peer to peer set operations Deprecated
@anchor{users/subsystems id43}@anchor{a6}@anchor{users/subsystems result-elements}@anchor{a7}
@subsubsection Result Elements


The SET service has three `result modes' that determine how an
operation's result set is delivered to the client:


@itemize -

@item 
`Full Result Set.' All elements of set resulting from the set
operation are returned to the client.

@item 
`Added Elements.' Only elements that result from the operation and
are not already in the local peer's set are returned. Note that for
some operations (like set intersection) this result mode will never
return any elements. This can be useful if only the remove peer is
actually interested in the result of the set operation.

@item 
`Removed Elements.' Only elements that are in the local peer's
initial set but not in the operation's result set are returned. Note
that for some operations (like set union) this result mode will never
return any elements. This can be useful if only the remove peer is
actually interested in the result of the set operation.
@end itemize

@geindex subsystem; SETI

@node SETI — Peer to peer set intersections,SETU — Peer to peer set unions,SET — Peer to peer set operations Deprecated,Subsystems
@anchor{users/subsystems seti-peer-to-peer-set-intersections}@anchor{a8}@anchor{users/subsystems seti-subsystem}@anchor{a9}
@subsection SETI — Peer to peer set intersections


The SETI service implements efficient set intersection between two peers
over a CADET tunnel. Elements of a set consist of an `element type' and
arbitrary binary `data'. The size of an element's data is limited to
around 62 KB.

@menu
* Intersection Sets:: 
* Set Intersection Modifications:: 
* Set Intersection Operations:: 
* Intersection Result Elements:: 

@end menu

@node Intersection Sets,Set Intersection Modifications,,SETI — Peer to peer set intersections
@anchor{users/subsystems id44}@anchor{aa}@anchor{users/subsystems intersection-sets}@anchor{ab}
@subsubsection Intersection Sets


Sets created by a local client can be modified (by adding additional
elements) and reused for multiple operations. If elements are to be
removed, a fresh set must be created by the client.

@node Set Intersection Modifications,Set Intersection Operations,Intersection Sets,SETI — Peer to peer set intersections
@anchor{users/subsystems id45}@anchor{ac}@anchor{users/subsystems set-intersection-modifications}@anchor{ad}
@subsubsection Set Intersection Modifications


Even when set operations are active, one can add elements to a set.
However, these changes will only be visible to operations that have been
created after the changes have taken place. That is, every set operation
only sees a snapshot of the set from the time the operation was started.
This mechanism is `not' implemented by copying the whole set, but by
attaching `generation information' to each element and operation.

@node Set Intersection Operations,Intersection Result Elements,Set Intersection Modifications,SETI — Peer to peer set intersections
@anchor{users/subsystems id46}@anchor{ae}@anchor{users/subsystems set-intersection-operations}@anchor{af}
@subsubsection Set Intersection Operations


Set operations can be started in two ways: Either by accepting an
operation request from a remote peer, or by requesting a set operation
from a remote peer. Set operations are uniquely identified by the
involved `peers', an `application id' and the `operation type'.

The client is notified of incoming set operations by `set listeners'. A
set listener listens for incoming operations of a specific operation
type and application id. Once notified of an incoming set request, the
client can accept the set request (providing a local set for the
operation) or reject it.

@node Intersection Result Elements,,Set Intersection Operations,SETI — Peer to peer set intersections
@anchor{users/subsystems id47}@anchor{b0}@anchor{users/subsystems intersection-result-elements}@anchor{b1}
@subsubsection Intersection Result Elements


The SET service has two `result modes' that determine how an operation's
result set is delivered to the client:


@itemize -

@item 
`Return intersection.' All elements of set resulting from the set
intersection are returned to the client.

@item 
`Removed Elements.' Only elements that are in the local peer's
initial set but not in the intersection are returned.
@end itemize

@geindex SETU; subsystem

@node SETU — Peer to peer set unions,,SETI — Peer to peer set intersections,Subsystems
@anchor{users/subsystems setu-peer-to-peer-set-unions}@anchor{b2}@anchor{users/subsystems setu-subsystem}@anchor{b3}
@subsection SETU — Peer to peer set unions


The SETU service implements efficient set union operations between two
peers over a CADET tunnel. Elements of a set consist of an `element
type' and arbitrary binary `data'. The size of an element's data is
limited to around 62 KB.

@menu
* Union Sets:: 
* Set Union Modifications:: 
* Set Union Operations:: 
* Union Result Elements:: 

@end menu

@node Union Sets,Set Union Modifications,,SETU — Peer to peer set unions
@anchor{users/subsystems id48}@anchor{b4}@anchor{users/subsystems union-sets}@anchor{b5}
@subsubsection Union Sets


Sets created by a local client can be modified (by adding additional
elements) and reused for multiple operations. If elements are to be
removed, a fresh set must be created by the client.

@node Set Union Modifications,Set Union Operations,Union Sets,SETU — Peer to peer set unions
@anchor{users/subsystems id49}@anchor{b6}@anchor{users/subsystems set-union-modifications}@anchor{b7}
@subsubsection Set Union Modifications


Even when set operations are active, one can add elements to a set.
However, these changes will only be visible to operations that have been
created after the changes have taken place. That is, every set operation
only sees a snapshot of the set from the time the operation was started.
This mechanism is `not' implemented by copying the whole set, but by
attaching `generation information' to each element and operation.

@node Set Union Operations,Union Result Elements,Set Union Modifications,SETU — Peer to peer set unions
@anchor{users/subsystems id50}@anchor{b8}@anchor{users/subsystems set-union-operations}@anchor{b9}
@subsubsection Set Union Operations


Set operations can be started in two ways: Either by accepting an
operation request from a remote peer, or by requesting a set operation
from a remote peer. Set operations are uniquely identified by the
involved `peers', an `application id' and the `operation type'.

The client is notified of incoming set operations by `set listeners'. A
set listener listens for incoming operations of a specific operation
type and application id. Once notified of an incoming set request, the
client can accept the set request (providing a local set for the
operation) or reject it.

@node Union Result Elements,,Set Union Operations,SETU — Peer to peer set unions
@anchor{users/subsystems id51}@anchor{ba}@anchor{users/subsystems union-result-elements}@anchor{bb}
@subsubsection Union Result Elements


The SET service has three `result modes' that determine how an
operation's result set is delivered to the client:


@itemize -

@item 
`Locally added Elements.' Elements that are in the union but not
already in the local peer's set are returned.

@item 
`Remote added Elements.' Additionally, notify the client if the
remote peer lacked some elements and thus also return to the local
client those elements that we are sending to the remote peer to be
added to its union. Obtaining these elements requires setting the
@code{GNUNET_SETU_OPTION_SYMMETRIC} option.
@end itemize

@node The GNU Name System,re claimID,Subsystems,User Manual
@anchor{users/gns doc}@anchor{bc}@anchor{users/gns the-gnu-name-system}@anchor{bd}
@section The GNU Name System


The GNU Name System (GNS) is secure and decentralized naming system. It
allows its users to register names as top-level domains (TLDs) and
resolve other namespaces within their TLDs.

GNS is designed to provide:


@itemize -

@item 
Censorship resistance

@item 
Query privacy

@item 
Secure name resolution

@item 
Compatibility with DNS
@end itemize

Unlike DNS, GNS does not rely on central root zones or authorities.
Instead any user administers their own root and can can create arbitrary
name value mappings. Furthermore users can delegate resolution to other
users’ zones just like DNS NS records do. Zones are uniquely identified
via public keys and resource records are signed using the corresponding
public key. Delegation to another user’s zone is done using special delegation
records and petnames. A petname is a name that can be freely chosen by
the user. This results in non-unique name-value mappings as www.bob to
one user might be www.friend for someone else.

For a complete specification of the protocol, we refer to LSD0001@footnote{https://lsd.gnunet.org/lsd0001}.

@menu
* Start Zones:: 
* Local zone maintenance:: 
* Application integration:: 
* Zone dissemination:: 
* Resource Records:: 
* DNS Migration:: 

@end menu

@node Start Zones,Local zone maintenance,,The GNU Name System
@anchor{users/gns start-zones}@anchor{be}
@subsection Start Zones


In GNS you are the master of your own Root.
This means that you are able (and encouraged!) to manage your own `Start Zones'.
A `Start Zone' is a mapping from a name suffix (e.g. @code{.gnunet.org}) to
a GNS zone key.
A user's configuration of `Start Zones' is the equivalent of DNS's Root Zone@footnote{https://en.wikipedia.org/wiki/DNS_root_zone}.
In other words, in GNS you can be your very own ICANN@footnote{https://en.wikipedia.org/wiki/ICANN}.

There are three types of `Start Zones':

@quotation


@enumerate 

@item 
Mappings to `remote' GNS zones.

@item 
Mappings to your own `local' GNS zones.

@item 
Explicit zone Top-Level-Domains (zTLDs)
@end enumerate
@end quotation

@menu
* Remote:: 
* Local:: 
* Zone Top-Level-Domains:: 

@end menu

@node Remote,Local,,Start Zones
@anchor{users/gns remote}@anchor{bf}
@subsubsection Remote


Your GNUnet installation ships with a default configuration of remote
`Start Zones'.
The first is @code{.gnunet.org}, which points to the authoritative zone of the
GNUnet project.
This `Start Zone' allows you to resolve names ending with @code{.gnunet.org}.
It can be used to resolve, for example, @code{www.gnunet.org}.

You can try it out for yourself (if you are connected to the peer-to-peer network):

@example
$ gnunet-gns --lookup=www.gnunet.org
@end example

Another `Start Zone' configured by default is @code{.pin.gns.alt}.
It points to a special zone also managed by the GNUnet project.
Users may register subdomains on a first-come first-served-basis at @indicateurl{https://fcfs.gnunet.org}.

Use @code{gnunet-config -s gns} to view the GNS configuration, including all
configured external zones that are operated by other users.
The respective configuration entry names start with a @code{.}, e.g. @code{.pin.gns.alt}.

You can change this default configuration at any time and add `Start Zone' to the
respective zones of your friends! For this, simply obtain the respective
public key (you will learn how below) and extend the configuration:

@example
$ gnunet-config -s gns -o .myfriend -V PUBLIC_KEY
@end example

@node Local,Zone Top-Level-Domains,Remote,Start Zones
@anchor{users/gns local}@anchor{c0}
@subsubsection Local


In order to create a local GNS zone you have to create a so-called @code{ego} in GNUnet.
Egos may correspond to pseudonyms or real-world identities.
It is recommended to use separate egos for separate activities.
All egos can potentially serve as a GNS zone, but you do not have to use
all egos as GNS zones.
All your local zones are used as `Start Zones'.

Technically, an ego is first of all a public-private key pair, and thus
egos also always correspond to a GNS zone. Egos are managed through the
IDENTITY service and its tooling.
The IDENTITY service is used to create and stores private keys with associated
human-readable nicknames.
Those mappings serve as local `Start Zones'.

For example, you can create a new ego (or zone) with name `myzone' using the
@code{gnunet-identity} tool using:

@example
$ gnunet-identity --create="myzone"
@end example

Henceforth, on your system you control the TLD @code{myzone}.
You are able to resolve GNS names ending with @code{.myzone}.
However, the zone is still empty.
We will look at local zone maintenance in the next section.
Note that by default, this will create ECDSA key pairs.
GNS also supports the use of EdDSA keys.
In order to create a EdDSA zone, execute:

@example
$ gnunet-identity --create="myedzone" --eddsa
@end example

All of your zones can be listed (displayed) using the gnunet-identity
command line tool as well:

@example
$ gnunet-identity --display
@end example

@node Zone Top-Level-Domains,,Local,Start Zones
@anchor{users/gns zone-top-level-domains}@anchor{c1}
@subsubsection Zone Top-Level-Domains


If you know the public key of a GNS zone, you can resolve records in that zone
without a `Start Zone' mapping.
To do so, you must provide the string-representation of the zone key
as Top-Level Domain. Consider the zone key of @code{.gnunet.org} in the default
configuration. In order to resolve the same record as above using its respective
zTLD, you can do so as follows:

@example
$ gnunet-gns --lookup=www.000G0047M3HN599H57MPXZK4VB59SWK4M9NRD68E1JQFY3RWAHDMKAPN30
@end example

The use of zTLDs is mostly useful in the absence of a `Start Zone' configuration
for that zone or when querying names programmatically.

@node Local zone maintenance,Application integration,Start Zones,The GNU Name System
@anchor{users/gns local-zone-maintenance}@anchor{c2}
@subsection Local zone maintenance


So you are ready to maintain your own GNS zone? Great.
First you should review and optionally configure your NAMESTORE database which
will hold your zone's resource records.
You have the choice between @code{sqlite} and @code{postgres} as databases.
Most users will be served well with the @code{sqlite} backend and with its default
configuration.
To change the database backend of your NAMESTORE to @code{postgres} execute:

@example
$ gnunet-config -s namestore -o DATABASE -V postgres
@end example

You can now configure the @code{postgres} database backend to find your database.
See the @code{namestore-postgres} configuration section on possible options:

@example
$ gnunet-config -s namestore-postgres
@end example

If you ever need to reset your zone database, you can use @code{gnunet-namestore-dbtool}
with the @code{--reset} option.
`DANGER: This command will delete ALL of your records in all zones!'

@menu
* Orphaning:: 
* Records:: 
* Revocation:: 
* Backup:: 
* Large Zones:: 

@end menu

@node Orphaning,Records,,Local zone maintenance
@anchor{users/gns orphaning}@anchor{c3}
@subsubsection Orphaning


It can happen that you accidentally or semi-accidentally delete your ego, e.g.
using @code{gnunet-identity --delete=myzone}.
In this case the records stored in the zone are still in the database but the
missing ego causes them to be `orphaned'.

Orphaned records are still published in the DHT but access to them is limited.
For example, managing records through the NAMESTORE API is not longer possible.

You can check for orphaned records using:

@example
$ gnunet-namestore --list-orphans
@end example

This will output any orphaned records in the format

@example
<$LABEL>.<$PRIVATE_KEY>:
$RECORD_DATA
@end example

In order to recover (and un-orphan) the records, you can re-add the ego by
executing

@example
$ gnunet-identity --create=<myzone> --privkey=$PRIVATE_KEY
@end example

If instead you should decide that you want to `purge' all orphaned records,
you can execute

@example
$ gnunet-namestore --purge-orphans
@end example

@node Records,Revocation,Orphaning,Local zone maintenance
@anchor{users/gns records}@anchor{c4}
@subsubsection Records


At this point you can add (or edit, or remove) records in your GNS zone using the
using the gnunet-namestore command-line tool.
Your records will be stored in a database
of the NAMESTORE service. Note that for multi-user setups, the NAMESTORE
database will include the combined records of all users.
However, users will not be able to see each other’s records if they are marked
as private.

To provide a short example for editing your own zone, suppose you have
your own web server with the IP @code{1.2.3.4}. Then you can put an A record (A
records in DNS are for IPv4 IP addresses) into your local zone @code{myzone}
using the command:

@example
$ gnunet-namestore --zone=myzone \
                   --add \
                   --name=www \
                   --type=A \
                   --value=1.2.3.4 \
                   --expiration=1d
@end example

Similar commands will work for other types of DNS and GNS records, the
syntax largely depending on the type of the record. Naturally, most
users may find editing the zones using the gnunet-namestore-gtk GUI to
be easier.

Note that by default, records are `private'. This means that they are not
published and cannot be resolved by other users that may have configured your
zone as one of their @code{Start Zones}.
In order to create a public record, provide the @code{--public} switch to the
@code{gnunet-namestore} command above.

You can now try to resolve your own GNS record. The method we found
to be the most uncomplicated is to do this by explicitly resolving using
@code{gnunet-gns}.
In the shell, type:

@example
$ gnunet-gns -u www.myzone
@end example

You should receive the record as configured above.
At this point, you can try addition additional records to @code{www} or add other
names to your zone.

Each zone in GNS has a public-private key. Usually, gnunet-namestore and
gnunet-setup will access your private key as necessary, so you do not
have to worry about those. What is important is your public key (or
rather, the hash of your public key), as you will likely want to give it
to others so that they can securely link to you.

A central operation in GNS is the ability to securely delegate to other
zones. Basically, by adding a delegation you make all of the names from
the other zone available to yourself. This section describes how to
create delegations.

Suppose you have a friend who you call ’bob’ who also uses GNS. You can
then delegate resolution of names to Bob’s zone by adding a PKEY record
to their local zone:

@example
$ gnunet-namestore -a -n bob --type PKEY -V $BOBKEY -e 1d -z myzone
@end example

Note that @code{$BOBKEY} in the command above must be replaced with the hash of
Bob’s public key (the output your friend obtained using the
gnunet-identity command from the previous section and told you, for
example by giving you a business card containing this information as a
QR code).

Assuming Bob has an @code{A} record for their website under the name of @code{www}
in his zone, you can then access Bob’s website under @code{www.bob.myzone} -
as well as any (public) GNS record that Bob has in their zone by
replacing www with the respective name of the record in Bob’s zone.

Furthermore, if Bob has themselves a (public) delegation to Carol’s zone
under `carol', you can access Carol’s records under
`NAME.carol.bob.myzone' (where `NAME' is the name of Carol’s record you
want to access).

You can find more detailed information in available record types in the
@ref{c5,,Resource Records} section.

@node Revocation,Backup,Records,Local zone maintenance
@anchor{users/gns revocation}@anchor{c6}
@subsubsection Revocation


Now, in the situation of an attacker gaining access to the private key
of one of your egos, the attacker can create records in the respective
GNS zone and publish them as if you published them. Anyone resolving
your domain will get these new records and when they verify they seem
authentic because the attacker has signed them with your key.

To address this potential security issue, you can pre-compute a
revocation certificate corresponding to your ego. This certificate, when
published on the P2P network, flags your private key as invalid, and all
further resolutions or other checks involving the key will fail.

A revocation certificate is thus a useful tool when things go out of
control, but at the same time it should be stored securely. Generation
of the revocation certificate for a zone can be done through
@code{gnunet-revocation}. For example, the following command (as unprivileged
user) generates a revocation file @code{revocation.dat} for the zone @code{myzone}:

@example
$ gnunet-revocation -f revocation.dat -R myzone
@end example

The above command only pre-computes a revocation certificate. It does
not revoke the given zone. Pre-computing a revocation certificate
involves computing a proof-of-work and hence may take up to 4 to 5 days
on a modern processor. Note that you can abort and resume the
calculation at any time. Also, even if you did not finish the
calculation, the resulting file will contain the signature, which is
sufficient to complete the revocation process even without access to the
private key. So instead of waiting for a few days, you can just abort
with @code{CTRL-C}, backup the revocation certificate and run the calculation
only if your key actually was compromised. This has the disadvantage of
revocation taking longer after the incident, but the advantage of saving
a significant amount of energy. So unless you believe that a key
compromise will need a rapid response, we urge you to wait with
generating the revocation certificate. Also, the calculation is
deliberately expensive, to deter people from doing this just for fun (as
the actual revocation operation is expensive for the network, not for
the peer performing the revocation).

To avoid TL;DR ones from accidentally revoking their zones, we are not
giving away the command, but it is uncomplicated: the actual revocation
is performed by using the @code{-p} option of @code{gnunet-revocation}.

@node Backup,Large Zones,Revocation,Local zone maintenance
@anchor{users/gns backup}@anchor{c7}
@subsubsection Backup


One should always backup their files, especially in these SSD days (our
team has suffered 3 SSD crashes over a span of 2 weeks). Backing up peer
identity and zones is achieved by copying the following files:

The peer identity file can be found in
@code{~/.local/share/gnunet/private_key.ecc}.

The private keys of your egos are stored in the directory
@code{~/.local/share/gnunet/identity/egos/}. They are stored in files whose
filenames correspond to the zones’ ego names. These are probably the
most important files you want to backup from a GNUnet installation.

Note: All these files contain cryptographic keys and they are stored
without any encryption. So it is advisable to backup encrypted copies of
them.

This does `NOT' backup your record data.
Your record data is stored in the respective database backend, and hence
you may want to backup the respective databases as well.

@node Large Zones,,Backup,Local zone maintenance
@anchor{users/gns large-zones}@anchor{c8}
@subsubsection Large Zones


Large zones in DNS such as Top-Level Domains can grow to millions of names.
For example, the @code{.fr} DNS zone contained over 3.9 million entries in 2022.
Such use cases require good performance and careful consideration of a couple
of choices in GNS and NAMESTORE.

The cryptographic operations of EdDSA are significantly faster than their
ECDSA counterparts.
Most users will be unable to observe this difference.
But, if you plan on managing large zones, you may want to consider using EdDSA
zones.

For large zones, using the PostgreSQL NAMESTORE database backend is a must.
Further, when importing large numbers of records at once, it is advised to make
use of the bulk import functionality of the NAMESTORE API.
There is a @ref{39,,C API for bulk imports}.
There is also a @ref{c9,,REST API endpoint} which calls this API.

Zones are published and signed by the ZONEMASTER service.
In order to speed up this process for large zones, you can edit the number of
spawned worker threads for the service:

@example
$ gnunet-config -s zonemaster -o WORKER_COUNT -V $COUNT
@end example

It is reasonable to choose your number of CPUs as @code{$COUNT}.

@node Application integration,Zone dissemination,Local zone maintenance,The GNU Name System
@anchor{users/gns application-integration}@anchor{ca}
@subsection Application integration


Once you have verified that resolution of names using @code{gnunet-gns} works,
we can start integrating GNS with applications.
We can distinguish between two types of applications:

@quotation


@enumerate 

@item 
`GNS-aware applications': Such applications know what GNS is and how it
can be used to resolve names. Examples are some of GNUnet's services such
as re:claimID@footnote{reclaimID-Identity-Provider} or Messenger@footnote{Using-the-GNUnet-Messenger}.

@item 
`GNS-unaware applications': Applications that implicitly assume the names
are resolved through DNS. Such applications use OS-specific APIs or query
configured DNS servers directly.
@end enumerate
@end quotation

In the following, we will discuss integration paths of GNS-unaware applications.

@menu
* Integration with Browsers (DNS2GNS service): Integration with Browsers DNS2GNS service. 
* Integration with Browsers (SOCKS proxy): Integration with Browsers SOCKS proxy. 
* NSS plugin:: 

@end menu

@node Integration with Browsers DNS2GNS service,Integration with Browsers SOCKS proxy,,Application integration
@anchor{users/gns integration-with-browsers-dns2gns-service}@anchor{cb}
@subsubsection Integration with Browsers (DNS2GNS service)


Most OSes allow you to either modify your @code{/etc/resolv.conf} directly
or through @code{resolvectl}. We are going to configure the dns2gns service
in order to translate DNS name queries by applications to GNS name
queries where applicable and else fall back to DNS.

Optionally, you may want to configure your dns2gns service to run on a
non-privileged port like 5353. But, in case you are going to edit
@code{/etc/resolv.conf} directly, the dns2gns service MUST run on port 53
as you cannot specify the port number. A @code{$FALLBACK_DNS} variable
should be a DNS server you trust such as your local router:

@example
$ gnunet-config -s dns2gns -o OPTIONS -V "-d $FALLBACK_DNS -p 5252"
$ gnunet-arm -i dns2gns
@end example

If you edit your resolv.conf directly, it should contain and entry like
this:

@example
nameserver 127.0.0.1
@end example

In any case, it is very likely that the method of modification of your
resolver is OS specific. Recently, the combination of NetworkManager and
systemd-resolved is becoming increasingly popular.

If you use resolvectl and systemd-resolved you can temporarily set the
nameserver like this:

@example
$ resolvectl $INTERFACE 127.0.0.1:5353
@end example

Where @code{$INTERFACE} is your network interface such as @code{eth0}.

In order to automatically set the DNS2GNS server if it is running
already you can use NetworkManager-dispatcher. First, enable it:

@example
$ sudo systemctl enable NetworkManager-dispatcher.service
$ sudo systemctl start NetworkManager-dispatcher.service
@end example

Then, create a script @code{/etc/NetworkManager/dispatch.h/10-dns2-gns.sh}:

@example
#!/bin/sh
interface=$1
status=$2

if [ "$interface" = "eth0" ]; then
  case $status in
    up)
      if nc -u -z 127.0.0.1 5353; then
      resolvectl dns $interface 127.0.0.1:5353
    fi
    ;;
    down)
    ;;
  esac
fi
@end example

Make sure the script is owned by root and executable:

@example
$ sudo root:root /etc/NetworkManager/dispatch.d/10-dns2gns.sh
$ sudo +x /etc/NetworkManager/dispatch.d/10-dns2gns.sh
@end example

You can test accessing this website using your browser or curl:

@example
$ curl www.gnunet.org
@end example

Note that `gnunet.org' is a domain that also exists in DNS and for which
the GNUnet project webservers can provide trusted TLS certificates. When
using non-DNS names with GNS or aliases, this may result in issues when
accessing HTTPS websites with browsers. In order learn how to provide
relief for this issue, read on.

@node Integration with Browsers SOCKS proxy,NSS plugin,Integration with Browsers DNS2GNS service,Application integration
@anchor{users/gns integration-with-browsers-socks-proxy}@anchor{cc}
@subsubsection Integration with Browsers (SOCKS proxy)


While we recommend integrating GNS using the DNS2GNS service or the
NSSwitch plugin, you can also integrate GNS directly with your browser
via the gnunet-gns-proxy. This method can have the advantage that the
proxy can validate TLS/X.509 records and thus strengthen web security;
however, the proxy is still a bit brittle, so expect subtle failures. We
have had reasonable success with Chromium, and various frustrations with
Firefox in this area recently.

The first step is to start the proxy. As the proxy is (usually) not
started by default, this is done as a unprivileged user using gnunet-arm
-i gns-proxy. Use gnunet-arm -I as a unprivileged user to check that the
proxy was actually started. (The most common error for why the proxy may
fail to start is that you did not run gnunet-gns-proxy-setup-ca during
installation.) The proxy is a SOCKS5 proxy running (by default) on port
7777. Thus, you need to now configure your browser to use this proxy.
With Chromium, you can do this by starting the browser as a unprivileged
user using chromium –proxy-server=*socks5://localhost:7777* For Firefox
(or Icecat), select `Edit-Preferences' in the menu, and then select the
`Advanced' tab in the dialog and then `Network':

Here, select `Settings…' to open the proxy settings dialog. Select
`Manual proxy configuration' and enter localhost with port 7777 under
SOCKS Host. Furthermore, set the checkbox `Proxy DNS when using SOCKS
v5' at the bottom of the dialog. Finally, push `OK'.

You must also go to @indicateurl{about:config} and change the
browser.fixup.alternate.enabled option to false, otherwise the browser
will autoblunder an address like www.gnu to www.gnu.com. If you want to
resolve @@ in your own TLDs, you must additionally set
browser.fixup.dns_first_use_for_single_words to true.

After configuring your browser, you might want to first confirm that it
continues to work as before. (The proxy is still experimental and if you
experience `odd' failures with some webpages, you might want to disable
it again temporarily.) Next, test if things work by typing
`http://test.gnu/' into the URL bar of your browser. This currently
fails with (my version of) Firefox as Firefox is super-smart and tries
to resolve `http://www.test.gnu/' instead of `test.gnu'. Chromium can be
convinced to comply if you explicitly include the `http://' prefix —
otherwise a Google search might be attempted, which is not what you
want. If successful, you should see a simple website.

Note that while you can use GNS to access ordinary websites, this is
more an experimental feature and not really our primary goal at this
time. Still, it is a possible use-case and we welcome help with testing
and development.

@node NSS plugin,,Integration with Browsers SOCKS proxy,Application integration
@anchor{users/gns nss-plugin}@anchor{cd}
@subsubsection NSS plugin


To activate the plugin, you need to edit your
@code{/etc/nsswitch.conf} where you should find a line like this:

@example
hosts: files mdns4_minimal [NOTFOUND=return] dns mdns4
@end example

The exact details may differ a bit, which is fine. Add the text
@code{gns [NOTFOUND=return]} after @code{files}:

@example
hosts: files gns [NOTFOUND=return] mdns4_minimal [NOTFOUND=return] dns mdns4
@end example

Note that some systems, in particular those running systemd-resolvd, manual modification of @code{/etc/nsswitch.conf} is not possible as it is autogenerated.
In such cases it is better to use the `DNS2GNS' service approach in combination with @code{resolvectl} as explained above.

@node Zone dissemination,Resource Records,Application integration,The GNU Name System
@anchor{users/gns zone-dissemination}@anchor{ce}
@subsection Zone dissemination


So you have decided to maintain and publish one or more GNS zones.
You are probably asking yourself how your zone finds its way into the
`Start Zone' configuration of other users.

Below are pointers on how to make your zone known.

@menu
* The .pin zone: The pin zone. 
* Be Social:: 
* Creating a business card:: 
* The GNUnet default Start Zones:: 
* The GNUnet GNS Registrar:: 

@end menu

@node The pin zone,Be Social,,Zone dissemination
@anchor{users/gns the-pin-zone}@anchor{cf}
@subsubsection The .pin zone


First, you may want to visit @indicateurl{https://fcfs.gnunet.org} and register your zone(s)
with the @code{.pin.gns.alt} top-level domain shipped by default with GNUnet.
This will allow other GNUnet users to resolve your zone under the name you
managed to acquire for yourself.

The registration policy of PIN is "first-come, first-served".

@node Be Social,Creating a business card,The pin zone,Zone dissemination
@anchor{users/gns be-social}@anchor{d0}
@subsubsection Be Social


Next, you should print out your business card and be social. Find a
friend, help them install GNUnet and exchange business cards with them.
Or, if you’re a desperate loner, you might try the next step with your
own card. Still, it’ll be hard to have a conversation with yourself
later, so it would be better if you could find a friend. You might also
want a camera attached to your computer, so you might need a trip to the
store together.

Before we get started, we need to tell gnunet-qr which zone it should
import new records into. For this, run:

@example
$ gnunet-identity -s namestore -e NAME
@end example

where NAME is the name of the zone you want to import records into. In
our running example, this would be `gnu'.

Henceforth, for every business card you collect, simply run:

@example
$ gnunet-qr
@end example

to open a window showing whatever your camera points at. Hold up your
friend’s business card and tilt it until the QR code is recognized. At
that point, the window should automatically close. At that point, your
friend’s NICKname and their public key should have been automatically
imported into your zone.

Assuming both of your peers are properly integrated in the GNUnet
network at this time, you should thus be able to resolve your friends
names. Suppose your friend’s nickname is `Bob'. Then, type

@example
$ gnunet-gns -u test.bob
@end example

to check if your friend was as good at following instructions as you
were.

@node Creating a business card,The GNUnet default Start Zones,Be Social,Zone dissemination
@anchor{users/gns creating-a-business-card}@anchor{d1}
@subsubsection Creating a business card


Before we can really use GNS, you should create a business card. Note
that this requires having LaTeX installed on your system. If you are
using a Debian GNU/Linux based operating system, the following command
should install the required components. Keep in mind that this requires
3GB of downloaded data and possibly even more when unpacked. On a GNU
Guix based system texlive 2017 has returns a DAG size of 5032.4 MiB. The
packages which are confirmed to be required are:


@itemize -

@item 
texlive-units

@item 
texlive-labels

@item 
texlive-pst-barcode

@item 
texlive-luatex85

@item 
texlive-preview

@item 
texlive-pdfcrop

@item 
texlive-koma-script
@end itemize

`We welcome any help in identifying the required components of the
TexLive Distribution. This way we could just state the required
components without pulling in the full distribution of TexLive.'

@example
apt-get install texlive-full
@end example

Start creating a business card by clicking the `Copy' button in
gnunet-namestore-gtk. Next, you should start the gnunet-bcd program (in
the terminal, on the command-line). You do not need to pass any options,
and please be not surprised if there is no output:

@example
$ gnunet-bcd
@end example

Then, start a browser and point it to @indicateurl{http://localhost:8888/} where
gnunet-bcd is running a Web server!

First, you might want to fill in the `GNS Public Key' field by
right-clicking and selecting `Paste', filling in the public key from the
copy you made in gnunet-namestore-gtk. Then, fill in all of the other
fields, including your GNS NICKname. Adding a GPG fingerprint is
optional. Once finished, click `Submit Query'. If your LaTeX
installation is incomplete, the result will be disappointing. Otherwise,
you should get a PDF containing fancy 5x2 double-sided translated
business cards with a QR code containing your public key and a GNUnet
logo. We’ll explain how to use those a bit later. You can now go back to
the shell running gnunet-bcd and press CTRL-C to shut down the Web
server.

@node The GNUnet default Start Zones,The GNUnet GNS Registrar,Creating a business card,Zone dissemination
@anchor{users/gns the-gnunet-default-start-zones}@anchor{d2}
@subsubsection The GNUnet default Start Zones


The GNUnet default `Start Zones' are managed through GANA@footnote{https://gana.gnunet.org} and
follows a restrictive registration policy@footnote{https://git.gnunet.org/gana.git/tree/gnu-name-system-default-tlds/POLICY}.

@node The GNUnet GNS Registrar,,The GNUnet default Start Zones,Zone dissemination
@anchor{users/gns the-gnunet-gns-registrar}@anchor{d3}
@subsubsection The GNUnet GNS Registrar


The GNUnet GNS Registrar is a standalone package availiable as download
in the GNU FTP Mirror@footnote{https://ftpmirror.gnu.org/gnunet}.

You can compile it using

@example
$ ./configure && make
@end example

To run it execute

@example
$ ./gnunet-gns-registrar
@end example

The registrar requires configuration of a GNU Taler Merchant@footnote{https://www.taler.net/en/}.
You can look at the example configuration for possible options.
@indicateurl{https://fcfs.gnunet.org} is running a GNS registrar instance connected to the GNU Taler
demo exchanges with a merchant running on @indicateurl{https://merchant.gnunet.org} for testing purposes
where you can exchange KUDOS (which you get for free upon registration at the GNU Taler Demo
system).
Consequently, while requiring KUDOS as payment to register, this is still effectively a
first-come-first server service.

@node Resource Records,DNS Migration,Zone dissemination,The GNU Name System
@anchor{users/gns resource-records}@anchor{c5}
@subsection Resource Records


GNS supports the majority of the DNS records as defined in RFC 1035@footnote{http://www.ietf.org/rfc/rfc1035.txt}. Additionally, GNS defines
some new record types the are unique to the GNS system. For example,
GNS-specific resource records are used to give petnames for zone
delegation, revoke zone keys and provide some compatibility features.

For some DNS records, GNS does extended processing to increase their
usefulness in GNS. In particular, GNS introduces special names referred
to as `zone relative names'. Zone relative names are allowed in some
resource record types (for example, in NS and CNAME records) and can
also be used in links on webpages. Zone relative names end in @cite{.+} which
indicates that the name needs to be resolved relative to the current
authoritative zone. The extended processing of those names will expand
the @cite{.+} with the correct delegation chain to the authoritative zone
(replacing @cite{.+} with the name of the location where the name was
encountered) and hence generate a valid GNS name.

The GNS currently supports the record types as defined in
GANA@footnote{https://git.gnunet.org/gana.git/tree/gnu-name-system-record-types/registry.rec}.
In addition, GNS supports DNS record types, such as @cite{A}, @cite{AAAA} or @cite{TXT}.

For a complete description of the records, please refer to the
specification at LSD0001@footnote{https://lsd.gnunet.org/lsd0001}.

In the following, we discuss GNS records with specific behavior or
special handling in GNUnet.

@menu
* VPN:: 

@end menu

@node VPN,,,Resource Records
@anchor{users/gns vpn}@anchor{d4}
@subsubsection VPN


GNS allows easy access to services provided by the GNUnet Virtual Public
Network. When the GNS resolver encounters a VPN record it will contact
the VPN service to try and allocate an IPv4/v6 address (if the queries
record type is an IP address) that can be used to contact the service.

`Example'

I want to provide access to the VPN service `web.gnu.' on port 80 on
peer ABC012: Name: www; RRType: VPN; Value: 80 ABC012 web.gnu.

The peer ABC012 is configured to provide an exit point for the service
`web.gnu.' on port 80 to it’s server running locally on port 8080 by
having the following lines in the gnunet.conf configuration file:

@example
[web.gnunet.]
TCP_REDIRECTS = 80:localhost4:8080
@end example

@node DNS Migration,,Resource Records,The GNU Name System
@anchor{users/gns dns-migration}@anchor{d5}
@subsection DNS Migration


The following tools facilitate DNS to GNS zone migrations.

The Zoneimport mechanism through @code{gnunet-zoneimport} allows you to mirror delegations in a TLD. We provide an example for AFNIC's @code{.fr}.
Why would you do it this way instead of a zone transfer using AXFR?
Because some TLD authorities, such as AFNIC, do not support AXFR and only publish a list
of delegated names.
In such cases, @code{gnunet-zoneimport} comes in handy. While not perfect in terms
of mirroring record updates and expirations, it is better than not being able to
mirror the @code{.fr} TLD at all.
In general, if the naming authority supports AXFR, then you should consider using Ascension (see below).
If you are the naming authority, then you should consider manually managing synchronization
between DNS and GNS or use the @code{gnunet-namestore-zonefile} tool to import DNS zonefiles
into GNS.

Note that the namestore by default also populates the namecache. This
pre-population is cryptographically expensive. Thus, on systems that
only serve to import a large (millions of records) DNS zone and that do
not have a local gns service in use, it is thus advisable to disable the
namecache by setting the option `DISABLE' to `YES' in section
`[namecache]'.

@menu
* Zoneimport:: 
* Zonefile:: 
* Ascension:: 

@end menu

@node Zoneimport,Zonefile,,DNS Migration
@anchor{users/gns zoneimport}@anchor{d6}
@subsubsection Zoneimport


If you want to support GNS but the master database for a zone is only
available and maintained in DNS, GNUnet includes the gnunet-zoneimport
tool to monitor a DNS zone and automatically import records into GNS.
Today, the tool does not yet support DNS AF(X)R, as we initially used it
on the `.fr' zone which does not allow us to perform a DNS zone
transfer. Instead, gnunet-zoneimport reads a list of DNS domain names
from stdin, issues DNS queries for each, converts the obtained records
(if possible) and stores the result in the namestore.

The zonemaster service then takes the records from the namestore,
publishes them into the DHT which makes the result available to the GNS
resolver. In the GNS configuration, non-local zones can be configured to
be intercepted by specifying `.tld = PUBLICKEY' in the configuration
file in the `[gns]' section.

For example, assuming you have the AFNIC NomsDeDomainEnPointFr CSV file you can execute:

@example
$ cat OPENDATA_A-NomsDeDomaineEnPointFr.csv | cut -d';' -f 1 | tail -n+2 | gnunet-zoneimport 194.0.9.1
@end example

This reads the first value of each line in the CSV as the domain name to import.
The first line is skipped because it (usually) is a header, not an entry.
The authoritative DNS server for @code{fr} is provided as IP.
Note that you need to provide authoritative DNS servers.

@node Zonefile,Ascension,Zoneimport,DNS Migration
@anchor{users/gns zonefile}@anchor{d7}
@subsubsection Zonefile


In order to import DNS zones which are already defined in a
zone file@footnote{https://en.wikipedia.org/wiki/Zone_file} in use
with existing DNS daemons such as BIND you may try to import your DNS zones
into DNS using @code{gnunet-namestore-zonefile}.
Assuming your zonefile is called @code{myzonefile}, execute:

@example
$ gnunet-namestore-zonefile < myzonefile
@end example

As you can see, the program reads the zonefile from standard input.
Upon encountering an @code{$ORIGIN} directive a new ego will be created with that
name unless it already exists.
Note that @code{$ORIGIN} is a FQDN and ends with a @code{.}, e.g. @code{example.com.}.
The ego will be called @code{example.com} (without the @code{.}).
All records following the directive will be imported into the zone identified by
this origin.
Multiple @code{$ORIGIN} directives within the same zone file are supported and
egos will be created accordingly.
If no @code{$ORIGIN} is explicitly specified in the beginning of the file
(as is sometimes the case in BIND setups) or if you want to override the (first)
occurence, you can provide an ego to use:

@example
$ gnunet-namestore-zonefile -z myorigin < myzonefile
@end example

Note that in the conversion process some information found in the zone file
is lost and other information is less precise than in GNS.
For example, records stored in the @code{$ORIGIN} can be specified in multiple
ways:

@example
$ORIGIN example.com.
$TTL 3600
example.com.    IN  MX    10 mail.example.com.  ; mail.example.com is the mailserver for example.com
@@               IN  MX    20 mail2.example.com. ; equivalent to above line, "@@" represents zone origin
@end example

GNS will map the origin @code{example.com.} to the empty label @code{@@} whenever possible.
Further, @code{$TTL} is given in seconds. GNS record expirations are defined in
microseconds. The seconds will be converted to their equivalent in microseconds.
Note how this means that conversion of GNS records (in particular their expirations)
to a DNS zone file cannot be done without loss of information.
This also means that converting the GNS zone back into a zone file (a function
currently not implemented) would not yield the same zone file that was imported.

@node Ascension,,Zonefile,DNS Migration
@anchor{users/gns ascension}@anchor{d8}
@subsubsection Ascension


Ascension is a tool to migrate existing DNS zones into GNS.

Compared to the gnunet-zoneimport tool it strictly uses AXFR or IXFR
depending on whether or not there exists a SOA record for the zone. If
that is the case it will take the serial as a reference point and
request the zone. The server will either answer the IXFR request with a
correct incremental zone or with the entire zone, which depends on the
server configuration.

Before you can migrate any zone though, you need to start a local GNUnet
peer. To migrate the Syrian top level domain - one of the few top level
domains that support zone transfers - into GNS use the following
command:

@example
$ ascension sy. -n ns1.tld.sy. -p
@end example

The -p flag will tell GNS to put these records on the DHT so that other
users may resolve these records by using the public key of the zone.

Once the zone is migrated, Ascension will output a message telling you,
that it will refresh the zone after the time has elapsed. You can
resolve the names in the zone directly using GNS or if you want to use
it with your browser, check out the GNS manual section. Configuring the
GNU Name System. To resolve the records from another system you need the
respective zones PKEY. To get the zones public key, you can run the
following command:

@example
$ gnunet-identity -dqe sy
@end example

Where `sy' is the name of the zone you want to migrate.

You can share the PKEY of the zone with your friends. They can then
resolve records in the zone by doing a lookup replacing the zone label
with your PKEY:

@example
$ gnunet-gns -t SOA -u "$PKEY"
@end example

The program will continue to run as a daemon and update once the refresh
time specified in the zones SOA record has elapsed.

DNSCurve style records are supported in the latest release and they are
added as a PKEY record to be referred to the respective GNS public key.
Key distribution is still a problem but provided someone else has a
public key under a given label it can be looked up.

There is an unofficial Debian package called python3-ascension that adds
a system user ascension and runs a GNUnet peer in the background.

Ascension-bind is also an unofficial Debian package that on installation
checks for running DNS zones and whether or not they are transferable
using DNS zone transfer (AXFR). It asks the administrator which zones to
migrate into GNS and installs a systemd unit file to keep the zone up to
date. If you want to migrate different zones you might want to check the
unit file from the package as a guide.

@node re claimID,File-sharing,The GNU Name System,User Manual
@anchor{users/reclaim doc}@anchor{d9}@anchor{users/reclaim re-claimid}@anchor{da}@anchor{users/reclaim reclaimid-identity-provider}@anchor{db}
@section re:claimID


The re:claimID Identity Provider (IdP) is a decentralized IdP service.
It allows its users to manage and authorize third parties to access
their identity attributes such as email or shipping addresses.

It basically mimics the concepts of centralized IdPs, such as those
offered by Google or Facebook. Like other IdPs, reclaimID features an
(optional) OpenID Connect 1.0-compliant protocol layer that can be used
for websites to integrate reclaimID as an Identity Provider with little
effort.

@menu
* Managing Attributes:: 
* Managing Credentials:: 
* Sharing Attributes with Third Parties:: 
* Revoking Authorizations of Third Parties:: 
* OpenID Connect:: 
* Providing Third Party Attestation:: 

@end menu

@node Managing Attributes,Managing Credentials,,re claimID
@anchor{users/reclaim id1}@anchor{dc}@anchor{users/reclaim managing-attributes}@anchor{dd}
@subsection Managing Attributes


Before adding attributes to an identity, you must first create an ego:

@example
$ gnunet-identity --create="user"
@end example

Henceforth, you can manage a new user profile of the user "user".

To add an email address to your user profile, simply use the
@code{gnunet-reclaim} command line tool:

@example
::
@end example

@quotation

$ gnunet-reclaim -e "user" -a "email" -V "@email{username@@example.gnunet}"
@end quotation

All of your attributes can be listed using the @code{gnunet-reclaim}
command line tool as well:

@example
$ gnunet-reclaim -e "user" -D
@end example

Currently, and by default, attribute values are interpreted as plain
text. In the future there might be more value types such as X.509
certificate credentials.

@node Managing Credentials,Sharing Attributes with Third Parties,Managing Attributes,re claimID
@anchor{users/reclaim id2}@anchor{de}@anchor{users/reclaim managing-credentials}@anchor{df}
@subsection Managing Credentials


Attribute values may reference a claim in a third party attested
credential. Such a credential can have a variety of formats such as
JSON-Web-Tokens or X.509 certificates. Currently, reclaimID only
supports JSON-Web-Token credentials.

To add a credential to your user profile, invoke the @code{gnunet-reclaim}
command line tool as follows:

@example
$ gnunet-reclaim -e "user"\
                 --credential-name="email"\
                 --credential-type="JWT"\
                 --value="ey..."
@end example

All of your credentials can be listed using the @code{gnunet-reclaim}
command line tool as well:

@example
$ gnunet-reclaim -e "user" --credentials
@end example

In order to add an attribe backed by a credential, specify the attribute
value as the claim name in the credential to reference along with the
credential ID:

@example
$ gnunet-reclaim -e "user"\
                 --add="email"\
                 --value="verified_email"\
                 --credential-id="<CREDENTIAL_ID>"
@end example

@node Sharing Attributes with Third Parties,Revoking Authorizations of Third Parties,Managing Credentials,re claimID
@anchor{users/reclaim id3}@anchor{e0}@anchor{users/reclaim sharing-attributes-with-third-parties}@anchor{e1}
@subsection Sharing Attributes with Third Parties


If you want to allow a third party such as a website or friend to access
to your attributes (or a subset thereof) execute:

@example
$ TICKET=$(gnunet-reclaim -e "user"\
                          -r "$RP_KEY"\
                          -i "attribute1,attribute2,...")
@end example

The command will return a "ticket" string. You must give $TICKET to
the requesting third party.

$RP_KEY is the public key of the third party and
"attribute1,attribute2,..." is a comma-separated list of attribute
names, such as "email,name,...", that you want to share.

The third party may retrieve the key in string format for use in the
above call using "gnunet-identity":

@example
$ RP_KEY=$(gnunet-identity -d | grep "relyingparty" | awk '@{print $3@}')
@end example

The third party can then retrieve your shared identity attributes using:

@example
$ gnunet-reclaim -e "relyingparty" -C "ticket"
@end example

Where "relyingparty" is the name for the identity behind $RP_KEY that
the requesting party is using. This will retrieve and list the shared
identity attributes. The above command will also work if the user is
currently offline since the attributes are retrieved from GNS. Further,
$TICKET can be re-used later to retrieve up-to-date attributes in case
"friend" has changed the value(s). For instance, because his email
address changed.

To list all given authorizations (tickets) you can execute:

@example
$ gnunet-reclaim -e "user" -T
@end example

@node Revoking Authorizations of Third Parties,OpenID Connect,Sharing Attributes with Third Parties,re claimID
@anchor{users/reclaim id4}@anchor{e2}@anchor{users/reclaim revoking-authorizations-of-third-parties}@anchor{e3}
@subsection Revoking Authorizations of Third Parties


If you want to revoke the access of a third party to your attributes you
can execute:

@example
$ gnunet-reclaim -e "user" -R $TICKET
@end example

This will prevent the third party from accessing the attribute in the
future. Please note that if the third party has previously accessed the
attribute, there is not way in which the system could have prevented the
thiry party from storing the data. As such, only access to updated data
in the future can be revoked. This behaviour is _exactly the same_ as
with other IdPs.

@node OpenID Connect,Providing Third Party Attestation,Revoking Authorizations of Third Parties,re claimID
@anchor{users/reclaim id5}@anchor{e4}@anchor{users/reclaim openid-connect}@anchor{e5}
@subsection OpenID Connect


There is an
https://openid.net/specs/openid-connect-core-1_0.html@footnote{OpenIDConnect}
API for use with re:claimID. However, its use is quite complicated to
setup.

@example
https://api.reclaim/openid/authorize
http://localhost:7776/openid/token
http://localhost:7776/openid/userinfo
http://localhost:7776/openid/login
@end example

The token endpoint is protected using HTTP basic authentication. You can
authenticate using any username and the password configured under:

@example
$ gnunet-config -s reclaim-rest-plugin -o OIDC_CLIENT_SECRET
@end example

The authorize endpoint is protected using a Cookie which can be obtained
through a request against the login endpoint. This functionality is
meant to be used in the context of the OpenID Connect authorization flow
to collect user consent interactively. Without a Cookie, the authorize
endpoint redirects to a URI configured under:

@example
$ gnunet-config -s reclaim-rest-plugin -o ADDRESS
@end example

The token endpoint is protected using OAuth2 and expects the grant which
is retrieved from the authorization endpoint according to the standard.

The userinfo endpoint is protected using OAuth2 and expects a bearer
access token which is retrieved from a token request.

In order to make use of OpenID Connect flows as a user, you need to
install the browser plugin:


@itemize -

@item 
Firefox Add-on@footnote{https://addons.mozilla.org/addon/reclaimid/}

@item 
Chrome Web Store@footnote{https://chrome.google.com/webstore/detail/reclaimid/jiogompmdejcnacmlnjhnaicgkefcfll}
@end itemize

In order to create and register an OpenID Connect client as a relying
party, you need to execute the following steps:

@example
$ gnunet-identity -C <client_name>
$ gnunet-namestore -z <client_name> -a -n "@@" -t RECLAIM_OIDC_REDIRECT -V <redirect_uri> -e 1d -p
$ gnunet-namestore -z <client_name> -a -n "@@" -t RECLAIM_OIDC_CLIENT -V "My OIDC Client" -e 1d -p
@end example

The "client_id" for use in OpenID Connect is the public key of the
client as displayed using:

@example
$ gnunet-identity -d grep "relyingparty" | awk '@{print $3@}'
@end example

The RECLAIM_OIDC_REDIRECT record contains your website redirect URI. You
may use any globally unique DNS or GNS URI. The RECLAIM_OIDC_CLIENT
record represents the client description which whill be displayed to
users in an authorization request.

Any website or relying party must use the authorization endpoint
@indicateurl{https://api.reclaim/openid/authorize} in its authorization redirects,
e.g.

@example
<a href="https://api.reclaim/openid/authorize?client_id=<PKEY>\
                                             &scope=openid email\
                                             &redirect_uri=<redirect_uri>\
                                             &nonce=<random>">Login</a>
@end example

This will direct the user's browser onto his local reclaimID instance.
After giving consent, you will be provided with the OpenID Connect
authorization code according to the specifications at your provided
redirect URI.

The ID Tokens issues by the token endpoints are signed using HS512 with
the shared secret configured under:

@example
$ gnunet-config -s reclaim-rest-plugin -o JWT_SECRET
@end example

The authorization code flow optionally supports Proof Key for Code Exchange@footnote{https://tools.ietf.org/html/rfc7636}. If PKCE is used, the
client does not need to authenticate against the token endpoint.

@node Providing Third Party Attestation,,OpenID Connect,re claimID
@anchor{users/reclaim id6}@anchor{e6}@anchor{users/reclaim providing-third-party-attestation}@anchor{e7}
@subsection Providing Third Party Attestation


If you are running an identity provider (IdP) service you may be able to
support providing credentials for re:claimID users. IdPs can issue JWT
credentials as long as they support OpenID Connect and OpenID Connect Discovery@footnote{https://openid.net/specs/openid-connect-discovery-1_0.html}.

In order to allow users to import attributes through the re:claimID user
interface, you need to register the following public OAuth2/OIDC client:


@itemize -

@item 
client_id: reclaimid

@item 
client_secret: none

@item 
redirect_uri: @indicateurl{https://ui.reclaim} (The URI of the re:claimID
webextension)

@item 
grant_type: authorization_code with PKCE
(RFC7636@footnote{https://tools.ietf.org/html/rfc7636})

@item 
scopes: all you want to offer.

@item 
id_token: JWT
@end itemize

When your users add an attribute with name "email" which supports
webfinger discovery they will be prompted with the option to retrieve
the OpenID Connect ID Token through the user interface.

@node File-sharing,Virtual Public Network,re claimID,User Manual
@anchor{users/fs doc}@anchor{e8}@anchor{users/fs file-sharing}@anchor{e9}
@section File-sharing


This chapter documents the GNUnet file-sharing application. The original
file-sharing implementation for GNUnet was designed to provide anonymous
file-sharing. However, over time, we have also added support for
non-anonymous file-sharing (which can provide better performance).
Anonymous and non-anonymous file-sharing are quite integrated in GNUnet
and, except for routing, share most of the concepts and implementation.
There are three primary file-sharing operations: publishing, searching
and downloading. For each of these operations, the user specifies an
anonymity level. If both the publisher and the searcher/downloader
specify “no anonymity”, non-anonymous file-sharing is used. If either
user specifies some desired degree of anonymity, anonymous file-sharing
will be used.

After a short introduction, we will first look at the various concepts
in GNUnet’s file-sharing implementation. Then, we will discuss specifics
as to how they impact users that publish, search or download files.

@menu
* Searching:: 
* Downloading:: 
* Publishing:: 
* Indexing vs Inserting:: 
* Concepts:: 
* Namespace Management:: 
* File-Sharing URIs:: 

@end menu

@node Searching,Downloading,,File-sharing
@anchor{users/fs searching}@anchor{ea}
@subsection Searching


The command @code{gnunet-search} can be used to search for content on
GNUnet. The format is:

@example
$ gnunet-search [-t TIMEOUT] KEYWORD
@end example

The @code{-t} option specifies that the query should timeout after
approximately @code{TIMEOUT} seconds. A value of zero (“0”) is interpreted
as no timeout, which is the default. In this case, gnunet-search will
never terminate (unless you press CTRL-C).

If multiple words are passed as keywords, they will all be considered
optional. Prefix keywords with a “+” to make them mandatory.

Note that searching using:

@example
$ gnunet-search Das Kapital
@end example

is not the same as searching for

@example
$ gnunet-search "Das Kapital"
@end example

as the first will match files shared under the keywords “Das” or
“Kapital” whereas the second will match files shared under the keyword
“Das Kapital”.

Search results are printed like this:

@example
#15:
gnunet-download -o "COPYING" gnunet://fs/chk/PGK8M...3EK130.75446
@end example

The whole line is the command you would have to enter to download the
file. The first argument passed to @code{-o} is the suggested filename (you
may change it to whatever you like). It is followed by the key for
decrypting the file, the query for searching the file, a checksum (in
hexadecimal) finally the size of the file in bytes.

@node Downloading,Publishing,Searching,File-sharing
@anchor{users/fs downloading}@anchor{eb}
@subsection Downloading


In order to download a file, you need the whole line returned by
gnunet-search. You can then use the tool @code{gnunet-download} to obtain
the file:

@example
$ gnunet-download -o <FILENAME> <GNUNET-URL>
@end example

@code{FILENAME} specifies the name of the file where GNUnet is supposed to
write the result. Existing files are overwritten. If the existing file
contains blocks that are identical to the desired download, those blocks
will not be downloaded again (automatic resume).

If you want to download the GPL from the previous example, you do the
following:

@example
$ gnunet-download -o "COPYING" gnunet://fs/chk/PGK8M...3EK130.75446
@end example

If you ever have to abort a download, you can continue it at any time by
re-issuing gnunet-download with the same filename. In that case, GNUnet
will `not' download blocks again that are already present.

GNUnet’s file-encoding mechanism will ensure file integrity, even if the
existing file was not downloaded from GNUnet in the first place.

You may want to use the @code{-V} switch to turn on verbose reporting. In
this case, gnunet-download will print the current number of bytes
downloaded whenever new data was received.

@node Publishing,Indexing vs Inserting,Downloading,File-sharing
@anchor{users/fs publishing}@anchor{ec}
@subsection Publishing


The command @code{gnunet-publish} can be used to add content to the
network. The basic format of the command is:

@example
$ gnunet-publish [-n] [-k KEYWORDS]* [-m TYPE:VALUE] FILENAME
@end example

For example:

@example
$ gnunet-publish -m "description:GNU License" -k gpl -k test -m "mimetype:text/plain" COPYING
@end example

The option @code{-k} is used to specify keywords for the file that should
be inserted. You can supply any number of keywords, and each of the
keywords will be sufficient to locate and retrieve the file. Please note
that you must use the @code{-k} option more than once – one for each
expression you use as a keyword for the filename.

The @code{-m} option is used to specify meta-data, such as descriptions.
You can use @code{-m} multiple times. The @code{TYPE} passed must be from the
list of meta-data types known to libextractor. You can obtain this list
by running @code{extract -L}. Use quotes around the entire meta-data
argument if the value contains spaces. The meta-data is displayed to
other users when they select which files to download. The meta-data and
the keywords are optional and may be inferred using GNU libextractor.

@code{gnunet-publish} has a few additional options to handle namespaces and
directories. Refer to the man-page for details.

@node Indexing vs Inserting,Concepts,Publishing,File-sharing
@anchor{users/fs indexing-vs-inserting}@anchor{ed}
@subsection Indexing vs Inserting


By default, GNUnet indexes a file instead of making a full copy. This is
much more efficient, but requires the file to stay unaltered at the
location where it was when it was indexed. If you intend to move, delete
or alter a file, consider using the option @code{-n} which will force
GNUnet to make a copy of the file in the database.

Since it is much less efficient, this is strongly discouraged for large
files. When GNUnet indexes a file (default), GNUnet does `not' create
an additional encrypted copy of the file but just computes a summary (or
index) of the file. That summary is approximately two percent of the
size of the original file and is stored in GNUnet’s database. Whenever a
request for a part of an indexed file reaches GNUnet, this part is
encrypted on-demand and send out. This way, there is no need for an
additional encrypted copy of the file to stay anywhere on the drive.
This is different from other systems, such as Freenet, where each file
that is put online must be in Freenet’s database in encrypted format,
doubling the space requirements if the user wants to preserve a directly
accessible copy in plaintext.

Thus indexing should be used for all files where the user will keep
using this file (at the location given to gnunet-publish) and does not
want to retrieve it back from GNUnet each time. If you want to remove a
file that you have indexed from the local peer, use the tool
gnunet-unindex to un-index the file.

The option @code{-n} may be used if the user fears that the file might be
found on their drive (assuming the computer comes under the control of
an adversary). When used with the @code{-n} flag, the user has a much
better chance of denying knowledge of the existence of the file, even if
it is still (encrypted) on the drive and the adversary is able to crack
the encryption (e.g. by guessing the keyword).

@node Concepts,Namespace Management,Indexing vs Inserting,File-sharing
@anchor{users/fs concepts}@anchor{ee}@anchor{users/fs fs-002dconcepts}@anchor{ef}
@subsection Concepts


For better results with filesharing it is useful to understand the
following concepts. In addition to anonymous routing GNUnet attempts to
give users a better experience in searching for content. GNUnet uses
cryptography to safely break content into smaller pieces that can be
obtained from different sources without allowing participants to corrupt
files. GNUnet makes it difficult for an adversary to send back bogus
search results. GNUnet enables content providers to group related
content and to establish a reputation. Furthermore, GNUnet allows
updates to certain content to be made available. This section is
supposed to introduce users to the concepts that are used to achieve
these goals.

@menu
* Files:: 
* Keywords:: 
* Directories:: 
* Egos and File-Sharing:: 
* Namespaces:: 
* Advertisements:: 
* Anonymity level:: 
* Content Priority:: 
* Replication:: 

@end menu

@node Files,Keywords,,Concepts
@anchor{users/fs files}@anchor{f0}@anchor{users/fs id1}@anchor{f1}
@subsubsection Files


A file in GNUnet is just a sequence of bytes. Any file-format is allowed
and the maximum file size is theoretically @math{2^64 - 1} bytes,
except that it would take an impractical amount of time to share such a
file. GNUnet itself never interprets the contents of shared files,
except when using GNU libextractor to obtain keywords.

@node Keywords,Directories,Files,Concepts
@anchor{users/fs id2}@anchor{f2}@anchor{users/fs keywords}@anchor{f3}
@subsubsection Keywords


Keywords are the most simple mechanism to find files on GNUnet. Keywords
are `case-sensitive' and the search string must always match
`exactly' the keyword used by the person providing the file. Keywords
are never transmitted in plaintext. The only way for an adversary to
determine the keyword that you used to search is to guess it (which then
allows the adversary to produce the same search request). Since
providing keywords by hand for each shared file is tedious, GNUnet uses
GNU libextractor to help automate this process. Starting a keyword
search on a slow machine can take a little while since the keyword
search involves computing a fresh RSA key to formulate the request.

@node Directories,Egos and File-Sharing,Keywords,Concepts
@anchor{users/fs directories}@anchor{f4}@anchor{users/fs id3}@anchor{f5}
@subsubsection Directories


A directory in GNUnet is a list of file identifiers with meta data. The
file identifiers provide sufficient information about the files to allow
downloading the contents. Once a directory has been created, it cannot
be changed since it is treated just like an ordinary file by the
network. Small files (of a few kilobytes) can be inlined in the
directory, so that a separate download becomes unnecessary.

Directories are shared just like ordinary files. If you download a
directory with @code{gnunet-download}, you can use @code{gnunet-directory} to
list its contents. The canonical extension for GNUnet directories when
stored as files in your local file-system is ".gnd". The contents of a
directory are URIs and meta data. The URIs contain all the information
required by @code{gnunet-download} to retrieve the file. The meta data
typically includes the mime-type, description, a filename and other meta
information, and possibly even the full original file (if it was small).

@node Egos and File-Sharing,Namespaces,Directories,Concepts
@anchor{users/fs egos-and-file-002dsharing}@anchor{f6}@anchor{users/fs egos-and-file-sharing}@anchor{f7}
@subsubsection Egos and File-Sharing


When sharing files, it is sometimes desirable to build a reputation as a
source for quality information. With egos, publishers can
(cryptographically) sign files, thereby demonstrating that various files
were published by the same entity. An ego thus allows users to link
different publication events, thereby deliberately reducing anonymity to
pseudonymity.

Egos used in GNUnet's file-sharing for such pseudonymous publishing also
correspond to the egos used to identify and sign zones in the GNU Name
System. However, if the same ego is used for file-sharing and for a GNS
zone, this will weaken the privacy assurances provided by the anonymous
file-sharing protocol.

Note that an ego is NOT bound to a GNUnet peer. There can be multiple
egos for a single user, and users could (theoretically) share the
private keys of an ego by copying the respective private keys.

@node Namespaces,Advertisements,Egos and File-Sharing,Concepts
@anchor{users/fs id4}@anchor{f8}@anchor{users/fs namespaces}@anchor{f9}
@subsubsection Namespaces


A namespace is a set of files that were signed by the same ego. Today,
namespaces are implemented independently of GNS zones, but in the future
we plan to merge the two such that a GNS zone can basically contain
files using a file-sharing specific record type.

Files (or directories) that have been signed and placed into a namespace
can be updated. Updates are identified as authentic if the same secret
key was used to sign the update.

@node Advertisements,Anonymity level,Namespaces,Concepts
@anchor{users/fs advertisements}@anchor{fa}@anchor{users/fs id5}@anchor{fb}
@subsubsection Advertisements


Advertisements are used to notify other users about the existence of a
namespace. Advertisements are propagated using the normal keyword
search. When an advertisement is received (in response to a search), the
namespace is added to the list of namespaces available in the
namespace-search dialogs of gnunet-fs-gtk and printed by
@code{gnunet-identity}. Whenever a namespace is created, an appropriate
advertisement can be generated. The default keyword for the advertising
of namespaces is "namespace".

@node Anonymity level,Content Priority,Advertisements,Concepts
@anchor{users/fs anonymity-level}@anchor{fc}@anchor{users/fs id6}@anchor{fd}
@subsubsection Anonymity level


The anonymity level determines how hard it should be for an adversary to
determine the identity of the publisher or the searcher/downloader. An
anonymity level of zero means that anonymity is not required. The
default anonymity level of "1" means that anonymous routing is
desired, but no particular amount of cover traffic is necessary. A
powerful adversary might thus still be able to deduce the origin of the
traffic using traffic analysis. Specifying higher anonymity levels
increases the amount of cover traffic required.

The specific numeric value (for anonymity levels above 1) is simple:
Given an anonymity level L (above 1), each request FS makes on your
behalf must be hidden in L-1 equivalent requests of cover traffic
(traffic your peer routes for others) in the same time-period. The
time-period is twice the average delay by which GNUnet artificially
delays traffic.

While higher anonymity levels may offer better privacy, they can also
significantly hurt performance.

@node Content Priority,Replication,Anonymity level,Concepts
@anchor{users/fs content-priority}@anchor{fe}@anchor{users/fs id7}@anchor{ff}
@subsubsection Content Priority


Depending on the peer's configuration, GNUnet peers migrate content
between peers. Content in this sense are individual blocks of a file,
not necessarily entire files. When peers run out of space (due to local
publishing operations or due to migration of content from other peers),
blocks sometimes need to be discarded. GNUnet first always discards
expired blocks (typically, blocks are published with an expiration of
about two years in the future; this is another option). If there is
still not enough space, GNUnet discards the blocks with the lowest
priority. The priority of a block is decided by its popularity (in terms
of requests from peers we trust) and, in case of blocks published
locally, the base-priority that was specified by the user when the block
was published initially.

@node Replication,,Content Priority,Concepts
@anchor{users/fs id8}@anchor{100}@anchor{users/fs replication}@anchor{101}
@subsubsection Replication


When peers migrate content to other systems, the replication level of a
block is used to decide which blocks need to be migrated most urgently.
GNUnet will always push the block with the highest replication level
into the network, and then decrement the replication level by one. If
all blocks reach replication level zero, the selection is simply random.

@node Namespace Management,File-Sharing URIs,Concepts,File-sharing
@anchor{users/fs id9}@anchor{102}@anchor{users/fs namespace-management}@anchor{103}
@subsection Namespace Management


The @code{gnunet-identity} tool can be used to create egos. By default,
@code{gnunet-identity --display} simply lists all locally available egos.

@menu
* Creating Egos:: 
* Deleting Egos:: 

@end menu

@node Creating Egos,Deleting Egos,,Namespace Management
@anchor{users/fs creating-egos}@anchor{104}@anchor{users/fs id10}@anchor{105}
@subsubsection Creating Egos


With the @code{--create=NICK} option it can also be used to create a new
ego. An ego is the virtual identity of the entity in control of a
namespace or GNS zone. Anyone can create any number of egos. The
provided NICK name automatically corresponds to a GNU Name System domain
name. Thus, henceforth name resolution for any name ending in ".NICK"
will use the NICK's zone. You should avoid using NICKs that collide with
well-known DNS names.

Currently, the IDENTITY subsystem supports two types of identity keys:
ECDSA and EdDSA. By default, ECDSA identities are creates with ECDSA
keys. In order to create an identity with EdDSA keys, you can use the
@code{--eddsa} flag.

@node Deleting Egos,,Creating Egos,Namespace Management
@anchor{users/fs deleting-egos}@anchor{106}@anchor{users/fs id11}@anchor{107}
@subsubsection Deleting Egos


With the @code{-D NICK} option egos can be deleted. Once the ego has been
deleted it is impossible to add content to the corresponding namespace
or zone. However, the existing GNS zone data is currently not dropped.
This may change in the future.

Deleting the pseudonym does not make the namespace or any content in it
unavailable.

@node File-Sharing URIs,,Namespace Management,File-sharing
@anchor{users/fs file-002dsharing-uris}@anchor{108}@anchor{users/fs file-sharing-uris}@anchor{109}
@subsection File-Sharing URIs


GNUnet (currently) uses four different types of URIs for file-sharing.
They all begin with "gnunet://fs/". This section describes the four
different URI types in detail.

For FS URIs empty KEYWORDs are not allowed. Quotes are allowed to denote
whitespace between words. Keywords must contain a balanced number of
double quotes. Doubles quotes can not be used in the actual keywords.
This means that the string '""foo bar""' will be turned into two
OR-ed keywords 'foo' and 'bar', not into '"foo bar"'.

@menu
* Encoding of hash values in URIs:: 
* Content Hash Key (chk): Content Hash Key chk. 
* Location identifiers (loc): Location identifiers loc. 
* Keyword queries (ksk): Keyword queries ksk. 
* Namespace content (sks): Namespace content sks. 

@end menu

@node Encoding of hash values in URIs,Content Hash Key chk,,File-Sharing URIs
@anchor{users/fs encoding-of-hash-values-in-uris}@anchor{10a}@anchor{users/fs id12}@anchor{10b}
@subsubsection Encoding of hash values in URIs


Most URIs include some hash values. Hashes are encoded using base32hex
(RFC 2938).

chk-uri
.. _Content-Hash-Key-_0028chk_0029:

@node Content Hash Key chk,Location identifiers loc,Encoding of hash values in URIs,File-Sharing URIs
@anchor{users/fs content-hash-key-chk}@anchor{10c}
@subsubsection Content Hash Key (chk)


A chk-URI is used to (uniquely) identify a file or directory and to
allow peers to download the file. Files are stored in GNUnet as a tree
of encrypted blocks. The chk-URI thus contains the information to
download and decrypt those blocks. A chk-URI has the format
"gnunet://fs/chk/KEYHASH.QUERYHASH.SIZE". Here, "SIZE" is the size
of the file (which allows a peer to determine the shape of the tree),
KEYHASH is the key used to decrypt the file (also the hash of the
plaintext of the top block) and QUERYHASH is the query used to request
the top-level block (also the hash of the encrypted block).

loc-uri
.. _Location-identifiers-_0028loc_0029:

@node Location identifiers loc,Keyword queries ksk,Content Hash Key chk,File-Sharing URIs
@anchor{users/fs location-identifiers-loc}@anchor{10d}
@subsubsection Location identifiers (loc)


For non-anonymous file-sharing, loc-URIs are used to specify which peer
is offering the data (in addition to specifying all of the data from a
chk-URI). Location identifiers include a digital signature of the peer
to affirm that the peer is truly the origin of the data. The format is
"gnunet://fs/loc/KEYHASH.QUERYHASH.SIZE.PEER.SIG.EXPTIME". Here,
"PEER" is the public key of the peer (in GNUnet format in base32hex),
SIG is the RSA signature (in GNUnet format in base32hex) and EXPTIME
specifies when the signature expires (in milliseconds after 1970).

ksk-uri
.. _Keyword-queries-_0028ksk_0029:

@node Keyword queries ksk,Namespace content sks,Location identifiers loc,File-Sharing URIs
@anchor{users/fs keyword-queries-ksk}@anchor{10e}
@subsubsection Keyword queries (ksk)


A keyword-URI is used to specify that the desired operation is the
search using a particular keyword. The format is simply
"gnunet://fs/ksk/KEYWORD". Non-ASCII characters can be specified using
the typical URI-encoding (using hex values) from HTTP. "+" can be used
to specify multiple keywords (which are then logically "OR"-ed in the
search, results matching both keywords are given a higher rank):
"gnunet://fs/ksk/KEYWORD1+KEYWORD2". ksk-URIs must not begin or end
with the plus ('+') character. Furthermore they must not contain '++'.

sks-uri
.. _Namespace-content-_0028sks_0029:

@node Namespace content sks,,Keyword queries ksk,File-Sharing URIs
@anchor{users/fs namespace-content-sks}@anchor{10f}
@subsubsection Namespace content (sks)


`Please note that the text in this subsection is outdated and needs'
`to be rewritten for version 0.10!' `This especially concerns the
terminology of Pseudonym/Ego/Identity.'

Namespaces are sets of files that have been approved by some (usually
pseudonymous) user --- typically by that user publishing all of the
files together. A file can be in many namespaces. A file is in a
namespace if the owner of the ego (aka the namespace's private key)
signs the CHK of the file cryptographically. An SKS-URI is used to
search a namespace. The result is a block containing meta data, the CHK
and the namespace owner's signature. The format of a sks-URI is
"gnunet://fs/sks/NAMESPACE/IDENTIFIER". Here, "NAMESPACE" is the
public key for the namespace. "IDENTIFIER" is a freely chosen keyword
(or password!). A commonly used identifier is "root" which by
convention refers to some kind of index or other entry point into the
namespace.

@node Virtual Public Network,Messenger,File-sharing,User Manual
@anchor{users/vpn doc}@anchor{110}@anchor{users/vpn using-the-virtual-public-network}@anchor{111}@anchor{users/vpn virtual-public-network}@anchor{112}
@section Virtual Public Network


Using the GNUnet Virtual Public Network (VPN) application you can tunnel
IP traffic over GNUnet. Moreover, the VPN comes with built-in protocol
translation and DNS-ALG support, enabling IPv4-to-IPv6 protocol
translation (in both directions). This chapter documents how to use the
GNUnet VPN.

The first thing to note about the GNUnet VPN is that it is a public
network. All participating peers can participate and there is no secret
key to control access. So unlike common virtual private networks, the
GNUnet VPN is not useful as a means to provide a "private" network
abstraction over the Internet. The GNUnet VPN is a virtual network in
the sense that it is an overlay over the Internet, using its own routing
mechanisms and can also use an internal addressing scheme. The GNUnet
VPN is an Internet underlay --- TCP/IP applications run on top of it.

The VPN is currently only supported on GNU/Linux systems. Support for
operating systems that support TUN (such as FreeBSD) should be easy to
add (or might not even require any coding at all --- we just did not
test this so far). Support for other operating systems would require
re-writing the code to create virtual network interfaces and to
intercept DNS requests.

The VPN does not provide good anonymity. While requests are routed over
the GNUnet network, other peers can directly see the source and
destination of each (encapsulated) IP packet. Finally, if you use the
VPN to access Internet services, the peer sending the request to the
Internet will be able to observe and even alter the IP traffic. We will
discuss additional security implications of using the VPN later in this
chapter.

@menu
* Setting up an Exit node:: 
* Fedora and the Firewall:: 
* Setting up VPN node for protocol translation and tunneling:: 

@end menu

@node Setting up an Exit node,Fedora and the Firewall,,Virtual Public Network
@anchor{users/vpn id1}@anchor{113}@anchor{users/vpn setting-up-an-exit-node}@anchor{114}
@subsection Setting up an Exit node


Any useful operation with the VPN requires the existence of an exit node
in the GNUnet Peer-to-Peer network. Exit functionality can only be
enabled on peers that have regular Internet access. If you want to play
around with the VPN or support the network, we encourage you to setup
exit nodes. This chapter documents how to setup an exit node.

There are four types of exit functions an exit node can provide, and
using the GNUnet VPN to access the Internet will only work nicely if the
first three types are provided somewhere in the network. The four exit
functions are:


@itemize -

@item 
DNS: allow other peers to use your DNS resolver

@item 
IPv4: allow other peers to access your IPv4 Internet connection

@item 
IPv6: allow other peers to access your IPv6 Internet connection

@item 
Local service: allow other peers to access a specific TCP or UDP
service your peer is providing
@end itemize

By enabling "exit" in gnunet-setup and checking the respective boxes
in the "exit" tab, you can easily choose which of the above exit
functions you want to support.

Note, however, that by supporting the first three functions you will
allow arbitrary other GNUnet users to access the Internet via your
system. This is somewhat similar to running a Tor exit node. The
Torproject has a nice article about what to consider if you want to do
this here. We believe that generally running a DNS exit node is
completely harmless.

The exit node configuration does currently not allow you to restrict the
Internet traffic that leaves your system. In particular, you cannot
exclude SMTP traffic (or block port 25) or limit to HTTP traffic using
the GNUnet configuration. However, you can use your host firewall to
restrict outbound connections from the virtual tunnel interface. This is
highly recommended. In the future, we plan to offer a wider range of
configuration options for exit nodes.

Note that by running an exit node GNUnet will configure your kernel to
perform IP-forwarding (for IPv6) and NAT (for IPv4) so that the traffic
from the virtual interface can be routed to the Internet. In order to
provide an IPv6-exit, you need to have a subnet routed to your host's
external network interface and assign a subrange of that subnet to the
GNUnet exit's TUN interface.

When running a local service, you should make sure that the local
service is (also) bound to the IP address of your EXIT interface (e.g.
169.254.86.1). It will NOT work if your local service is just bound to
loopback. You may also want to create a "VPN" record in your zone of
the GNU Name System to make it easy for others to access your service
via a name instead of just the full service descriptor. Note that the
identifier you assign the service can serve as a passphrase or shared
secret, clients connecting to the service must somehow learn the
service's name. VPN records in the GNU Name System can make this easier.

The config file configuration to offer a service @cite{http} on port 80 which
is forwarded to port 8080 on IP @cite{169.254.86.1} is as follows:

@example
[http.gnunet.]
TCP_REDIRECTS = 80:169.254.86.1:8080
@end example

You can then add a GNS record to this service:

@example
$  gnunet-namestore -z myzone -a -e "1 d" -p -t VPN -n www -V "6 <ZKEY> http"
@end example

Where @cite{myzone} is the name of your GNS zone.
We are adding a record of type @cite{VPN}, and the value is a string containing three values:
The first is a boolean indicating the use of TCP (@cite{6}) or UDP (@cite{17}),
your peer identity and the identifier of the service we used in the configuration section above
before the @cite{.gnunet.}.

When using the @cite{dns2gns} DNS server it will automatically synthesize @cite{A}/@cite{AAAA} records from
encountered @cite{VPN} records using the @cite{VPN} service.
Otherwise, this must be done by the client that resolved the @cite{VPN} record.

@node Fedora and the Firewall,Setting up VPN node for protocol translation and tunneling,Setting up an Exit node,Virtual Public Network
@anchor{users/vpn fedora-and-the-firewall}@anchor{115}@anchor{users/vpn id2}@anchor{116}
@subsection Fedora and the Firewall


When using an exit node on Fedora 15, the standard firewall can create
trouble even when not really exiting the local system! For IPv4, the
standard rules seem fine. However, for IPv6 the standard rules prohibit
traffic from the network range of the virtual interface created by the
exit daemon to the local IPv6 address of the same interface (which is
essentially loopback traffic, so you might suspect that a standard
firewall would leave this traffic alone). However, as somehow for IPv6
the traffic is not recognized as originating from the local system (and
as the connection is not already "established"), the firewall drops
the traffic. You should still get ICMPv6 packets back, but that's
obviously not very useful.

Possible ways to fix this include disabling the firewall (do you have a
good reason for having it on?) or disabling the firewall at least for
the GNUnet exit interface (or the respective IPv4/IPv6 address range).
The best way to diagnose these kinds of problems in general involves
setting the firewall to REJECT instead of DROP and to watch the traffic
using wireshark (or tcpdump) to see if ICMP messages are generated when
running some tests that should work.

@node Setting up VPN node for protocol translation and tunneling,,Fedora and the Firewall,Virtual Public Network
@anchor{users/vpn id3}@anchor{117}@anchor{users/vpn setting-up-vpn-node-for-protocol-translation-and-tunneling}@anchor{118}
@subsection Setting up VPN node for protocol translation and tunneling


The GNUnet VPN/PT subsystem enables you to tunnel IP traffic over the
VPN to an exit node, from where it can then be forwarded to the
Internet. This section documents how to setup VPN/PT on a node. Note
that you can enable both the VPN and an exit on the same peer. In this
case, IP traffic from your system may enter your peer's VPN and leave
your peer's exit. This can be useful as a means to do protocol
translation. For example, you might have an application that supports
only IPv4 but needs to access an IPv6-only site. In this case, GNUnet
would perform 4to6 protocol translation between the VPN (IPv4) and the
Exit (IPv6). Similarly, 6to4 protocol translation is also possible.
However, the primary use for GNUnet would be to access an Internet
service running with an IP version that is not supported by your ISP. In
this case, your IP traffic would be routed via GNUnet to a peer that has
access to the Internet with the desired IP version.

Setting up an entry node into the GNUnet VPN primarily requires you to
enable the "VPN/PT" option in "gnunet-setup". This will launch the
"gnunet-service-vpn", "gnunet-service-dns" and "gnunet-daemon-pt"
processes. The "gnunet-service-vpn" will create a virtual interface
which will be used as the target for your IP traffic that enters the
VPN. Additionally, a second virtual interface will be created by the
"gnunet-service-dns" for your DNS traffic. You will then need to
specify which traffic you want to tunnel over GNUnet. If your ISP only
provides you with IPv4 or IPv6-access, you may choose to tunnel the
other IP protocol over the GNUnet VPN. If you do not have an ISP (and
are connected to other GNUnet peers via WLAN), you can also choose to
tunnel all IP traffic over GNUnet. This might also provide you with some
anonymity. After you enable the respective options and restart your
peer, your Internet traffic should be tunneled over the GNUnet VPN.

The GNUnet VPN uses DNS-ALG to hijack your IP traffic. Whenever an
application resolves a hostname (like 'gnunet.org'), the
"gnunet-daemon-pt" will instruct the "gnunet-service-dns" to
intercept the request (possibly route it over GNUnet as well) and
replace the normal answer with an IP in the range of the VPN's
interface. "gnunet-daemon-pt" will then tell "gnunet-service-vpn" to
forward all traffic it receives on the TUN interface via the VPN to the
original destination.

For applications that do not use DNS, you can also manually create such
a mapping using the gnunet-vpn command-line tool. Here, you specify the
desired address family of the result (e.g. "-4"), and the intended
target IP on the Internet (e.g. "-i 131.159.74.67") and "gnunet-vpn"
will tell you which IP address in the range of your VPN tunnel was
mapped.

@code{gnunet-vpn} can also be used to access "internal" services offered
by GNUnet nodes. So if you happen to know a peer and a service offered
by that peer, you can create an IP tunnel to that peer by specifying the
peer's identity, service name and protocol (--tcp or --udp) and you will
again receive an IP address that will terminate at the respective peer's
service.

@node Messenger,Advanced Configuration,Virtual Public Network,User Manual
@anchor{users/messenger doc}@anchor{119}@anchor{users/messenger messenger}@anchor{11a}@anchor{users/messenger using-the-gnunet-messenger}@anchor{11b}
@section Messenger


The GNUnet Messenger subsystem allows decentralized message-based
communication inside of so called rooms. Each room can be hosted by a
variable amount of peers. Every member of a room has the possibility to
host the room on its own peer. A peer allows any amount of members to
join a room. The amount of members in a room is not restricted.

Messages in a room will be distributed between all peers hosting the
room or being internally (in context of the messenger service) connected
to a hosting peer. All received or sent messages will be stored on any
peer locally which is hosting the respective room or is internally
connected to such a hosting peer.

The Messenger service is built on the CADET subsystem to make internal
connections between peers using a reliable and encrypted transmission.
Additionally the service uses a discrete padding to few different sizes.
So kinds of messages and potential content can't be identified by the
size of traffic from any attacker being unable to break the encryption
of the transmission layer.

Another feature is additional end-to-end encryption for selected
messages which uses the public key of another member (the receiver) to
encrypt the message. Therefore it is ensured that only the selected
member can read its content. This will also use additional padding.

@menu
* Current state:: 
* Entering a room:: 
* Opening a room:: 
* Messaging in a room:: 
* Private messaging:: 

@end menu

@node Current state,Entering a room,,Messenger
@anchor{users/messenger current-state}@anchor{11c}@anchor{users/messenger id1}@anchor{11d}
@subsection Current state


Currently there is only a simplistic CLI application available to use
the messenger service. You can use this application with the
@code{gnunet-messenger} command.

This application was designed for testing purposes and it does not
provide full functionality in the current state. It is planned to
replace this CLI application in later stages with a fully featured one
using a client-side library designed for messenger applications.

@node Entering a room,Opening a room,Current state,Messenger
@anchor{users/messenger entering-a-room}@anchor{11e}@anchor{users/messenger id2}@anchor{11f}
@subsection Entering a room


You can enter any room by its ROOMKEY and any PEERIDENTITY of a hosting
peer. Optionally you can provide any IDENTITY which can represent a
local ego by its name. This will automatically select that ego's private
key to sign your messages with.

@example
$ gnunet-messenger [-e IDENTITY] -d PEERIDENTITY -r ROOMKEY
@end example

A PEERIDENTITY gets entered in encoded form. You can get your own peer
ID by using the @code{gnunet-core} command:

@example
$ gnunet-core -i
@end example

A ROOMKEY gets entered in readable text form. The service will then hash
the entered ROOMKEY and use the result as shared secret for transmission
through the CADET submodule. You can also optionally leave out the '-r'
parameter and the ROOMKEY to use the zeroed hash instead.

If no IDENTITY is provided you will not send any name to others, you
will be referred as "anonymous" instead and use the anonymous ego (a
shared key pair known to all peers). If you provide any IDENTITY a
matching ego will be used to sign your messages. If there is no matching
ego you will use the anonymous ego instead. The provided IDENTITY will
be distributed as your name for the service in any case.

@node Opening a room,Messaging in a room,Entering a room,Messenger
@anchor{users/messenger id3}@anchor{120}@anchor{users/messenger opening-a-room}@anchor{121}
@subsection Opening a room


You can open any room in a similar way to entering it. You just have to
leave out the '-d' parameter and the PEERIDENTITY of the hosting peer.

@example
$ gnunet-messenger [-e IDENTITY] -r ROOMKEY
@end example

Providing ROOMKEY and IDENTITY is identical to entering a room. Opening
a room will also make your peer to a host of this room. So others can
enter the room through your peer if they have the required ROOMKEY and
your peer ID.

If you want to use the zeroed hash as shared secret key for the room you
can also leave it out as well:

@example
$ gnunet-messenger
@end example

@node Messaging in a room,Private messaging,Opening a room,Messenger
@anchor{users/messenger id4}@anchor{122}@anchor{users/messenger messaging-in-a-room}@anchor{123}
@subsection Messaging in a room


Once joined a room by entering it or opening it you can write text-based
messages which will be distributed between all internally conntected
peers. All sent messages will be displayed in the same way as received
messages.

This relates to the internal handling of sent and received messages
being mostly identical on application layer. Every handled message will
be represented visually depending on its kind, content and sender. A
sender can usually be identified by the encoded member ID or their name.

@example
[17X37K] * 'anonymous' says: "hey"
@end example

@node Private messaging,,Messaging in a room,Messenger
@anchor{users/messenger id5}@anchor{124}@anchor{users/messenger private-messaging}@anchor{125}
@subsection Private messaging


As referred in the introduction the service allows sending private
messages with additional end-to-end encryption. These messages will be
visually represented by messages of the kind 'PRIVATE' in case they
can't be decrypted with your used private key. Members who can't decrypt
the message can potentially only identify its sender but they can't
identify its receiver. This prevents other members from collecting more
metadata than necessary about you.

@example
[17X37K] ~ message: PRIVATE
@end example

If they can be decrypted they will appear as their secret message
instead but marked visually.

@example
[17X37K] ** 'anonymous' says: "hey"
@end example

Currently you can only activate sending such encrypted text messages
instead of usual text messages by adding the '-p' parameter:

@example
$ gnunet-messenger [-e IDENTITY] -d PEERIDENTITY -r ROOMKEY -p
@end example

Notice that you can only send such encrypted messages to members who use
a private key which is not publicly known via the anonymous ego to ensure
transparency. If any user could decrypt these messages they would not be
private. So as receiver of such messages the IDENTITY is required and it
has to match a local ego.

@node Advanced Configuration,,Messenger,User Manual
@anchor{users/configuration doc}@anchor{126}@anchor{users/configuration advanced-configuration}@anchor{127}
@section Advanced Configuration


@menu
* Config file format:: 
* The Single-User Setup:: 
* The Multi-User Setup:: 
* Access Control for GNUnet:: 
* Configuring the hostlist to bootstrap:: 
* Disable default bootstrap (private network): Disable default bootstrap private network. 
* Manually connecting peers:: 
* Configuration of the HOSTLIST proxy settings:: 
* Configuring your peer to provide a hostlist:: 
* Configuring the datastore:: 
* Configuring the Postgres database:: 
* Reasons to use Postgres:: 
* Reasons not to use Postgres:: 
* Manual setup instructions:: 
* Testing the setup manually:: 
* Configuring the datacache:: 
* Configuring the file-sharing service:: 
* Configuring logging:: 
* Configuring the transport service and plugins:: 
* Configuring the WLAN transport plugin:: 
* Configuring HTTP(S) reverse proxy functionality using Apache or nginx: Configuring HTTP S reverse proxy functionality using Apache or nginx. 
* Blacklisting peers:: 
* Configuration of the HTTP and HTTPS transport plugins:: 
* Configuring the GNUnet VPN:: 
* Bandwidth Configuration:: 
* Configuring NAT:: 
* Peer configuration for distributors (e.g. Operating Systems): Peer configuration for distributors e g Operating Systems. 

@end menu

@node Config file format,The Single-User Setup,,Advanced Configuration
@anchor{users/configuration config-file-format}@anchor{128}@anchor{users/configuration id1}@anchor{129}
@subsection Config file format


In GNUnet realm, all components obey the same pattern to get
configuration values. According to this pattern, once the component has
been installed, the installation deploys default values in
@code{$prefix/share/gnunet/config.d/}, in @code{.conf} files. In order to
override these defaults, the user can write a custom @code{.conf} file and
either pass it to the component at execution time, or name it
@code{gnunet.conf} and place it under @code{$HOME/.config/}.

A config file is a text file containing sections, and each section
contains its values. The right format follows:

@example
[section1]
value1 = string
value2 = 23

[section2]
value21 = string
value22 = /path22
@end example

Throughout any configuration file, it is possible to use @code{$}-prefixed
variables, like @code{$VAR}, especially when they represent filenames in in
the filesystem. It is also possible to provide defaults values for those
variables that are unset, by using the following syntax:

@example
$@{VAR:-default@}
@end example

However, there are two ways a user can set @code{$}-prefixable variables:
(a) by defining them under a @code{[paths]} section

@example
[paths]
GNUNET_DEPLOYMENT_SHARED = $@{HOME@}/shared-data
..
[section-x]
path-x = $@{GNUNET_DEPLOYMENT_SHARED@}/x
@end example

or (b) by setting them in the environment

@example
$ export VAR=/x
@end example

The configuration loader will give precedence to variables set under
@code{[path]}, though.

The utility '@code{gnunet-config}', which gets installed along with
GNUnet, serves to get and set configuration values without directly
editing the @code{.conf} file. The option '@code{-f}' is particularly
useful to resolve filenames, when they use several levels of
@code{$}-expanded variables. See '@code{gnunet-config --help}'.

Note that, in this stage of development, the file
@code{$HOME/.config/gnunet.conf} can contain sections for `all' the
components.
.. _The-Single_002dUser-Setup:

@node The Single-User Setup,The Multi-User Setup,Config file format,Advanced Configuration
@anchor{users/configuration the-single-user-setup}@anchor{12a}
@subsection The Single-User Setup


For the single-user setup, you do not need to do anything special and
can just start the GNUnet background processes using @code{gnunet-arm}. By
default, GNUnet looks in @code{~/.config/gnunet.conf} for a configuration
(or @code{$XDG_CONFIG_HOME/gnunet.conf} if @code{$XDG_CONFIG_HOME} is
defined). If your configuration lives elsewhere, you need to pass the
@code{-c FILENAME} option to all GNUnet commands.

Assuming the configuration file is called @code{~/.config/gnunet.conf}, you
start your peer using the @code{gnunet-arm} command (say as user
@code{gnunet}) using:

@example
gnunet-arm -c ~/.config/gnunet.conf -s
@end example

The "-s" option here is for "start". The command should return
almost instantly. If you want to stop GNUnet, you can use:

@example
gnunet-arm -c ~/.config/gnunet.conf -e
@end example

The "-e" option here is for "end".

Note that this will only start the basic peer, no actual applications
will be available. If you want to start the file-sharing service, use
(after starting GNUnet):

@example
gnunet-arm -c ~/.config/gnunet.conf -i fs
@end example

The "-i fs" option here is for "initialize" the "fs"
(file-sharing) application. You can also selectively kill only
file-sharing support using

@example
gnunet-arm -c ~/.config/gnunet.conf -k fs
@end example

Assuming that you want certain services (like file-sharing) to be always
automatically started whenever you start GNUnet, you can activate them
by setting "IMMEDIATE_START=YES" in the respective section of the
configuration file (for example, "[fs]"). Then GNUnet with
file-sharing support would be started whenever you enter:

@example
gnunet-arm -c ~/.config/gnunet.conf -s
@end example

Alternatively, you can combine the two options:

@example
gnunet-arm -c ~/.config/gnunet.conf -s -i fs
@end example

Using @code{gnunet-arm} is also the preferred method for initializing
GNUnet from @code{init}.

Finally, you should edit your @code{crontab} (using the @code{crontab}
command) and insert a line

@example
@@reboot gnunet-arm -c ~/.config/gnunet.conf -s
@end example

to automatically start your peer whenever your system boots.

@node The Multi-User Setup,Access Control for GNUnet,The Single-User Setup,Advanced Configuration
@anchor{users/configuration the-multi-002duser-setup}@anchor{12b}@anchor{users/configuration the-multi-user-setup}@anchor{12c}
@subsection The Multi-User Setup


This requires you to create a user @code{gnunet} and an additional group
@code{gnunetdns}, prior to running @code{make install} during installation.
Then, you create a configuration file @code{/etc/gnunet.conf} which should
contain the lines:

@example
[arm]
START_SYSTEM_SERVICES = YES
START_USER_SERVICES = NO
@end example

Then, perform the same steps to run GNUnet as in the per-user
configuration, except as user @code{gnunet} (including the @code{crontab}
installation). You may also want to run @code{gnunet-setup} to configure
your peer (databases, etc.). Make sure to pass @code{-c /etc/gnunet.conf}
to all commands. If you run @code{gnunet-setup} as user @code{gnunet}, you
might need to change permissions on @code{/etc/gnunet.conf} so that the
@code{gnunet} user can write to the file (during setup).

Afterwards, you need to perform another setup step for each normal user
account from which you want to access GNUnet. First, grant the normal
user (@code{$USER}) permission to the group gnunet:

@example
# adduser $USER gnunet
@end example

Then, create a configuration file in @code{~/.config/gnunet.conf} for the
$USER with the lines:

@example
[arm]
START_SYSTEM_SERVICES = NO
START_USER_SERVICES = YES
@end example

This will ensure that @code{gnunet-arm} when started by the normal user
will only run services that are per-user, and otherwise rely on the
system-wide services. Note that the normal user may run gnunet-setup,
but the configuration would be ineffective as the system-wide services
will use @code{/etc/gnunet.conf} and ignore options set by individual
users.

Again, each user should then start the peer using @code{gnunet-arm -s} ---
and strongly consider adding logic to start the peer automatically to
their crontab.

Afterwards, you should see two (or more, if you have more than one USER)
@code{gnunet-service-arm} processes running in your system.

@node Access Control for GNUnet,Configuring the hostlist to bootstrap,The Multi-User Setup,Advanced Configuration
@anchor{users/configuration access-control-for-gnunet}@anchor{12d}@anchor{users/configuration id2}@anchor{12e}
@subsection Access Control for GNUnet


This chapter documents how we plan to make access control work within
the GNUnet system for a typical peer. It should be read as a
best-practice installation guide for advanced users and builders of
binary distributions. The recommendations in this guide apply to
POSIX-systems with full support for UNIX domain sockets only.

Note that this is an advanced topic. The discussion presumes a very good
understanding of users, groups and file permissions. Normal users on
hosts with just a single user can just install GNUnet under their own
account (and possibly allow the installer to use SUDO to grant
additional permissions for special GNUnet tools that need additional
rights). The discussion below largely applies to installations where
multiple users share a system and to installations where the best
possible security is paramount.

A typical GNUnet system consists of components that fall into four
categories:


@table @asis

@item User interfaces

User interfaces are not security sensitive and are supposed to be run
and used by normal system users. The GTK GUIs and most command-line
programs fall into this category. Some command-line tools (like
gnunet-transport) should be excluded as they offer low-level access
that normal users should not need.

@item System services and support tools

System services should always run and offer services that can then be
accessed by the normal users. System services do not require special
permissions, but as they are not specific to a particular user, they
probably should not run as a particular user. Also, there should
typically only be one GNUnet peer per host. System services include
the gnunet-service and gnunet-daemon programs; support tools include
command-line programs such as gnunet-arm.

@item Privileged helpers

Some GNUnet components require root rights to open raw sockets or
perform other special operations. These gnunet-helper binaries are
typically installed SUID and run from services or daemons.

@item Critical services

Some GNUnet services (such as the DNS service) can manipulate the
service in deep and possibly highly security sensitive ways. For
example, the DNS service can be used to intercept and alter any DNS
query originating from the local machine. Access to the APIs of these
critical services and their privileged helpers must be tightly
controlled.
@end table

@cartouche
@quotation Todo 
Shorten these subsection titles
@end quotation
@end cartouche

@menu
* Recommendation - Disable access to services via TCP:: 
* Recommendation - Run most services as system user "gnunet":: 
* Recommendation - Control access to services using group "gnunet":: 
* Recommendation - Limit access to certain SUID binaries by group "gnunet":: 
* Recommendation - Limit access to critical gnunet-helper-dns to group "gnunetdns":: 
* Differences between "make install" and these recommendations:: 

@end menu

@node Recommendation - Disable access to services via TCP,Recommendation - Run most services as system user "gnunet",,Access Control for GNUnet
@anchor{users/configuration recommendation-002d-disable-access-to-services-via-tcp}@anchor{12f}@anchor{users/configuration recommendation-disable-access-to-services-via-tcp}@anchor{130}
@subsubsection Recommendation - Disable access to services via TCP


GNUnet services allow two types of access: via TCP socket or via UNIX
domain socket. If the service is available via TCP, access control can
only be implemented by restricting connections to a particular range of
IP addresses. This is acceptable for non-critical services that are
supposed to be available to all users on the local system or local
network. However, as TCP is generally less efficient and it is rarely
the case that a single GNUnet peer is supposed to serve an entire local
network, the default configuration should disable TCP access to all
GNUnet services on systems with support for UNIX domain sockets. Since
GNUnet 0.9.2, configuration files with TCP access disabled should be
generated by default. Users can re-enable TCP access to particular
services simply by specifying a non-zero port number in the section of
the respective service.

@node Recommendation - Run most services as system user "gnunet",Recommendation - Control access to services using group "gnunet",Recommendation - Disable access to services via TCP,Access Control for GNUnet
@anchor{users/configuration recommendation-002d-run-most-services-as-system-user-0022gnunet-0022}@anchor{131}@anchor{users/configuration recommendation-run-most-services-as-system-user-gnunet}@anchor{132}
@subsubsection Recommendation - Run most services as system user "gnunet"


GNUnet's main services should be run as a separate user "gnunet" in a
special group "gnunet". The user "gnunet" should start the peer
using "gnunet-arm -s" during system startup. The home directory for
this user should be @code{/var/lib/gnunet} and the configuration file
should be @code{/etc/gnunet.conf}. Only the @code{gnunet} user should have the
right to access @code{/var/lib/gnunet} (`mode: 700').

@node Recommendation - Control access to services using group "gnunet",Recommendation - Limit access to certain SUID binaries by group "gnunet",Recommendation - Run most services as system user "gnunet",Access Control for GNUnet
@anchor{users/configuration recommendation-002d-control-access-to-services-using-group-0022gnunet-0022}@anchor{133}@anchor{users/configuration recommendation-control-access-to-services-using-group-gnunet}@anchor{134}
@subsubsection Recommendation - Control access to services using group "gnunet"


Users that should be allowed to use the GNUnet peer should be added to
the group "gnunet". Using GNUnet's access control mechanism for UNIX
domain sockets, those services that are considered useful to ordinary
users should be made available by setting "UNIX_MATCH_GID=YES" for
those services. Again, as shipped, GNUnet provides reasonable defaults.
Permissions to access the transport and core subsystems might
additionally be granted without necessarily causing security concerns.
Some services, such as DNS, must NOT be made accessible to the
"gnunet" group (and should thus only be accessible to the "gnunet"
user and services running with this UID).

@node Recommendation - Limit access to certain SUID binaries by group "gnunet",Recommendation - Limit access to critical gnunet-helper-dns to group "gnunetdns",Recommendation - Control access to services using group "gnunet",Access Control for GNUnet
@anchor{users/configuration recommendation-002d-limit-access-to-certain-suid-binaries-by-group-0022gnunet-0022}@anchor{135}@anchor{users/configuration recommendation-limit-access-to-certain-suid-binaries-by-group-gnunet}@anchor{136}
@subsubsection Recommendation - Limit access to certain SUID binaries by group "gnunet"


Most of GNUnet's SUID binaries should be safe even if executed by normal
users. However, it is possible to reduce the risk a little bit more by
making these binaries owned by the group "gnunet" and restricting
their execution to user of the group "gnunet" as well (4750).

@node Recommendation - Limit access to critical gnunet-helper-dns to group "gnunetdns",Differences between "make install" and these recommendations,Recommendation - Limit access to certain SUID binaries by group "gnunet",Access Control for GNUnet
@anchor{users/configuration recommendation-002d-limit-access-to-critical-gnunet-002dhelper-002ddns-to-group-0022gnunetdns-0022}@anchor{137}@anchor{users/configuration recommendation-limit-access-to-critical-gnunet-helper-dns-to-group-gnunetdns}@anchor{138}
@subsubsection Recommendation - Limit access to critical gnunet-helper-dns to group "gnunetdns"


A special group "gnunetdns" should be created for controlling access
to the "gnunet-helper-dns". The binary should then be owned by root
and be in group "gnunetdns" and be installed SUID and only be
group-executable (2750). `Note that the group "gnunetdns" should have
no users in it at all, ever.' The "gnunet-service-dns" program should
be executed by user "gnunet" (via gnunet-service-arm) with the binary
owned by the user "root" and the group "gnunetdns" and be SGID
(2700). This way, `only' "gnunet-service-dns" can change its group
to "gnunetdns" and execute the helper, and the helper can then run as
root (as per SUID). Access to the API offered by "gnunet-service-dns"
is in turn restricted to the user "gnunet" (not the group!), which
means that only "benign" services can manipulate DNS queries using
"gnunet-service-dns".

@node Differences between "make install" and these recommendations,,Recommendation - Limit access to critical gnunet-helper-dns to group "gnunetdns",Access Control for GNUnet
@anchor{users/configuration differences-between-0022make-install-0022-and-these-recommendations}@anchor{139}@anchor{users/configuration differences-between-make-install-and-these-recommendations}@anchor{13a}
@subsubsection Differences between "make install" and these recommendations


The current build system does not set all permissions automatically
based on the recommendations above. In particular, it does not use the
group "gnunet" at all (so setting gnunet-helpers other than the
gnunet-helper-dns to be owned by group "gnunet" must be done
manually). Furthermore, 'make install' will silently fail to set the DNS
binaries to be owned by group "gnunetdns" unless that group already
exists (!). An alternative name for the "gnunetdns" group can be
specified using the @code{--with-gnunetdns=GRPNAME} configure option.

@node Configuring the hostlist to bootstrap,Disable default bootstrap private network,Access Control for GNUnet,Advanced Configuration
@anchor{users/configuration configuring-the-hostlist-to-bootstrap}@anchor{13b}@anchor{users/configuration id4}@anchor{13c}
@subsection Configuring the hostlist to bootstrap


After installing the software you need to get connected to the GNUnet
network. The configuration file included in your download is already
configured to connect you to the GNUnet network. In this section the
relevant configuration settings are explained.

To get an initial connection to the GNUnet network and to get to know
peers already connected to the network you can use the so called
"bootstrap servers". These servers can give you a list of peers
connected to the network. To use these bootstrap servers you have to
configure the hostlist daemon to activate bootstrapping.

To activate bootstrapping, edit the @code{[hostlist]}-section in your
configuration file. You have to set the argument @code{-b} in the options
line:

@example
[hostlist]
OPTIONS = -b
@end example

Additionally you have to specify which server you want to use. The
default bootstrapping server is "@indicateurl{http://v10.gnunet.org/hostlist}". [^]
To set the server you have to edit the line "SERVERS" in the hostlist
section. To use the default server you should set the lines to

@example
SERVERS = http://v10.gnunet.org/hostlist [^]
@end example

To use bootstrapping your configuration file should include these lines:

@example
[hostlist]
OPTIONS = -b
SERVERS = http://v10.gnunet.org/hostlist [^]
@end example

Besides using bootstrap servers you can configure your GNUnet peer to
receive hostlist advertisements. Peers offering hostlists to other peers
can send advertisement messages to peers that connect to them. If you
configure your peer to receive these messages, your peer can download
these lists and connect to the peers included. These lists are
persistent, which means that they are saved to your hard disk regularly
and are loaded during startup.

To activate hostlist learning you have to add the @code{-e} switch to the
@code{OPTIONS} line in the hostlist section:

@example
[hostlist]
OPTIONS = -b -e
@end example

Furthermore you can specify in which file the lists are saved. To save
the lists in the file @code{hostlists.file} just add the line:

@example
HOSTLISTFILE = hostlists.file
@end example

Best practice is to activate both bootstrapping and hostlist learning.
So your configuration file should include these lines:

@example
[hostlist]
OPTIONS = -b -e
HTTPPORT = 8080
SERVERS = http://v10.gnunet.org/hostlist [^]
HOSTLISTFILE = $SERVICEHOME/hostlists.file
@end example

@node Disable default bootstrap private network,Manually connecting peers,Configuring the hostlist to bootstrap,Advanced Configuration
@anchor{users/configuration disable-default-bootstrap}@anchor{13d}@anchor{users/configuration disable-default-bootstrap-private-network}@anchor{13e}
@subsection Disable default bootstrap (private network)


A public node will, by default, connect to a gnunet.org peer to learn
of other peers to bootstrap the network.

To avoid this behavior, either:


@itemize -

@item 
before build, remove the peer entry in @code{$REPO/contrib/hellos}

@item 
after build, remove the peer entry in @code{$PREFIX/share/gnunet/hellos}
@end itemize

Conversely, any public keys added to the same directories will make the
node `always' make explicit connections to those corresponding peers.

The use of the HELLOs in this folder can be controlled with the configuration
setting @code{USE_INCLUDED_HELLOS} of the @code{peerstore} service:

@example
$ gnunet-config -s peerstore -o USE_INCLUDED_HELLOS
@end example

Note, however, that once the included HELLOs have been parsed, the @code{peerstore}
will cache them locally in its databse. To purge included HELLOs in this case,
the database will have to be deleted.

Unless you want to establish a private network, you should not have to touch
this option.

@node Manually connecting peers,Configuration of the HOSTLIST proxy settings,Disable default bootstrap private network,Advanced Configuration
@anchor{users/configuration id5}@anchor{13f}@anchor{users/configuration manually-connecting-peers}@anchor{140}
@subsection Manually connecting peers


A gnunet node will learn peers to connect to from hostlist servers and/or
gossip from connected peers. It will however only connect to a selection
of peers on the network.

If you wish to connect to a specific peer apart from the automatically
negotiated connections, you can use the @code{hello} URI of the peer. The
URI is returned by the following command to `peer to be connected to':

@example
$ gnunet-hello --export-hello
@end example

The URI output is passed to the @code{gnunet-hello} command of `peer
that is connecting':

@example
$ echo "gnunet://hello/..." | gnunet-hello --import-hello
@end example

@node Configuration of the HOSTLIST proxy settings,Configuring your peer to provide a hostlist,Manually connecting peers,Advanced Configuration
@anchor{users/configuration configuration-of-the-hostlist-proxy-settings}@anchor{141}@anchor{users/configuration configuration-of-the-hostlist-proxy-settings-cli}@anchor{142}
@subsection Configuration of the HOSTLIST proxy settings


The hostlist client can be configured to use a proxy to connect to the
hostlist server.

The hostlist client supports the following proxy types at the moment:


@itemize -

@item 
HTTP and HTTP 1.0 only proxy

@item 
SOCKS 4/4a/5/5 with hostname
@end itemize

In addition authentication at the proxy with username and password can
be configured.

To provide these options directly in the configuration, you can enter
the following settings in the @code{[hostlist]} section of the
configuration:

@example
# Type of proxy server,
# Valid values: HTTP, HTTP_1_0, SOCKS4, SOCKS5, SOCKS4A, SOCKS5_HOSTNAME
# Default: HTTP
# PROXY_TYPE = HTTP

# Hostname or IP of proxy server
# PROXY =
# User name for proxy server
# PROXY_USERNAME =
# User password for proxy server
# PROXY_PASSWORD =
@end example

@node Configuring your peer to provide a hostlist,Configuring the datastore,Configuration of the HOSTLIST proxy settings,Advanced Configuration
@anchor{users/configuration configuring-your-peer-to-provide-a-hostlist}@anchor{143}@anchor{users/configuration id6}@anchor{144}
@subsection Configuring your peer to provide a hostlist


If you operate a peer permanently connected to GNUnet you can configure
your peer to act as a hostlist server, providing other peers the list of
peers known to him.

Your server can act as a bootstrap server and peers needing to obtain a
list of peers can contact it to download this list. To download this
hostlist the peer uses HTTP. For this reason you have to build your peer
with libgnurl (or libcurl) and microhttpd support.

To configure your peer to act as a bootstrap server you have to add the
@code{-p} option to @code{OPTIONS} in the @code{[hostlist]} section of your
configuration file. Besides that you have to specify a port number for
the http server. In conclusion you have to add the following lines:

@example
[hostlist]
HTTPPORT = 12980
OPTIONS = -p
@end example

If your peer acts as a bootstrap server other peers should know about
that. You can advertise the hostlist your are providing to other peers.
Peers connecting to your peer will get a message containing an
advertisement for your hostlist and the URL where it can be downloaded.
If this peer is in learning mode, it will test the hostlist and, in the
case it can obtain the list successfully, it will save it for
bootstrapping.

To activate hostlist advertisement on your peer, you have to set the
following lines in your configuration file:

@example
[hostlist]
EXTERNAL_DNS_NAME = example.org
HTTPPORT = 12981
OPTIONS = -p -a
@end example

With this configuration your peer will a act as a bootstrap server and
advertise this hostlist to other peers connecting to it. The URL used to
download the list will be @code{http://example.org:12981/}.

Please notice:


@itemize -

@item 
The hostlist is `not' human readable, so you should not try to
download it using your webbrowser. Just point your GNUnet peer to the
address!

@item 
Advertising without providing a hostlist does not make sense and will
not work.
@end itemize

@node Configuring the datastore,Configuring the Postgres database,Configuring your peer to provide a hostlist,Advanced Configuration
@anchor{users/configuration configuring-the-datastore}@anchor{145}@anchor{users/configuration id7}@anchor{146}
@subsection Configuring the datastore


The datastore is what GNUnet uses for long-term storage of file-sharing
data. Note that long-term does not mean 'forever' since content does
have an expiration date, and of course storage space is finite (and
hence sometimes content may have to be discarded).

Use the @code{QUOTA} option to specify how many bytes of storage space you
are willing to dedicate to GNUnet.

In addition to specifying the maximum space GNUnet is allowed to use for
the datastore, you need to specify which database GNUnet should use to
do so. Currently, you have the choice between sqlite and
Postgres.

@node Configuring the Postgres database,Reasons to use Postgres,Configuring the datastore,Advanced Configuration
@anchor{users/configuration configuring-the-postgres-database}@anchor{147}@anchor{users/configuration id8}@anchor{148}
@subsection Configuring the Postgres database


This text describes how to setup the Postgres database for GNUnet.

This Postgres plugin was developed for Postgres 8.3 but might work for
earlier versions as well.

@node Reasons to use Postgres,Reasons not to use Postgres,Configuring the Postgres database,Advanced Configuration
@anchor{users/configuration id9}@anchor{149}@anchor{users/configuration reasons-to-use-postgres}@anchor{14a}
@subsection Reasons to use Postgres



@itemize -

@item 
Easier to setup than MySQL

@item 
Real database
@end itemize

@node Reasons not to use Postgres,Manual setup instructions,Reasons to use Postgres,Advanced Configuration
@anchor{users/configuration id10}@anchor{14b}@anchor{users/configuration reasons-not-to-use-postgres}@anchor{14c}
@subsection Reasons not to use Postgres



@itemize -

@item 
Quite slow

@item 
Still some manual setup required
@end itemize

@node Manual setup instructions,Testing the setup manually,Reasons not to use Postgres,Advanced Configuration
@anchor{users/configuration id11}@anchor{14d}@anchor{users/configuration manual-setup-instructions}@anchor{14e}
@subsection Manual setup instructions



@itemize -

@item 
In @code{gnunet.conf} set in section @code{DATASTORE} the value for
@code{DATABASE} to @code{postgres}.

@item 
Access Postgres to create a user:


@table @asis

@item with Postgres 8.x, use:

@example
# su - postgres
$ createuser
@end example

and enter the name of the user running GNUnet for the role
interactively. Then, when prompted, do not set it to superuser,
allow the creation of databases, and do not allow the creation of
new roles.

@item with Postgres 9.x, use:

@example
# su - postgres
$ createuser -d $GNUNET_USER
@end example

where $GNUNET_USER is the name of the user running GNUnet.
@end table

@item 
As that user (so typically as user "gnunet"), create a database (or
two):

@example
$ createdb gnunet
# this way you can run "make check"
$ createdb gnunetcheck
@end example
@end itemize

Now you should be able to start @code{gnunet-arm}.

@node Testing the setup manually,Configuring the datacache,Manual setup instructions,Advanced Configuration
@anchor{users/configuration id12}@anchor{14f}@anchor{users/configuration testing-the-setup-manually}@anchor{150}
@subsection Testing the setup manually


You may want to try if the database connection works. First, again login
as the user who will run @code{gnunet-arm}. Then use:

@example
$ psql gnunet # or gnunetcheck
gnunet=> \dt
@end example

If, after you have started @code{gnunet-arm} at least once, you get a
@code{gn090} table here, it probably works.

@node Configuring the datacache,Configuring the file-sharing service,Testing the setup manually,Advanced Configuration
@anchor{users/configuration configuring-the-datacache}@anchor{151}@anchor{users/configuration id13}@anchor{152}
@subsection Configuring the datacache


The datacache is what GNUnet uses for storing temporary data. This data
is expected to be wiped completely each time GNUnet is restarted (or the
system is rebooted).

You need to specify how many bytes GNUnet is allowed to use for the
datacache using the @code{QUOTA} option in the section @code{[dhtcache]}.
Furthermore, you need to specify which database backend should be used
to store the data. Currently, you have the choice between sqLite, MySQL
and Postgres.

@node Configuring the file-sharing service,Configuring logging,Configuring the datacache,Advanced Configuration
@anchor{users/configuration configuring-the-file-002dsharing-service}@anchor{153}@anchor{users/configuration configuring-the-file-sharing-service}@anchor{154}
@subsection Configuring the file-sharing service


In order to use GNUnet for file-sharing, you first need to make sure
that the file-sharing service is loaded. This is done by setting the
@code{START_ON_DEMAND} option in section @code{[fs]} to "YES".
Alternatively, you can run

@example
$ gnunet-arm -i fs
@end example

to start the file-sharing service by hand.

Except for configuring the database and the datacache the only important
option for file-sharing is content migration.

Content migration allows your peer to cache content from other peers as
well as send out content stored on your system without explicit
requests. This content replication has positive and negative impacts on
both system performance and privacy.

FIXME: discuss the trade-offs. Here is some older text about it...

Setting this option to YES allows gnunetd to migrate data to the local
machine. Setting this option to YES is highly recommended for
efficiency. Its also the default. If you set this value to YES, GNUnet
will store content on your machine that you cannot decrypt. While this
may protect you from liability if the judge is sane, it may not (IANAL).
If you put illegal content on your machine yourself, setting this option
to YES will probably increase your chances to get away with it since you
can plausibly deny that you inserted the content. Note that in either
case, your anonymity would have to be broken first (which may be
possible depending on the size of the GNUnet network and the strength of
the adversary).

@node Configuring logging,Configuring the transport service and plugins,Configuring the file-sharing service,Advanced Configuration
@anchor{users/configuration configuring-logging}@anchor{155}@anchor{users/configuration id14}@anchor{156}
@subsection Configuring logging


Since version 0.9.0, logging in GNUnet is controlled via the @code{-L} and
@code{-l} options. Using @code{-L}, a log level can be specified. With log
level @code{ERROR} only serious errors are logged. The default log level is
@code{WARNING} which causes anything of concern to be logged. Log level
@code{INFO} can be used to log anything that might be interesting
information whereas @code{DEBUG} can be used by developers to log debugging
messages (but you need to run @code{meson setup} with
@code{-Dlogging=verbose} to get them compiled). The @code{-l} option is
used to specify the log file.

Since most GNUnet services are managed by @code{gnunet-arm}, using the
@code{-l} or @code{-L} options directly is not possible. Instead, they can be
specified using the @code{OPTIONS} configuration value in the respective
section for the respective service. In order to enable logging globally
without editing the @code{OPTIONS} values for each service, @code{gnunet-arm}
supports a @code{GLOBAL_POSTFIX} option. The value specified here is given
as an extra option to all services for which the configuration does
contain a service-specific @code{OPTIONS} field.

@code{GLOBAL_POSTFIX} can contain the special sequence "@{@}" which is
replaced by the name of the service that is being started. Furthermore,
@code{GLOBAL_POSTFIX} is special in that sequences starting with "$"
anywhere in the string are expanded (according to options in @code{PATHS});
this expansion otherwise is only happening for filenames and then the
"$" must be the first character in the option. Both of these
restrictions do not apply to @code{GLOBAL_POSTFIX}. Note that specifying
@code{%} anywhere in the @code{GLOBAL_POSTFIX} disables both of these
features.

In summary, in order to get all services to log at level @code{INFO} to
log-files called @code{SERVICENAME-logs}, the following global prefix
should be used:

@example
GLOBAL_POSTFIX = -l $SERVICEHOME/@{@}-logs -L INFO
@end example

@node Configuring the transport service and plugins,Configuring the WLAN transport plugin,Configuring logging,Advanced Configuration
@anchor{users/configuration configuring-the-transport-service-and-plugins}@anchor{157}@anchor{users/configuration id15}@anchor{158}
@subsection Configuring the transport service and plugins


The transport service in GNUnet is responsible to maintain basic
connectivity to other peers. Besides initiating and keeping connections
alive it is also responsible for address validation.

The GNUnet transport supports more than one transport protocol. These
protocols are configured together with the transport service.

The configuration section for the transport service itself is quite
similar to all the other services

@example
START_ON_DEMAND = YES
@@UNIXONLY@@ PORT = 2091
HOSTNAME = localhost
HOME = $SERVICEHOME
CONFIG = $DEFAULTCONFIG
BINARY = gnunet-service-transport
#PREFIX = valgrind
NEIGHBOUR_LIMIT = 50
ACCEPT_FROM = 127.0.0.1;
ACCEPT_FROM6 = ::1;
PLUGINS = tcp udp
UNIXPATH = /tmp/gnunet-service-transport.sock
@end example

Different are the settings for the plugins to load @code{PLUGINS}. The
first setting specifies which transport plugins to load.


@itemize -

@item 
transport-unix A plugin for local only communication with UNIX domain
sockets. Used for testing and available on unix systems only. Just
set the port

@example
[transport-unix]
PORT = 22086
TESTING_IGNORE_KEYS = ACCEPT_FROM;
@end example

@item 
transport-tcp A plugin for communication with TCP. Set port to 0 for
client mode with outbound only connections

@example
[transport-tcp]
# Use 0 to ONLY advertise as a peer behind NAT (no port binding)
PORT = 2086
ADVERTISED_PORT = 2086
TESTING_IGNORE_KEYS = ACCEPT_FROM;
# Maximum number of open TCP connections allowed
MAX_CONNECTIONS = 128
@end example

@item 
transport-udp A plugin for communication with UDP. Supports peer
discovery using broadcasts.

@example
[transport-udp]
PORT = 2086
BROADCAST = YES
BROADCAST_INTERVAL = 30 s
MAX_BPS = 1000000
TESTING_IGNORE_KEYS = ACCEPT_FROM;
@end example

@item 
transport-http HTTP and HTTPS support is split in two part: a client
plugin initiating outbound connections and a server part accepting
connections from the client. The client plugin just takes the maximum
number of connections as an argument.

@example
[transport-http_client]
MAX_CONNECTIONS = 128
TESTING_IGNORE_KEYS = ACCEPT_FROM;
@end example

@example
[transport-https_client]
MAX_CONNECTIONS = 128
TESTING_IGNORE_KEYS = ACCEPT_FROM;
@end example

The server has a port configured and the maximum number of
connections. The HTTPS part has two files with the certificate key
and the certificate file.

The server plugin supports reverse proxies, so a external hostname
can be set using the @code{EXTERNAL_HOSTNAME} setting. The webserver
under this address should forward the request to the peer and the
configure port.

@example
[transport-http_server]
EXTERNAL_HOSTNAME = fulcrum.net.in.tum.de/gnunet
PORT = 1080
MAX_CONNECTIONS = 128
TESTING_IGNORE_KEYS = ACCEPT_FROM;
@end example

@example
[transport-https_server]
PORT = 4433
CRYPTO_INIT = NORMAL
KEY_FILE = https.key
CERT_FILE = https.cert
MAX_CONNECTIONS = 128
TESTING_IGNORE_KEYS = ACCEPT_FROM;
@end example

@item 
transport-wlan

The next section describes how to setup the WLAN plugin, so here only
the settings. Just specify the interface to use:

@example
[transport-wlan]
# Name of the interface in monitor mode (typically monX)
INTERFACE = mon0
# Real hardware, no testing
TESTMODE = 0
TESTING_IGNORE_KEYS = ACCEPT_FROM;
@end example
@end itemize

@node Configuring the WLAN transport plugin,Configuring HTTP S reverse proxy functionality using Apache or nginx,Configuring the transport service and plugins,Advanced Configuration
@anchor{users/configuration configuring-the-wlan-transport-plugin}@anchor{159}@anchor{users/configuration id16}@anchor{15a}
@subsection Configuring the WLAN transport plugin


The wlan transport plugin enables GNUnet to send and to receive data on
a wlan interface. It has not to be connected to a wlan network as long
as sender and receiver are on the same channel. This enables you to get
connection to GNUnet where no internet access is possible, for example
during catastrophes or when censorship cuts you off from the internet.

@menu
* Requirements for the WLAN plugin:: 
* Configuration:: 
* Before starting GNUnet:: 
* Limitations and known bugs:: 

@end menu

@node Requirements for the WLAN plugin,Configuration,,Configuring the WLAN transport plugin
@anchor{users/configuration id17}@anchor{15b}@anchor{users/configuration requirements-for-the-wlan-plugin}@anchor{15c}
@subsubsection Requirements for the WLAN plugin



@itemize -

@item 
wlan network card with monitor support and packet injection (see
aircrack-ng.org@footnote{http://www.aircrack-ng.org/})

@item 
Linux kernel with mac80211 stack, introduced in 2.6.22, tested with
2.6.35 and 2.6.38

@item 
Wlantools to create the a monitor interface, tested with airmon-ng of
the aircrack-ng package
@end itemize

@node Configuration,Before starting GNUnet,Requirements for the WLAN plugin,Configuring the WLAN transport plugin
@anchor{users/configuration configuration}@anchor{15d}@anchor{users/configuration id18}@anchor{15e}
@subsubsection Configuration


There are the following options for the wlan plugin (they should be like
this in your default config file, you only need to adjust them if the
values are incorrect for your system)

@example
# section for the wlan transport plugin
[transport-wlan]
# interface to use, more information in the
# "Before starting GNUnet" section of the handbook.
INTERFACE = mon0
# testmode for developers:
# 0 use wlan interface,
#1 or 2 use loopback driver for tests 1 = server, 2 = client
TESTMODE = 0
@end example

@node Before starting GNUnet,Limitations and known bugs,Configuration,Configuring the WLAN transport plugin
@anchor{users/configuration before-starting-gnunet}@anchor{15f}@anchor{users/configuration id19}@anchor{160}
@subsubsection Before starting GNUnet


Before starting GNUnet, you have to make sure that your wlan interface
is in monitor mode. One way to put the wlan interface into monitor mode
(if your interface name is wlan0) is by executing:

@example
sudo airmon-ng start wlan0
@end example

Here is an example what the result should look like:

@example
Interface Chipset Driver
wlan0 Intel 4965 a/b/g/n iwl4965 - [phy0]
(monitor mode enabled on mon0)
@end example

The monitor interface is mon0 is the one that you have to put into the
configuration file.

@node Limitations and known bugs,,Before starting GNUnet,Configuring the WLAN transport plugin
@anchor{users/configuration id20}@anchor{161}@anchor{users/configuration limitations-and-known-bugs}@anchor{162}
@subsubsection Limitations and known bugs


Wlan speed is at the maximum of 1 Mbit/s because support for choosing
the wlan speed with packet injection was removed in newer kernels.
Please pester the kernel developers about fixing this.

The interface channel depends on the wlan network that the card is
connected to. If no connection has been made since the start of the
computer, it is usually the first channel of the card. Peers will only
find each other and communicate if they are on the same channel.
Channels must be set manually, e.g. by using:

@example
iwconfig wlan0 channel 1
@end example

@node Configuring HTTP S reverse proxy functionality using Apache or nginx,Blacklisting peers,Configuring the WLAN transport plugin,Advanced Configuration
@anchor{users/configuration configuring-http-0028s-0029-reverse-proxy-functionality-using-apache-or-nginx}@anchor{163}@anchor{users/configuration configuring-http-s-reverse-proxy-functionality-using-apache-or-nginx}@anchor{164}
@subsection Configuring HTTP(S) reverse proxy functionality using Apache or nginx


The HTTP plugin supports data transfer using reverse proxies. A reverse
proxy forwards the HTTP request he receives with a certain URL to
another webserver, here a GNUnet peer.

So if you have a running Apache or nginx webserver you can configure it
to be a GNUnet reverse proxy. Especially if you have a well-known
website this improves censorship resistance since it looks as normal
surfing behaviour.

To do so, you have to do two things:


@itemize -

@item 
Configure your webserver to forward the GNUnet HTTP traffic

@item 
Configure your GNUnet peer to announce the respective address
@end itemize

As an example we want to use GNUnet peer running:


@itemize -

@item 
HTTP server plugin on @code{gnunet.foo.org:1080}

@item 
HTTPS server plugin on @code{gnunet.foo.org:4433}

@item 
A apache or nginx webserver on
http://www.foo.org:80/@footnote{http://www.foo.org/}

@item 
A apache or nginx webserver on @indicateurl{https://www.foo.org:443/}
@end itemize

And we want the webserver to accept GNUnet traffic under
@code{http://www.foo.org/bar/}. The required steps are described here:

@menu
* Reverse Proxy - Configure your Apache2 HTTP webserver:: 
* Reverse Proxy - Configure your Apache2 HTTPS webserver:: 
* Reverse Proxy - Configure your nginx HTTPS webserver:: 
* Reverse Proxy - Configure your nginx HTTP webserver:: 
* Reverse Proxy - Configure your GNUnet peer:: 

@end menu

@node Reverse Proxy - Configure your Apache2 HTTP webserver,Reverse Proxy - Configure your Apache2 HTTPS webserver,,Configuring HTTP S reverse proxy functionality using Apache or nginx
@anchor{users/configuration reverse-proxy-002d-configure-your-apache2-http-webserver}@anchor{165}@anchor{users/configuration reverse-proxy-configure-your-apache2-http-webserver}@anchor{166}
@subsubsection Reverse Proxy - Configure your Apache2 HTTP webserver


First of all you need mod_proxy installed.

Edit your webserver configuration. Edit @code{/etc/apache2/apache2.conf} or
the site-specific configuration file.

In the respective @code{server config},@code{virtual host} or @code{directory}
section add the following lines:

@example
ProxyTimeout 300
ProxyRequests Off
<Location /bar/ >
ProxyPass http://gnunet.foo.org:1080/
ProxyPassReverse http://gnunet.foo.org:1080/
</Location>
@end example

@node Reverse Proxy - Configure your Apache2 HTTPS webserver,Reverse Proxy - Configure your nginx HTTPS webserver,Reverse Proxy - Configure your Apache2 HTTP webserver,Configuring HTTP S reverse proxy functionality using Apache or nginx
@anchor{users/configuration reverse-proxy-002d-configure-your-apache2-https-webserver}@anchor{167}@anchor{users/configuration reverse-proxy-configure-your-apache2-https-webserver}@anchor{168}
@subsubsection Reverse Proxy - Configure your Apache2 HTTPS webserver


We assume that you already have an HTTPS server running, if not please
check how to configure a HTTPS host. An uncomplicated to use example is
the example configuration file for Apache2/HTTPD provided in
@code{apache2/sites-available/default-ssl}.

In the respective HTTPS @code{server config},@code{virtual host} or
@code{directory} section add the following lines:

@example
SSLProxyEngine On
ProxyTimeout 300
ProxyRequests Off
<Location /bar/ >
ProxyPass https://gnunet.foo.org:4433/
ProxyPassReverse https://gnunet.foo.org:4433/
</Location>
@end example

More information about the apache mod_proxy configuration can be found
in the Apache documentation@footnote{http://httpd.apache.org/docs/2.2/mod/mod_proxy.html#proxypass}.

@node Reverse Proxy - Configure your nginx HTTPS webserver,Reverse Proxy - Configure your nginx HTTP webserver,Reverse Proxy - Configure your Apache2 HTTPS webserver,Configuring HTTP S reverse proxy functionality using Apache or nginx
@anchor{users/configuration reverse-proxy-002d-configure-your-nginx-https-webserver}@anchor{169}@anchor{users/configuration reverse-proxy-configure-your-nginx-https-webserver}@anchor{16a}
@subsubsection Reverse Proxy - Configure your nginx HTTPS webserver


Since nginx does not support chunked encoding, you first of all have to
install the @code{chunkin}
module@footnote{http://wiki.nginx.org/HttpChunkinModule}.

To enable chunkin add:

@example
chunkin on;
error_page 411 = @@my_411_error;
location @@my_411_error @{
chunkin_resume;
@}
@end example

Edit your webserver configuration. Edit @code{/etc/nginx/nginx.conf} or the
site-specific configuration file.

In the @code{server} section add:

@example
location /bar/ @{
proxy_pass http://gnunet.foo.org:1080/;
proxy_buffering off;
proxy_connect_timeout 5; # more than http_server
proxy_read_timeout 350; # 60 default, 300s is GNUnet's idle timeout
proxy_http_version 1.1; # 1.0 default
proxy_next_upstream error timeout invalid_header http_500 http_503 http_502 http_504;
@}
@end example

@node Reverse Proxy - Configure your nginx HTTP webserver,Reverse Proxy - Configure your GNUnet peer,Reverse Proxy - Configure your nginx HTTPS webserver,Configuring HTTP S reverse proxy functionality using Apache or nginx
@anchor{users/configuration reverse-proxy-002d-configure-your-nginx-http-webserver}@anchor{16b}@anchor{users/configuration reverse-proxy-configure-your-nginx-http-webserver}@anchor{16c}
@subsubsection Reverse Proxy - Configure your nginx HTTP webserver


Edit your webserver configuration. Edit @code{/etc/nginx/nginx.conf} or the
site-specific configuration file.

In the @code{server} section add:

@example
ssl_session_timeout 6m;
location /bar/
@{
proxy_pass https://gnunet.foo.org:4433/;
proxy_buffering off;
proxy_connect_timeout 5; # more than http_server
proxy_read_timeout 350; # 60 default, 300s is GNUnet's idle timeout
proxy_http_version 1.1; # 1.0 default
proxy_next_upstream error timeout invalid_header http_500 http_503 http_502 http_504;
@}
@end example

@node Reverse Proxy - Configure your GNUnet peer,,Reverse Proxy - Configure your nginx HTTP webserver,Configuring HTTP S reverse proxy functionality using Apache or nginx
@anchor{users/configuration reverse-proxy-002d-configure-your-gnunet-peer}@anchor{16d}@anchor{users/configuration reverse-proxy-configure-your-gnunet-peer}@anchor{16e}
@subsubsection Reverse Proxy - Configure your GNUnet peer


To have your GNUnet peer announce the address, you have to specify the
@code{EXTERNAL_HOSTNAME} option in the @code{[transport-http_server]} section:

@example
[transport-http_server]
EXTERNAL_HOSTNAME = http://www.foo.org/bar/
@end example

and/or @code{[transport-https_server]} section:

@example
[transport-https_server]
EXTERNAL_HOSTNAME = https://www.foo.org/bar/
@end example

Now restart your webserver and your peer...

@node Blacklisting peers,Configuration of the HTTP and HTTPS transport plugins,Configuring HTTP S reverse proxy functionality using Apache or nginx,Advanced Configuration
@anchor{users/configuration blacklisting-peers}@anchor{16f}@anchor{users/configuration id21}@anchor{170}
@subsection Blacklisting peers


Transport service supports to deny connecting to a specific peer of to a
specific peer with a specific transport plugin using the blacklisting
component of transport service. With blacklisting it is possible to deny
connections to specific peers of to use a specific plugin to a specific
peer. Peers can be blacklisted using the configuration or a blacklist
client can be asked.

To blacklist peers using the configuration you have to add a section to
your configuration containing the peer id of the peer to blacklist and
the plugin if required.

Examples:

To blacklist connections to P565... on peer AG2P... using tcp add:

@cartouche
@quotation Todo 
too long?
@end quotation
@end cartouche

@cartouche
@quotation Todo 
verify whether these still produce errors in pdf output
@end quotation
@end cartouche

@example
[transport-blacklist AG2PHES1BARB9IJCPAMJTFPVJ5V3A72S3F2A8SBUB8DAQ2V0O3V8G6G2JU56FHGFOHMQVKBSQFV98TCGTC3RJ1NINP82G0RC00N1520]
P565723JO1C2HSN6J29TAQ22MN6CI8HTMUU55T0FUQG4CMDGGEQ8UCNBKUMB94GC8R9G4FB2SF9LDOBAJ6AMINBP4JHHDD6L7VD801G = tcp
@end example

To blacklist connections to P565... on peer AG2P... using all plugins
add:

@example
[transport-blacklist-AG2PHES1BARB9IJCPAMJTFPVJ5V3A72S3F2A8SBUB8DAQ2V0O3V8G6G2JU56FHGFOHMQVKBSQFV98TCGTC3RJ1NINP82G0RC00N1520]
P565723JO1C2HSN6J29TAQ22MN6CI8HTMUU55T0FUQG4CMDGGEQ8UCNBKUMB94GC8R9G4FB2SF9LDOBAJ6AMINBP4JHHDD6L7VD801G =
@end example

You can also add a blacklist client using the blacklist API. On a
blacklist check, blacklisting first checks internally if the peer is
blacklisted and if not, it asks the blacklisting clients. Clients are
asked if it is OK to connect to a peer ID, the plugin is omitted.

On blacklist check for (peer, plugin)


@itemize -

@item 
Do we have a local blacklist entry for this peer and this plugin?

@item 
YES: disallow connection

@item 
Do we have a local blacklist entry for this peer and all plugins?

@item 
YES: disallow connection

@item 
Does one of the clients disallow?

@item 
YES: disallow connection
@end itemize

@node Configuration of the HTTP and HTTPS transport plugins,Configuring the GNUnet VPN,Blacklisting peers,Advanced Configuration
@anchor{users/configuration configuration-of-the-http-and-https-transport-plugins}@anchor{171}@anchor{users/configuration configuration-of-the-http-and-https-transport-plugins-cli}@anchor{172}
@subsection Configuration of the HTTP and HTTPS transport plugins


The client parts of the http and https transport plugins can be
configured to use a proxy to connect to the hostlist server.

Both the HTTP and HTTPS clients support the following proxy types at the
moment:


@itemize -

@item 
HTTP 1.1 proxy

@item 
SOCKS 4/4a/5/5 with hostname
@end itemize

In addition authentication at the proxy with username and password can
be configured.

To configure these options directly in the configuration, you can
configure the following settings in the @code{[transport-http_client]} and
@code{[transport-https_client]} section of the configuration:

@example
# Type of proxy server,
# Valid values: HTTP, SOCKS4, SOCKS5, SOCKS4A, SOCKS5_HOSTNAME
# Default: HTTP
# PROXY_TYPE = HTTP

# Hostname or IP of proxy server
# PROXY =
# User name for proxy server
# PROXY_USERNAME =
# User password for proxy server
# PROXY_PASSWORD =
@end example

@node Configuring the GNUnet VPN,Bandwidth Configuration,Configuration of the HTTP and HTTPS transport plugins,Advanced Configuration
@anchor{users/configuration configuring-the-gnunet-vpn}@anchor{173}@anchor{users/configuration id24}@anchor{174}
@subsection Configuring the GNUnet VPN


Before configuring the GNUnet VPN, please make sure that system-wide DNS
interception is configured properly as described in the section on the
GNUnet DNS setup. see @ref{175,,Configuring the GNU Name System}, if you haven't done so
already.

The default options for the GNUnet VPN are usually sufficient to use
GNUnet as a Layer 2 for your Internet connection. However, what you
always have to specify is which IP protocol you want to tunnel: IPv4,
IPv6 or both. Furthermore, if you tunnel both, you most likely should
also tunnel all of your DNS requests. You theoretically can tunnel
"only" your DNS traffic, but that usually makes little sense.

The other options as shown on the gnunet-setup tool are:

@menu
* IPv4 address for interface:: 
* IPv6 address for interface:: 
* Configuring the GNUnet VPN DNS:: 
* Configuring the GNUnet VPN Exit Service:: 
* IP Address of external DNS resolver:: 
* IPv4 address for Exit interface:: 
* IPv6 address for Exit interface:: 

@end menu

@node IPv4 address for interface,IPv6 address for interface,,Configuring the GNUnet VPN
@anchor{users/configuration id25}@anchor{176}@anchor{users/configuration ipv4-address-for-interface}@anchor{177}
@subsubsection IPv4 address for interface


This is the IPv4 address the VPN interface will get. You should pick a
'private' IPv4 network that is not yet in use for you system. For
example, if you use @code{10.0.0.1/255.255.0.0} already, you might use
@code{10.1.0.1/255.255.0.0}. If you use @code{10.0.0.1/255.0.0.0} already,
then you might use @code{192.168.0.1/255.255.0.0}. If your system is not in
a private IP-network, using any of the above will work fine. You should
try to make the mask of the address big enough (@code{255.255.0.0} or, even
better, @code{255.0.0.0}) to allow more mappings of remote IP Addresses
into this range. However, even a @code{255.255.255.0} mask will suffice for
most users.

@node IPv6 address for interface,Configuring the GNUnet VPN DNS,IPv4 address for interface,Configuring the GNUnet VPN
@anchor{users/configuration id26}@anchor{178}@anchor{users/configuration ipv6-address-for-interface}@anchor{179}
@subsubsection IPv6 address for interface


The IPv6 address the VPN interface will get. Here you can specify any
non-link-local address (the address should not begin with @code{fe80:}). A
subnet Unique Local Unicast (@code{fd00::/8} prefix) that you are currently
not using would be a good choice.

@node Configuring the GNUnet VPN DNS,Configuring the GNUnet VPN Exit Service,IPv6 address for interface,Configuring the GNUnet VPN
@anchor{users/configuration configuring-the-gnunet-vpn-dns}@anchor{17a}@anchor{users/configuration id27}@anchor{17b}
@subsubsection Configuring the GNUnet VPN DNS


To resolve names for remote nodes, activate the DNS exit option.

@node Configuring the GNUnet VPN Exit Service,IP Address of external DNS resolver,Configuring the GNUnet VPN DNS,Configuring the GNUnet VPN
@anchor{users/configuration configuring-the-gnunet-vpn-exit-service}@anchor{17c}@anchor{users/configuration id28}@anchor{17d}
@subsubsection Configuring the GNUnet VPN Exit Service


If you want to allow other users to share your Internet connection (yes,
this may be dangerous, just as running a Tor exit node) or want to
provide access to services on your host (this should be less dangerous,
as long as those services are secure), you have to enable the GNUnet
exit daemon.

You then get to specify which exit functions you want to provide. By
enabling the exit daemon, you will always automatically provide exit
functions for manually configured local services (this component of the
system is under development and not documented further at this time). As
for those services you explicitly specify the target IP address and
port, there is no significant security risk in doing so.

Furthermore, you can serve as a DNS, IPv4 or IPv6 exit to the Internet.
Being a DNS exit is usually pretty harmless. However, enabling IPv4 or
IPv6-exit without further precautions may enable adversaries to access
your local network, send spam, attack other systems from your Internet
connection and do other mischiefs that will appear to come from your
machine. This may or may not get you into legal trouble. If you want to
allow IPv4 or IPv6-exit functionality, you should strongly consider
adding additional firewall rules manually to protect your local network
and to restrict outgoing TCP traffic (e.g. by not allowing access to
port 25). While we plan to improve exit-filtering in the future, you're
currently on your own here. Essentially, be prepared for any kind of
IP-traffic to exit the respective TUN interface (and GNUnet will enable
IP-forwarding and NAT for the interface automatically).

Additional configuration options of the exit as shown by the
gnunet-setup tool are:

@node IP Address of external DNS resolver,IPv4 address for Exit interface,Configuring the GNUnet VPN Exit Service,Configuring the GNUnet VPN
@anchor{users/configuration id29}@anchor{17e}@anchor{users/configuration ip-address-of-external-dns-resolver}@anchor{17f}
@subsubsection IP Address of external DNS resolver


If DNS traffic is to exit your machine, it will be send to this DNS
resolver. You can specify an IPv4 or IPv6 address.

@node IPv4 address for Exit interface,IPv6 address for Exit interface,IP Address of external DNS resolver,Configuring the GNUnet VPN
@anchor{users/configuration id30}@anchor{180}@anchor{users/configuration ipv4-address-for-exit-interface}@anchor{181}
@subsubsection IPv4 address for Exit interface


This is the IPv4 address the Interface will get. Make the mask of the
address big enough (255.255.0.0 or, even better, 255.0.0.0) to allow
more mappings of IP addresses into this range. As for the VPN interface,
any unused, private IPv4 address range will do.

@node IPv6 address for Exit interface,,IPv4 address for Exit interface,Configuring the GNUnet VPN
@anchor{users/configuration id31}@anchor{182}@anchor{users/configuration ipv6-address-for-exit-interface}@anchor{183}
@subsubsection IPv6 address for Exit interface


The public IPv6 address the interface will get. If your kernel is not a
very recent kernel and you are willing to manually enable IPv6-NAT, the
IPv6 address you specify here must be a globally routed IPv6 address of
your host.

Suppose your host has the address @code{2001:4ca0::1234/64}, then using
@code{2001:4ca0::1:0/112} would be fine (keep the first 64 bits, then
change at least one bit in the range before the bitmask, in the example
above we changed bit 111 from 0 to 1).

You may also have to configure your router to route traffic for the
entire subnet (@code{2001:4ca0::1:0/112} for example) through your computer
(this should be automatic with IPv6, but obviously anything can be
disabled).

@node Bandwidth Configuration,Configuring NAT,Configuring the GNUnet VPN,Advanced Configuration
@anchor{users/configuration bandwidth-configuration}@anchor{184}@anchor{users/configuration id32}@anchor{185}
@subsection Bandwidth Configuration


You can specify how many bandwidth GNUnet is allowed to use to receive
and send data. This is important for users with limited bandwidth or
traffic volume.

@node Configuring NAT,Peer configuration for distributors e g Operating Systems,Bandwidth Configuration,Advanced Configuration
@anchor{users/configuration configuring-nat}@anchor{186}@anchor{users/configuration id33}@anchor{187}
@subsection Configuring NAT


Most hosts today do not have a normal global IP address but instead are
behind a router performing Network Address Translation (NAT) which
assigns each host in the local network a private IP address. As a
result, these machines cannot trivially receive inbound connections from
the Internet. GNUnet supports NAT traversal to enable these machines to
receive incoming connections from other peers despite their limitations.

In an ideal world, you can press the "Attempt automatic configuration"
button in gnunet-setup to automatically configure your peer correctly.
Alternatively, your distribution might have already triggered this
automatic configuration during the installation process. However,
automatic configuration can fail to determine the optimal settings,
resulting in your peer either not receiving as many connections as
possible, or in the worst case it not connecting to the network at all.

To manually configure the peer, you need to know a few things about your
network setup. First, determine if you are behind a NAT in the first
place. This is always the case if your IP address starts with "10.*"
or "192.168.*". Next, if you have control over your NAT router, you
may choose to manually configure it to allow GNUnet traffic to your
host. If you have configured your NAT to forward traffic on ports 2086
(and possibly 1080) to your host, you can check the "NAT ports have
been opened manually" option, which corresponds to the "PUNCHED_NAT"
option in the configuration file. If you did not punch your NAT box, it
may still be configured to support UPnP, which allows GNUnet to
automatically configure it. In that case, you need to install the
"upnpc" command, enable UPnP (or PMP) on your NAT box and set the
"Enable NAT traversal via UPnP or PMP" option (corresponding to
"ENABLE_UPNP" in the configuration file).

Some NAT boxes can be traversed using the autonomous NAT traversal
method. This requires certain GNUnet components to be installed with
"SUID" privileges on your system (so if you're installing on a system
you do not have administrative rights to, this will not work). If you
installed as 'root', you can enable autonomous NAT traversal by checking
the "Enable NAT traversal using ICMP method". The ICMP method requires
a way to determine your NAT's external (global) IP address. This can be
done using either UPnP, DynDNS, or by manual configuration. If you have
a DynDNS name or know your external IP address, you should enter that
name under "External (public) IPv4 address" (which corresponds to the
"EXTERNAL_ADDRESS" option in the configuration file). If you leave the
option empty, GNUnet will try to determine your external IP address
automatically (which may fail, in which case autonomous NAT traversal
will then not work).

Finally, if you yourself are not behind NAT but want to be able to
connect to NATed peers using autonomous NAT traversal, you need to check
the "Enable connecting to NATed peers using ICMP method" box.

@node Peer configuration for distributors e g Operating Systems,,Configuring NAT,Advanced Configuration
@anchor{users/configuration peer-configuration-for-distributors-0028e-002eg-002e-operating-systems-0029}@anchor{188}@anchor{users/configuration peer-configuration-for-distributors-e-g-operating-systems}@anchor{189}
@subsection Peer configuration for distributors (e.g. Operating Systems)


The "GNUNET_DATA_HOME" in "[PATHS]" in @code{/etc/gnunet.conf} should
be manually set to "/var/lib/gnunet/data/" as the default
"~/.local/share/gnunet/" is probably not that appropriate in this
case. Similarly, distributors may consider pointing
"GNUNET_RUNTIME_DIR" to "/var/run/gnunet/" and "GNUNET_HOME" to
"/var/lib/gnunet/". Also, should a distributor decide to override
system defaults, all of these changes should be done in a custom
@code{/etc/gnunet.conf} and not in the files in the @code{config.d/}
directory.

Given the proposed access permissions, the "gnunet-setup" tool must be
run as use "gnunet" (and with option "-c /etc/gnunet.conf" so that
it modifies the system configuration). As always, gnunet-setup should be
run after the GNUnet peer was stopped using "gnunet-arm -e".
Distributors might want to include a wrapper for gnunet-setup that
allows the desktop-user to "sudo" (e.g. using gtksudo) to the
"gnunet" user account and then runs "gnunet-arm -e",
"gnunet-setup" and "gnunet-arm -s" in sequence.

@node Developer Handbook,GUIs,User Manual,Top
@anchor{developers/index doc}@anchor{18a}@anchor{developers/index developer-handbook}@anchor{18b}@anchor{developers/index gnunet-developer-handbook}@anchor{18c}
@chapter Developer Handbook


This book is intended to be an introduction for programmers that want to
extend the GNUnet framework. GNUnet is more than a simple peer-to-peer
application.

For developers, GNUnet is:


@itemize *

@item 
developed by a community that believes in the GNU philosophy

@item 
Free Software (Free as in Freedom), licensed under the
GNU Affero General Public License@footnote{https://www.gnu.org/licenses/licenses.html#AGPL}

@item 
A set of standards, including coding conventions and architectural rules

@item 
A set of layered protocols, both specifying the communication
between peers as well as the communication between components
of a single peer

@item 
A set of libraries with well-defined APIs suitable for
writing extensions
@end itemize

In particular, the architecture specifies that a peer consists of many
processes communicating via protocols. Processes can be written in almost
any language.
@code{C}, @code{Java} and @code{Guile} APIs exist for accessing existing
services and for writing extensions.
It is possible to write extensions in other languages by
implementing the necessary IPC protocols.

GNUnet can be extended and improved along many possible dimensions, and
anyone interested in Free Software and Freedom-enhancing Networking is
welcome to join the effort. This Developer Handbook attempts to provide
an initial introduction to some of the key design choices and central
components of the system.

This part of the GNUnet documentation is far from complete,
and we welcome informed contributions, be it in the form of
new chapters, sections or insightful comments.

@menu
* Contributing:: 
* Style and Workflow:: 
* libgnunetutil:: 
* System Architecture:: 
* Cryptographic Material:: 
* Subsystem stability:: 
* Service APIs:: 
* REST API:: 
* C Tutorial:: 
* Specifications:: 
* Doxygen:: 

@end menu

@node Contributing,Style and Workflow,,Developer Handbook
@anchor{developers/contributing doc}@anchor{18d}@anchor{developers/contributing contributing}@anchor{18e}@anchor{developers/contributing gnunet-contributors-handbook}@anchor{18f}
@section Contributing


@menu
* Licenses of contributions:: 
* Minor contributions:: 
* Major contributions:: 
* Copyright Assignment:: 
* Reference Manual:: 

@end menu

@node Licenses of contributions,Minor contributions,,Contributing
@anchor{developers/contributing licenses-of-contributions}@anchor{190}
@subsection Licenses of contributions


GNUnet is a GNU@footnote{https://www.gnu.org/} package. All code
contributions must thus be put under the GNU Affero Public License (AGPL)@footnote{https://www.gnu.org/licenses/agpl.html}. All documentation
should be put under FSF approved licenses (see
fdl@footnote{https://www.gnu.org/copyleft/fdl.html}).

By submitting documentation, translations, and other content to GNUnet
you automatically grant the right to publish code under the GNU Public
License and documentation under either or both the GNU Public License or
the GNU Free Documentation License. When contributing to the GNUnet
project, GNU standards and the GNU philosophy@footnote{https://www.gnu.org/philosophy/philosophy.html} should be
adhered to.

@node Minor contributions,Major contributions,Licenses of contributions,Contributing
@anchor{developers/contributing minor-contributions}@anchor{191}
@subsection Minor contributions


Smaller contributions should be provided as patches and not in
developer branches that require git access (see below).
You may post patches on @indicateurl{https://bugs.gnunet.org} in a new or existing
(relevant) issue or on the mailing list @email{gnunet-developers@@gnu.org}.
You may use @cite{git send-email} to send patches to the mailing list.
In general, you may consult @indicateurl{https://docs.kernel.org/process/email-clients.html}
for best practices with respect to mailing patches.

@node Major contributions,Copyright Assignment,Minor contributions,Contributing
@anchor{developers/contributing major-contributions}@anchor{192}
@subsection Major contributions


You can find the GNUnet project repositories at @indicateurl{https://git.gnunet.org}.
The following applies to all repositories, but access policies are only
enforced for the main gnunet repository.
To gain write access to the repository, you will have to sign the
Copyright Assignment (see below).
Note that all commits `MUST' be signed with your GPG key.
The server will verify that any pushed commit is signed. It does not matter
which key is used (we do not keep a list of GPG public keys).

For any changes that are not minor, you `SHOULD' create an issue at
@indicateurl{https://bugs.gnunet.org} and work in a branch that you may
push to the project's server.
This will allow us to track the motivation and execution of a change
and list it in the release notes.

See @ref{193,,Style and Workflow} for details on the style and workflow.

@node Copyright Assignment,Reference Manual,Major contributions,Contributing
@anchor{developers/contributing copyright-assignment}@anchor{194}
@subsection Copyright Assignment


We require a formal copyright assignment for GNUnet contributors to
GNUnet e.V.; nevertheless, we do allow pseudonymous contributions. By
signing the copyright agreement and submitting your code (or
documentation) to us, you agree to share the rights to your code with
GNUnet e.V.; GNUnet e.V. receives non-exclusive ownership rights, and in
particular is allowed to dual-license the code. You retain non-exclusive
rights to your contributions, so you can also share your contributions
freely with other projects.

GNUnet e.V. will publish all accepted contributions under the AGPLv3 or
any later version. The association may decide to publish contributions
under additional licenses (dual-licensing).

We do not intentionally remove your name from your contributions;
however, due to extensive editing it is not always trivial to attribute
contributors properly. If you find that you significantly contributed to
a file (or the project as a whole) and are not listed in the respective
authors file or section, please do let us know.

Download Copyright Assignment here.@footnote{https://www.gnunet.org/static/pdf/copyright.pdf}

@node Reference Manual,,Copyright Assignment,Contributing
@anchor{developers/contributing contributing-to-the-reference-manual}@anchor{195}@anchor{developers/contributing reference-manual}@anchor{196}
@subsection Reference Manual



@itemize -

@item 
Keep line length below 74 characters, except for URLs. URLs break in
the PDF output when they contain linebreaks.

@item 
Do not use tab characters (see chapter 2.1 texinfo manual)

@item 
Write texts in the third person perspective.
@end itemize

@node Style and Workflow,libgnunetutil,Contributing,Developer Handbook
@anchor{developers/style doc}@anchor{197}@anchor{developers/style contributing-testcases}@anchor{198}@anchor{developers/style dev-style-workflow}@anchor{193}@anchor{developers/style style-and-workflow}@anchor{199}
@section Style and Workflow


This document contains normative rules for writing GNUnet
code and naming conventions used throughout the project.

@menu
* Coding style:: 
* Naming conventions:: 
* Continuous integration:: 
* Developer access and commit messages:: 
* Releases:: 

@end menu

@node Coding style,Naming conventions,,Style and Workflow
@anchor{developers/style coding-style}@anchor{19a}
@subsection Coding style


This project follows the GNU Coding Standards.

Indentation is done with two spaces per level, never with tabs.
Specific (though incomplete) indentation rules are defined in an @code{uncrustify}
configuration file (in @code{contrib/uncrustify.cfg}) and are enforced by Git hooks.

C99-style struct initialisation is acceptable and generally encouraged.

As in all good C code, we care about symbol space pollution and thus use
@code{static} to limit the scope where possible, even in the compilation
unit that contains @code{main}.

Only one variable should be declared per line:

@example
// bad
int i,j;

// good
int i;
int j;
@end example

This helps keep diffs small and forces developers to think precisely about
the type of every variable.

Note that >>:c:`char *`<< is different from >>:c:`const char*`<< and
>>:c:`int`<< is different from >>:c:`unsigned int`<< or >>:c:`uint32_t`<<.
Each variable type should be chosen with care.

While @code{goto} should generally be avoided, having a @code{goto} to the
end of a function to a block of clean up statements (free, close,
etc.) can be acceptable.

Conditions should be written with constants on the left (to avoid
accidental assignment) and with the @code{true} target being either the
@code{error} case or the significantly simpler continuation. For
example:

@example
if (0 != stat ("filename,"
               &sbuf))
@{
  error();
@}
else
@{
  /* handle normal case here */
@}
@end example

instead of

@example
if (stat ("filename," &sbuf) == 0) @{
  /* handle normal case here */
 @} else @{
  error();
 @}
@end example

If possible, the error clause should be terminated with a @code{return}
(or @code{goto} to some cleanup routine) and in this case, the @code{else}
clause should be omitted:

@example
if (0 != stat ("filename",
               &sbuf))
@{
  error();
  return;
@}
/* handle normal case here */
@end example

This serves to avoid deep nesting. The 'constants on the left' rule
applies to all constants (including. @code{GNUNET_SCHEDULER_NO_TASK}),
NULL, and enums). With the two above rules (constants on left, errors
in 'true' branch), there is only one way to write most branches
correctly.

Combined assignments and tests are allowed if they do not hinder code
clarity. For example, one can write:

@example
if (NULL == (value = lookup_function()))
@{
  error();
  return;
@}
@end example

Use @code{break} and @code{continue} wherever possible to avoid deep(er)
nesting. Thus, we would write:

@example
next = head;
while (NULL != (pos = next))
@{
  next = pos->next;
  if (! should_free (pos))
    continue;
  GNUNET_CONTAINER_DLL_remove (head,
                               tail,
                               pos);
  GNUNET_free (pos);
@}
@end example

instead of

@example
next = head; while (NULL != (pos = next)) @{
  next = pos->next;
  if (should_free (pos)) @{
    /* unnecessary nesting! */
    GNUNET_CONTAINER_DLL_remove (head, tail, pos);
    GNUNET_free (pos);
   @}
  @}
@end example

We primarily use @code{for} and @code{while} loops. A @code{while} loop is
used if the method for advancing in the loop is not a straightforward
increment operation. In particular, we use:

@example
next = head;
while (NULL != (pos = next))
@{
  next = pos->next;
  if (! should_free (pos))
    continue;
  GNUNET_CONTAINER_DLL_remove (head,
                               tail,
                               pos);
  GNUNET_free (pos);
@}
@end example

to free entries in a list (as the iteration changes the structure of
the list due to the free; the equivalent @code{for} loop does no longer
follow the simple @code{for} paradigm of @code{for(INIT;TEST;INC)}).
However, for loops that do follow the simple @code{for} paradigm we do
use @code{for}, even if it involves linked lists:

@example
/* simple iteration over a linked list */
for (pos = head;
     NULL != pos;
     pos = pos->next)
@{
   use (pos);
@}
@end example

The first argument to all higher-order functions in GNUnet must be
declared to be of type @code{void *} and is reserved for a closure. We
do not use inner functions, as trampolines would conflict with setups
that use non-executable stacks. The first statement in a higher-order
function, which unusually should be part of the variable
declarations, should assign the @code{cls} argument to the precise
expected type. For example:

@example
int
callback (void *cls,
          char *args)
@{
  struct Foo *foo = cls;
  int other_variables;

   /* rest of function */
@}
@end example

As shown in the example above, after the return type of a function
there should be a break. Each parameter should be on a new line.

It is good practice to write complex @code{if} expressions instead of
using deeply nested @code{if} statements. However, except for addition
and multiplication, all operators should use parens. This is fine:

@example
if ( (1 == foo) ||
     ( (0 == bar) &&
       (x != y) ) )
  return x;
@end example

However, this is not:

@example
if (1 == foo)
  return x;
if (0 == bar && x != y)
  return x;
@end example

Note that splitting the @code{if} statement above is debatable as the
@code{return x} is a very trivial statement. However, once the logic
after the branch becomes more complicated (and is still identical),
the "or" formulation should be used for sure.

There should be two empty lines between the end of the function and
the comments describing the following function. There should be a
single empty line after the initial variable declarations of a
function. If a function has no local variables, there should be no
initial empty line. If a long function consists of several complex
steps, those steps might be separated by an empty line (possibly
followed by a comment describing the following step). The code should
not contain empty lines in arbitrary places; if in doubt, it is
likely better to NOT have an empty line (this way, more code will fit
on the screen).

When command-line arguments become too long (and would result in some
particularly ugly @code{uncrustify} wrapping), we start all arguments on
a new line. As a result, there must never be a new line within an
argument declaration (i.e. between @code{struct} and the struct's name)
or between the type and the variable). Example:

@example
struct GNUNET_TRANSPORT_CommunicatorHandle *
GNUNET_TRANSPORT_communicator_connect (
  const struct GNUNET_CONFIGURATION_Handle *cfg,
  const char *config_section_name,
  const char *addr_prefix,
  ...);
@end example

Note that for short function names and arguments, the first argument
does remain on the same line. Example:

@example
void
fun (short i,
     short j);
@end example

@node Naming conventions,Continuous integration,Coding style,Style and Workflow
@anchor{developers/style naming-conventions}@anchor{19b}
@subsection Naming conventions


@menu
* Header files:: 
* Binaries:: 
* Logging:: 
* Symbols:: 
* Tests:: 

@end menu

@node Header files,Binaries,,Naming conventions
@anchor{developers/style header-files}@anchor{19c}
@subsubsection Header files


@c Not sure if "include" and "header" files are synonymous.

For header files, the following suffixes should be used:


@multitable {xxxxxxxxxxxxxxx} {xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx} 
@headitem

Suffix

@tab

Usage

@item

@code{_lib}

@tab

Libraries without associated processes

@item

@code{_service}

@tab

Libraries using service processes

@item

@code{_plugin}

@tab

Plugin definition

@item

@code{_protocol}

@tab

structs used in network protocol

@end multitable


There exist a few exceptions to these rules within the codebase:


@itemize *

@item 
@code{gnunet_config.h} is automatically generated.

@item 
@code{gnunet_common.h}, which defines fundamental routines.

@item 
@code{platform.h}, a collection of portability functions and macros.

@item 
@code{gettext.h}, the internationalization configuration for gettext in GNUnet.
@end itemize

@node Binaries,Logging,Header files,Naming conventions
@anchor{developers/style binaries}@anchor{19d}
@subsubsection Binaries


For binary files, the following convention should be used:


@multitable {xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx} {xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx} 
@headitem

Name format

@tab

Usage

@item

@code{gnunet-service-xxx}

@tab

Service processes (with listen sockets)

@item

@code{gnunet-daemon-xxx}

@tab

Daemon processes (without listen sockets)

@item

@code{gnunet-helper-xxx[-yyy]}

@tab

SUID helper for module xxx

@item

@code{gnunet-yyy}

@tab

End-user command line tools

@item

@code{libgnunet_plugin_xxx_yyy.so}

@tab

Plugin for API xxx

@item

@code{libgnunetxxx.so}

@tab

Library for API xxx

@end multitable


@node Logging,Symbols,Binaries,Naming conventions
@anchor{developers/style logging}@anchor{19e}
@subsubsection Logging


The convention is to define a macro on a per-file basis to manage logging:

@example
#define LOG(kind,...)
[logging_macro] (kind, "[component_name]", __VA_ARGS__)
@end example

The table below indicates the substitutions which should be made
for @code{[component_name]} and @code{[logging_macro]}.


@multitable {xxxxxxxxxxxxxxxxxxxxxxxxxx} {xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx} {xxxxxxxxxxxxxxxxxxxxx} 
@headitem

Software category

@tab

@code{[component_name]}

@tab

@code{[logging_macro]}

@item

Services and daemons

@tab

Directory name in @code{GNUNET_log_setup}

@tab

@code{GNUNET_log}

@item

Command line tools

@tab

Full name in @code{GNUNET_log_setup}

@tab

@code{GNUNET_log}

@item

Service access libraries

@tab

@code{[directory_name]}

@tab

@code{GNUNET_log_from}

@item

Pure libraries

@tab

Library name (without @code{lib} or @code{.so})

@tab

@code{GNUNET_log_from}

@item

Plugins

@tab

@code{[directory_name]-[plugin_name]}

@tab

@code{GNUNET_log_from}

@end multitable


@cartouche
@quotation Todo 
Clear up terminology within the style guide (_lib, _service mapped to appropriate software categories)
@end quotation
@end cartouche

@cartouche
@quotation Todo 
Interpret and write configuration style
@end quotation
@end cartouche

@node Symbols,Tests,Logging,Naming conventions
@anchor{developers/style symbols}@anchor{19f}
@subsubsection Symbols


Exported symbols must be prefixed with @code{GNUNET_[module_name]_} and be
defined in @code{[module_name].c}. There are exceptions to this rule such as
symbols defined in @code{gnunet_common.h}.

Private symbols, including @code{struct}s and macros, must not be prefixed.
In addition, they must not be exported in a way that linkers could use them
or other libraries might see them via headers. This means that they must
`never' be declared in @code{src/include}, and only declared or defined in
C source files or headers under @code{src/[module_name]}.

@node Tests,,Symbols,Naming conventions
@anchor{developers/style tests}@anchor{1a0}
@subsubsection Tests


In the core of GNUnet, we restrict new testcases to a small subset of
languages, in order of preference:


@enumerate 

@item 
C

@item 
Portable Shell Scripts

@item 
Python (3.7 or later)
@end enumerate

We welcome efforts to remove our existing Python 2.7 scripts to replace
them either with portable shell scripts or, at your choice, Python 3.7
or later.

If you contribute new python based testcases, we advise you to not
repeat our past misfortunes and write the tests in a standard test
framework like for example pytest.

For writing portable shell scripts, these tools are useful:


@itemize *

@item 

@table @asis

@item Shellcheck@footnote{https://github.com/koalaman/shellcheck}, for static

analysis of shell scripts.
@end table

@item 
@indicateurl{http://www.etalabs.net/sh_tricks.html},

@item 

@table @asis

@item @code{bash}-@code{dash} (@code{/bin/sh} on Debian) interoperability


@itemize *

@item 
checkbashisms@footnote{https://salsa.debian.org/debian/devscripts/blob/master/scripts/checkbashisms.pl},

@item 
@indicateurl{https://wiki.ubuntu.com/DashAsBinSh}, and

@item 
@indicateurl{https://mywiki.wooledge.org/Bashism}
@end itemize
@end table
@end itemize

Test cases and performance tests should follow the naming conventions
@code{test_[module-under-test]_[test_description].c} and
@code{perf_[module-under-test]_[test_description].c}, respectively.

In either case, if there is only a single test, @code{[test_description]}
may be omitted.

@node Continuous integration,Developer access and commit messages,Naming conventions,Style and Workflow
@anchor{developers/style continuous-integration}@anchor{1a1}
@subsection Continuous integration


The continuous integration buildbot can be found at
@indicateurl{https://buildbot.gnunet.org}. Repositories need to be enabled by a
buildbot admin in order to participate in the builds.

The buildbot can be configured to process scripts in your repository
root under @code{.buildbot/}:

The files @code{build.sh}, @code{install.sh} and @code{test.sh} are executed in
order if present. If you want a specific worker to behave differently,
you can provide a worker specific script, e.g. @code{myworker_build.sh}. In
this case, the generic step will not be executed.

For the @code{gnunet.git} repository, you may use "!tarball" or
"!coverity" in your commit messages. "!tarball" will trigger a
@code{make dist} of the gnunet source and verify that it can be compiled.
The artifact will then be published to
@indicateurl{https://buildbot.gnunet.org/artifacts}. This is a good way to create a
tarball for a release as it verifies the build on another machine.

The "!coverity" trigger will trigger a coverity build and submit the
results for analysis to coverity: @indicateurl{https://scan.coverity.com/}. Only
developers with accounts for the GNUnet project on coverity.com are able
to see the analysis results.

@node Developer access and commit messages,Releases,Continuous integration,Style and Workflow
@anchor{developers/style developer-access-and-commit-messages}@anchor{1a2}
@subsection Developer access and commit messages


Major changes `SHOULD' be developed in a developer branch.
A developer branch is a branch which matches a developer-specific
prefix. As a developer with git access, you should have a git username.
If you do not know it, please ask an admin. A developer branch has the
format:

@example
dev/<username>/<branchname>
@end example

Assuming the developer with username "jdoe" wants to create a new
branch for an i18n fix, the branch name could be:

@example
dev/jdoe/i18n_fx
@end example

You will be able to force push to and delete branches under
your prefix. It is highly recommended to work in your own developer
branches.
Most developers only have access to developer branches anyway.
Code which conforms to the commit message guidelines and
coding style, is tested and builds may be merged to the master branch.
Preferably, you would then...


@itemize -

@item 
(optional) ...squash your commits,

@item 
rebase to master and then

@item 
ask that your branch is merged into master.

@item 
(optional) Delete the branch.
@end itemize

In general, you may want to follow the rule "commit often, push tidy":
You can create smaller, succinct commits with limited meaning on the
commit messages. In the end and before you push or ask your branch to be
merged, you can then squash the commits or rename them.
You can ask the project maintainer to merge your branch through any channel,
but the @email{gnunet-developers@@gnu.org} mailing list is the preferred method.

Commit messages are required to convey what changes were
made in a self-contained fashion. Commit messages such as "fix" or
"cleanup" are not acceptable. You commit message should ideally start
with the subsystem name and be followed by a succinct description of the
change. Where applicable a reference to a bug number in the bugtracker
may also be included. Example:

@example
# <subsystem>: <description>. (#XXXX)
IDENTITY: Fix wrong key construction for anonymous ECDSA identity. (Fixes #12344)
@end example

We do not maintain a separate ChangeLog file anymore as the changes are
documented in the git repositories.
If you edit files that change user-facing behaviour (e.g. header files in src/include)
you will have to add at least one line in the commit message starting with "NEWS:".
If there is a special reason why you deem this unnecessary, you can add the line
"NEWS: -".
Any "NEWS" lines from commit messages will on release be put into the "NEWS" file in the tarball
to inform users and packages of noteworthy changes.

@example
# NEWS
NEWS: The API for foo has changed: lorem ipsum.
@end example

Note that if you run "./bootstrap" hooks will be installed that add message hints
to your commit message template when you run "git commit".

If you need to modify a commit you can do so using:

@example
$ git commit --amend
@end example

If you need to modify a number of successive commits you will have to
rebase and squash. Note that most branches are protected. This means
that you can only fix commits as long as they are not pushed. You can
only modify pushed commits in your own developer branches.

@node Releases,,Developer access and commit messages,Style and Workflow
@anchor{developers/style releases}@anchor{1a3}
@subsection Releases


Packaging a release can only be done by a maintainer.

In this order do:


@enumerate 0

@item 
Update your local tree

@item 
Update the submodule @code{contrib/handbook}

@item 
Run @code{./bootstrap} to regenerate files

@item 
`Optional': Update bootstrap HELLO file and update GANA generated files using @code{./scripts/gana_update.sh}

@item 
Update NEWS file and PO files and commit

@item 
Tag the release (Named commit!)

@item 
Create and test tarballs

@item 
Upload tarball

@item 
Write release announcement
@end enumerate

`(1)' Make sure the @code{contrib/handbook} submodule
is up to date.
In the `handbook repository', make sure you have a folder that
contains the @code{prebuilt} branch as a worktree:

@example
$ git worktree add _build prebuilt
@end example

Then compile the current handbook:

@example
$ sphinx-multiversion --dump-metadata . _build
@end example

Commit the changes (if any) and update the submoduce in the
gnunet repository under @code{doc/handbook} and execute:

@example
$ git pull origin prebuilt
@end example

If the submodule was updated, commit the new submodule.

`(2)' You should now re-run bootstrap:

@example
.. code-block:: console
@end example

@quotation

$ ./bootstrap
@end quotation

`(3)' If this is a major release, it makes sense to update the bootstrap peer

HELLO. For this, you need access to the bootstrap peer (@code{GKZS}) and create
the hello file. Make sure that you update the bootstrap peer first such that
any changes in the identity format are actually refrected!
It is recommended to have it expire within a year:

@example
$ gnunet-hello -e -E "1 year" -b -c gnunet.conf > GKZSYTE4H3N4RMYGFYEM95RV1CDRPH8QNJNTPMEB7C4QJGGR3KEG
@end example

You can then update the file in @code{data/hellos}.

`(4)' Then, in order to pre-populate the news file with news hints
from commit messages (see above), we execute

@example
$ sh contrib/scripts/update_news.sh v0.22.0
@end example

@code{v0.22.0} is the placeholder for the version number you want to release.
Now edit the @code{NEWS} file to include all the noteworthy changes
you want to add.

Then, update the po files:

@example
$ meson setup build-release && meson compile -C build-release gnunet-update-po
@end example

Commit all changes in the source tree now, and push the changes.

`(5)' You can now tag the release. Please `always' use named tags (important for our
nightly tarball build trigger).

@example
$ git tag v0.22.0 -m "GNUnet release v0.22.0"
@end example

`(6)' Then, we can re-run @code{./bootstrap} and make the tarball:

@example
$ ./bootstrap && meson setup --reconfigure build-release && meson dist -C build-release --formats gztar
@end example

Meson will automatically run the tests.

`(7)' To distribute the tarballs create a release triplet as defined in the GNU guidelines and
upload them to the FTP (@indicateurl{https://www.gnu.org/prep//maintain/html_node/Automated-FTP-Uploads.html}).

The file @code{gnunet-0.22.0.tar.gz} will be found inside @code{build/meson-dist}.
The file must be signed by a maintainer and the signature and tarball uploaded to
@indicateurl{https://buildbot.gnunet.org/releases/}.

`(8)' After the tarballs are uploaded, we need to write the announcement.
For minor releases, we only write a brief statement:

@example
See also: https://www.gnunet.org/en/news/2024-06-0.21.2.html

This is a bugfix release for gnunet 0.21.1. It primarily addresses some
connectivity issues introduced with our new transport subsystem.

Links

  Source: https://ftpmirror.gnu.org/gnunet/gnunet-0.21.2.tar.gz (https://ftpmirror.gnu.org/gnunet/gnunet-0.21.2.tar.gz.sig)
  Detailed list of changes: https://git.gnunet.org/gnunet.git/log/?h=v0.21.2
  NEWS: https://git.gnunet.org/gnunet.git/tree/NEWS?h=v0.21.2
  The list of closed issues in the bug tracker: https://bugs.gnunet.org/changelog_page.php?version_id=440

The GPG key used to sign is: 3D11063C10F98D14BD24D1470B0998EF86F59B6A

Note that due to mirror synchronization, not all links may be
functional early after the release. For direct access try
https://ftp.gnu.org/gnu/gnunet/
@end example

The announcement is posted to @code{gnunet-developers@@gnu.org} as well as
@code{help-gnunet@@gnu.org}.

For major releases, we use a full release announcement.
You can find the template in the @code{www.git}.
The full release announcement is sent to
@code{info-gnu@@gnu.org} as well.

@node libgnunetutil,System Architecture,Style and Workflow,Developer Handbook
@anchor{developers/util doc}@anchor{1a4}@anchor{developers/util gnunet-libgnunetutil}@anchor{1a5}@anchor{developers/util libgnunetutil}@anchor{1a6}
@section libgnunetutil


@c This section of the docs is particularly amenable to
@c integration with the doxygen output, as it is a library and not
@c an individual program.

libgnunetutil is the fundamental library that all GNUnet code builds upon.
Ideally, this library should contain most of the platform dependent code
(except for user interfaces and really special needs that only few
applications have). It is also supposed to offer basic services that most
if not all GNUnet binaries require. The code of libgnunetutil is in the
@code{src/util/} directory. The public interface to the library is in the
@code{gnunet_util.h} header.

The functions provided by libgnunetutil fall
roughly into the following categories (in roughly the order of importance
for new developers):


@itemize *

@item 
logging (common_logging.c)

@item 
memory allocation (common_allocation.c)

@item 
endianness conversion (common_endian.c)

@item 
internationalization (common_gettext.c)

@item 
String manipulation (string.c)

@item 
file access (disk.c)

@item 
buffered disk IO (bio.c)

@item 
time manipulation (time.c)

@item 
configuration parsing (configuration.c)

@item 
command-line handling (getopt*.c)

@item 
cryptography (crypto_*.c)

@item 
data structures (container_*.c)

@item 
CPS-style scheduling (scheduler.c)

@item 
Program initialization (program.c)

@item 
Networking (network.c, client.c, server*.c, service.c)

@item 
message queuing (mq.c)

@item 
bandwidth calculations (bandwidth.c)

@item 
Other OS-related (os*.c, plugin.c, signal.c)

@item 
Pseudonym management (pseudonym.c)
@end itemize

It should be noted that only developers that fully understand this entire
API will be able to write good GNUnet code.

Ideally, porting GNUnet should only require porting the gnunetutil
library. More testcases for the gnunetutil APIs are therefore a great
way to make porting of GNUnet easier.

@node System Architecture,Cryptographic Material,libgnunetutil,Developer Handbook
@anchor{developers/architecture doc}@anchor{1a7}@anchor{developers/architecture id1}@anchor{1a8}@anchor{developers/architecture system-architecture}@anchor{1a9}
@section System Architecture


@cartouche
@quotation Todo 
FIXME: For those irritated by the textflow, we are missing images here,
in the short term we should add them back, in the long term this should
work without images or have images with alt-text.
@end quotation
@end cartouche

@cartouche
@quotation Todo 
Adjust image sizes so that they are less obtrusive.
@end quotation
@end cartouche

GNUnet developers like LEGOs. The blocks are indestructible, can be
stacked together to construct complex buildings and it is generally easy
to swap one block for a different one that has the same shape. GNUnet's
architecture is based on LEGOs:


@image{gnunet-figures/service_lego_block,,,service_lego_block,png}


This chapter documents the GNUnet LEGO system, also known as GNUnet's
system architecture.

The most common GNUnet component is a service. Services offer an API (or
several, depending on what you count as "an API") which is implemented
as a library. The library communicates with the main process of the
service using a service-specific network protocol. The main process of
the service typically doesn't fully provide everything that is needed
--- it has holes to be filled by APIs to other services.

A special kind of component in GNUnet are user interfaces and daemons.
Like services, they have holes to be filled by APIs of other services.
Unlike services, daemons do not implement their own network protocol and
they have no API:


@image{gnunet-figures/daemon_lego_block,,,daemon_lego_block,png}


The GNUnet system provides a range of services, daemons and user
interfaces, which are then combined into a layered GNUnet instance (also
known as a peer).


@image{gnunet-figures/service_stack,,,service_stack,png}


Note that while it is generally possible to swap one service for another
compatible service, there is often only one implementation. However,
during development we often have a "new" version of a service in
parallel with an "old" version. While the "new" version is not
working, developers working on other parts of the service can continue
their development by simply using the "old" service. Alternative
design ideas can also be easily investigated by swapping out individual
components. This is typically achieved by simply changing the name of
the "BINARY" in the respective configuration section.

Key properties of GNUnet services are that they must be separate
processes and that they must protect themselves by applying tight error
checking against the network protocol they implement (thereby achieving
a certain degree of robustness).

On the other hand, the APIs are implemented to tolerate failures of the
service, isolating their host process from errors by the service. If the
service process crashes, other services and daemons around it should not
also fail, but instead wait for the service process to be restarted by
ARM.

@cartouche
@quotation Todo 

@table @asis

@item Missing subsystems:


@itemize *

@item 
TOPOLOGY, FRAGMENTATION

@item 
VPN and support services (DNS, PT, EXIT)

@item 
DATASTORE (only used by FS?)

@item 
MULTICAST and social services (PSYC, PSYCSTORE, SOCIAL)

@item 
GNS support services/applications (GNSRECORD, ZONEMASTER)

@item 
Set-based applications (SCALARPRODUCT, SECRETSHARING, CONSENSUS, VOTING)
@end itemize
@end table
@end quotation
@end cartouche

@node Cryptographic Material,Subsystem stability,System Architecture,Developer Handbook
@anchor{developers/cryptographic-material doc}@anchor{1aa}@anchor{developers/cryptographic-material cryptographic-material}@anchor{1ab}@anchor{developers/cryptographic-material id1}@anchor{1ac}
@section Cryptographic Material


This sections provides an overview over the cryptographic material that is used
throughout GNUnet and its intended purpose.

TODO

@quotation


@itemize -

@item 
transport

@item 
core
- peer id

@item 
cadet

@item 
gns

@item 
...
@end itemize
@end quotation

@node Subsystem stability,Service APIs,Cryptographic Material,Developer Handbook
@anchor{developers/stability doc}@anchor{1ad}@anchor{developers/stability id1}@anchor{1ae}@anchor{developers/stability subsystem-stability}@anchor{1af}
@section Subsystem stability


This page documents the current stability of the various GNUnet
subsystems. Stability here describes the expected degree of
compatibility with future versions of GNUnet.

For each subsystem we distinguish between compatibility on the P2P
network level (communication protocol between peers), the IPC level
(communication between the service and the service library) and the
API level (stability of the API).

P2P compatibility is relevant in terms of which applications are likely
going to be able to communicate with future versions of the network.
IPC communication is relevant for the implementation of language bindings
that re-implement the IPC messages. Finally, API compatibility is relevant
to developers that hope to be able to avoid changes to applications built
on top of the APIs of the framework.

The following table summarizes our current view of the stability of the
respective protocols or APIs:

@cartouche
@quotation Todo 
Make table automatically generated individual pages?
@end quotation
@end cartouche


@multitable {xxxxxxxxxxxxxxxxxxx} {xxxxxxxxxxxxxxxxxxx} {xxxxxxxxxxxxxxxxxxx} {xxxxxxxxxxxxxxxxxxx} 
@headitem

Subsystem

@tab

P2P

@tab

IPC

@tab

C API

@item

util

@tab

n/a

@tab

n/a

@tab

stable

@item

arm

@tab

n/a

@tab

stable

@tab

stable

@item

block

@tab

n/a

@tab

n/a

@tab

stable

@item

cadet

@tab

testing

@tab

testing

@tab

testing

@item

consensus

@tab

experimental

@tab

experimental

@tab

experimental

@item

core

@tab

unstable

@tab

unstable

@tab

unstable

@item

datacache

@tab

n/a

@tab

n/a

@tab

unstable

@item

datastore

@tab

n/a

@tab

unstable

@tab

unstable

@item

dht

@tab

testing

@tab

testing

@tab

testing

@item

dns

@tab

stable

@tab

stable

@tab

stable

@item

dv

@tab

testing

@tab

testing

@tab

n/a

@item

exit

@tab

testing

@tab

n/a

@tab

n/a

@item

fragmentation

@tab

stable

@tab

n/a

@tab

stable

@item

fs

@tab

unstable

@tab

unstable

@tab

unstable

@item

gns

@tab

stable

@tab

stable

@tab

stable

@item

hello

@tab

n/a

@tab

n/a

@tab

testing

@item

hostlist

@tab

unstable

@tab

unstable

@tab

n/a

@item

identity

@tab

testing

@tab

testing

@tab

n/a

@item

multicast

@tab

experimental

@tab

experimental

@tab

experimental

@item

namestore

@tab

n/a

@tab

testing

@tab

testing

@item

nat

@tab

n/a

@tab

n/a

@tab

unstable

@item

nse

@tab

stable

@tab

stable

@tab

stable

@item

pt

@tab

n/a

@tab

n/a

@tab

n/a

@item

regex

@tab

stable

@tab

stable

@tab

stable

@item

revocation

@tab

stable

@tab

stable

@tab

stable

@item

social

@tab

experimental

@tab

experimental

@tab

experimental

@item

statistics

@tab

n/a

@tab

stable

@tab

stable

@item

testbed

@tab

n/a

@tab

testing

@tab

testing

@item

testing

@tab

n/a

@tab

n/a

@tab

testing

@item

topology

@tab

n/a

@tab

n/a

@tab

n/a

@item

transport

@tab

experimental

@tab

experimental

@tab

experimental

@item

tun

@tab

n/a

@tab

n/a

@tab

stable

@item

vpn

@tab

testing

@tab

n/a

@tab

n/a

@end multitable


Here is a rough explanation of the values:

@cartouche
@quotation Todo 
0.10.x is outdated - rewrite "@code{stable}" to reflect
a time-independent meaning.
@end quotation
@end cartouche


@table @asis

@item '@code{stable}'

No incompatible changes are planned at this time; for IPC/APIs, if
there are incompatible changes, they will be minor and might only
require minimal changes to existing code; for P2P, changes will be
avoided if at all possible for the current major version.

@item '@code{testing}'

No incompatible changes are planned at this time, but the code is
still known to be in flux; so while we have no concrete plans, our
expectation is that there will still be minor modifications; for P2P,
changes will likely be extensions that should not break existing code

@item '@code{unstable}'

Changes are planned and will happen; however, they will not be
totally radical and the result should still resemble what is there
now; nevertheless, anticipated changes will break protocol/API
compatibility

@item '@code{experimental}'

Changes are planned and the result may look nothing like what the
API/protocol looks like today

@item '@code{unknown}'

Someone should think about where this subsystem headed

@item '@code{n/a}'

This subsystem does not implement a corresponding API/protocol
@end table

@node Service APIs,REST API,Subsystem stability,Developer Handbook
@anchor{developers/apis/index doc}@anchor{1b0}@anchor{developers/apis/index service-apis}@anchor{1b1}
@section Service APIs


These services comprise a backbone of core services for
peer-to-peer applications to use.

@geindex CADET; subsystem
@anchor{developers/apis/cadet cadet-subsystem-dev}@anchor{1b2}
@menu
* CADET:: 
* CORE:: 
* CONG:: 
* DHT: DHT<2>. 
* FS:: 
* GNS: GNS<2>. 
* IDENTITY:: 
* MESSENGER:: 
* NAMECACHE:: 
* NAMESTORE:: 
* NSE:: 
* PEERSTORE:: 
* REST:: 
* REVOCATION:: 
* Peer-to-Peer Set Operations:: 
* STATISTICS:: 
* TRANSPORT:: 

@end menu

@node CADET,CORE,,Service APIs
@anchor{developers/apis/cadet doc}@anchor{1b3}@anchor{developers/apis/cadet cadet}@anchor{1b4}
@subsection CADET


The CADET API (defined in @code{gnunet_cadet_service.h}) is the messaging
API used by P2P applications built using GNUnet. It provides
applications the ability to send and receive encrypted messages to any
peer participating in GNUnet. The API is heavily based on the CORE API.

CADET delivers messages to other peers in "channels". A channel is a
permanent connection defined by a destination peer (identified by its
public key) and a port number. Internally, CADET tunnels all channels
towards a destination peer using one session key and relays the data on
multiple "connections", independent from the channels.

Each channel has optional parameters, the most important being the
reliability flag. Should a message get lost on TRANSPORT/CORE level, if
a channel is created with as reliable, CADET will retransmit the lost
message and deliver it in order to the destination application.

GNUNET_CADET_connect

@c .. doxygenfunction:: GNUNET_CADET_connect

To communicate with other peers using CADET, it is necessary to first
connect to the service using @code{GNUNET_CADET_connect}. This function
takes several parameters in form of callbacks, to allow the client to
react to various events, like incoming channels or channels that
terminate, as well as specify a list of ports the client wishes to
listen to (at the moment it is not possible to start listening on
further ports once connected, but nothing prevents a client to connect
several times to CADET, even do one connection per listening port). The
function returns a handle which has to be used for any further
interaction with the service.

GNUNET_CADET_channel_create

@c .. doxygenfunction:: GNUNET_CADET_channel_create

To connect to a remote peer, a client has to call the
@code{GNUNET_CADET_channel_create} function. The most important parameters
given are the remote peer's identity (it public key) and a port, which
specifies which application on the remote peer to connect to, similar to
TCP/UDP ports. CADET will then find the peer in the GNUnet network and
establish the proper low-level connections and do the necessary key
exchanges to assure and authenticated, secure and verified
communication. Similar to
@code{GNUNET_CADET_connect},@code{GNUNET_CADET_create_channel} returns a
handle to interact with the created channel.

GNUNET_CADET_notify_transmit_ready

@c .. doxygenfunction:: GNUNET_CADET_notify_transmit_ready

For every message the client wants to send to the remote application,
@code{GNUNET_CADET_notify_transmit_ready} must be called, indicating the
channel on which the message should be sent and the size of the message
(but not the message itself!). Once CADET is ready to send the message,
the provided callback will fire, and the message contents are provided
to this callback.

Please note the CADET does not provide an explicit notification of when
a channel is connected. In loosely connected networks, like big wireless
mesh networks, this can take several seconds, even minutes in the worst
case. To be alerted when a channel is online, a client can call
@code{GNUNET_CADET_notify_transmit_ready} immediately after
@code{GNUNET_CADET_create_channel}. When the callback is activated, it
means that the channel is online. The callback can give 0 bytes to CADET
if no message is to be sent, this is OK.

GNUNET_CADET_notify_transmit_cancel

@c .. doxygenfunction:: GNUNET_CADET_notify_transmit_cancel

If a transmission was requested but before the callback fires it is no
longer needed, it can be canceled with
@code{GNUNET_CADET_notify_transmit_ready_cancel}, which uses the handle
given back by @code{GNUNET_CADET_notify_transmit_ready}. As in the case of
CORE, only one message can be requested at a time: a client must not
call @code{GNUNET_CADET_notify_transmit_ready} again until the callback is
called or the request is canceled.

GNUNET_CADET_channel_destroy

@c .. doxygenfunction:: GNUNET_CADET_notify_channel_destroy

When a channel is no longer needed, a client can call
@code{GNUNET_CADET_channel_destroy} to get rid of it. Note that CADET will
try to transmit all pending traffic before notifying the remote peer of
the destruction of the channel, including retransmitting lost messages
if the channel was reliable.

Incoming channels, channels being closed by the remote peer, and traffic
on any incoming or outgoing channels are given to the client when CADET
executes the callbacks given to it at the time of
@code{GNUNET_CADET_connect}.

GNUNET_CADET_disconnect

@c .. doxygenfunction:: GNUNET_CADET_disconnect

Finally, when an application no longer wants to use CADET, it should
call @code{GNUNET_CADET_disconnect}, but first all channels and pending
transmissions must be closed (otherwise CADET will complain).

@geindex CORE; subsystem
@anchor{developers/apis/core core-subsystem-dev}@anchor{1b5}
@node CORE,CONG,CADET,Service APIs
@anchor{developers/apis/core doc}@anchor{1b6}@anchor{developers/apis/core core}@anchor{1b7}
@subsection CORE


The CORE API (defined in @code{gnunet_core_service.h}) is the basic
messaging API used by P2P applications built using GNUnet. It provides
applications the ability to send and receive encrypted messages to the
peer's "directly" connected neighbours.

As CORE connections are generally "direct" connections, applications
must not assume that they can connect to arbitrary peers this way, as
"direct" connections may not always be possible. Applications using
CORE are notified about which peers are connected. Creating new
"direct" connections must be done using the TRANSPORT API.

The CORE API provides unreliable, out-of-order delivery. While the
implementation tries to ensure timely, in-order delivery, both message
losses and reordering are not detected and must be tolerated by the
application. Most important, the core will NOT perform retransmission if
messages could not be delivered.

Note that CORE allows applications to queue one message per connected
peer. The rate at which each connection operates is influenced by the
preferences expressed by local application as well as restrictions
imposed by the other peer. Local applications can express their
preferences for particular connections using the "performance" API of
the ATS service.

Applications that require more sophisticated transmission capabilities
such as TCP-like behavior, or if you intend to send messages to
arbitrary remote peers, should use the CADET API.

The typical use of the CORE API is to connect to the CORE service using
@code{GNUNET_CORE_connect}, process events from the CORE service (such as
peers connecting, peers disconnecting and incoming messages) and send
messages to connected peers using @code{GNUNET_CORE_notify_transmit_ready}.
Note that applications must cancel pending transmission requests if they
receive a disconnect event for a peer that had a transmission pending;
furthermore, queuing more than one transmission request per peer per
application using the service is not permitted.

The CORE API also allows applications to monitor all communications of
the peer prior to encryption (for outgoing messages) or after decryption
(for incoming messages). This can be useful for debugging, diagnostics
or to establish the presence of cover traffic (for anonymity). As
monitoring applications are often not interested in the payload, the
monitoring callbacks can be configured to only provide the message
headers (including the message type and size) instead of copying the
full data stream to the monitoring client.

The init callback of the @code{GNUNET_CORE_connect} function is called with
the hash of the public key of the peer. This public key is used to
identify the peer globally in the GNUnet network. Applications are
encouraged to check that the provided hash matches the hash that they
are using (as theoretically the application may be using a different
configuration file with a different private key, which would result in
hard to find bugs).

As with most service APIs, the CORE API isolates applications from
crashes of the CORE service. If the CORE service crashes, the
application will see disconnect events for all existing connections.
Once the connections are re-established, the applications will be
receive matching connect events.

core client-service protocol
.. _The-CORE-Client_002dService-Protocol:

@menu
* The CORE Client-Service Protocol:: 
* The CORE Peer-to-Peer Protocol:: 

@end menu

@node The CORE Client-Service Protocol,The CORE Peer-to-Peer Protocol,,CORE
@anchor{developers/apis/core the-core-client-service-protocol}@anchor{1b8}
@subsubsection The CORE Client-Service Protocol


This section describes the protocol between an application using the
CORE service (the client) and the CORE service process itself.

@menu
* Setup2:: 
* Notifications:: 
* Sending:: 

@end menu

@node Setup2,Notifications,,The CORE Client-Service Protocol
@anchor{developers/apis/core id1}@anchor{1b9}@anchor{developers/apis/core setup2}@anchor{1ba}
@subsubsection Setup2


When a client connects to the CORE service, it first sends a
@code{InitMessage} which specifies options for the connection and a set of
message type values which are supported by the application. The options
bitmask specifies which events the client would like to be notified
about. The options include:


@table @asis

@item `GNUNET_CORE_OPTION_NOTHING'

No notifications

@item `GNUNET_CORE_OPTION_STATUS_CHANGE'

Peers connecting and disconnecting

@item `GNUNET_CORE_OPTION_FULL_INBOUND'

All inbound messages (after decryption) with full payload

@item `GNUNET_CORE_OPTION_HDR_INBOUND'

Just the @code{MessageHeader} of all inbound messages

@item `GNUNET_CORE_OPTION_FULL_OUTBOUND'

All outbound messages (prior to encryption) with full payload

@item `GNUNET_CORE_OPTION_HDR_OUTBOUND'

Just the @code{MessageHeader} of all outbound messages
@end table

Typical applications will only monitor for connection status changes.

The CORE service responds to the @code{InitMessage} with an
@code{InitReplyMessage} which contains the peer's identity. Afterwards,
both CORE and the client can send messages.

@node Notifications,Sending,Setup2,The CORE Client-Service Protocol
@anchor{developers/apis/core id2}@anchor{1bb}@anchor{developers/apis/core notifications}@anchor{1bc}
@subsubsection Notifications


The CORE will send @code{ConnectNotifyMessage}s and
@code{DisconnectNotifyMessage}s whenever peers connect or disconnect from
the CORE (assuming their type maps overlap with the message types
registered by the client). When the CORE receives a message that matches
the set of message types specified during the @code{InitMessage} (or if
monitoring is enabled in for inbound messages in the options), it sends
a @code{NotifyTrafficMessage} with the peer identity of the sender and the
decrypted payload. The same message format (except with
@code{GNUNET_MESSAGE_TYPE_CORE_NOTIFY_OUTBOUND} for the message type) is
used to notify clients monitoring outbound messages; here, the peer
identity given is that of the receiver.

@node Sending,,Notifications,The CORE Client-Service Protocol
@anchor{developers/apis/core id3}@anchor{1bd}@anchor{developers/apis/core sending}@anchor{1be}
@subsubsection Sending


When a client wants to transmit a message, it first requests a
transmission slot by sending a @code{SendMessageRequest} which specifies
the priority, deadline and size of the message. Note that these values
may be ignored by CORE. When CORE is ready for the message, it answers
with a @code{SendMessageReady} response. The client can then transmit the
payload with a @code{SendMessage} message. Note that the actual message
size in the @code{SendMessage} is allowed to be smaller than the size in
the original request. A client may at any time send a fresh
@code{SendMessageRequest}, which then superceeds the previous
@code{SendMessageRequest}, which is then no longer valid. The client can
tell which @code{SendMessageRequest} the CORE service's
@code{SendMessageReady} message is for as all of these messages contain a
"unique" request ID (based on a counter incremented by the client for
each request).

CORE Peer-to-Peer Protocol
.. _The-CORE-Peer_002dto_002dPeer-Protocol:

@node The CORE Peer-to-Peer Protocol,,The CORE Client-Service Protocol,CORE
@anchor{developers/apis/core the-core-peer-to-peer-protocol}@anchor{1bf}
@subsubsection The CORE Peer-to-Peer Protocol


EphemeralKeyMessage creation
.. _Creating-the-EphemeralKeyMessage:

@menu
* Creating the EphemeralKeyMessage:: 
* Establishing a connection:: 
* Encryption and Decryption:: 
* Type maps:: 

@end menu

@node Creating the EphemeralKeyMessage,Establishing a connection,,The CORE Peer-to-Peer Protocol
@anchor{developers/apis/core creating-the-ephemeralkeymessage}@anchor{1c0}
@subsubsection Creating the EphemeralKeyMessage


When the CORE service starts, each peer creates a fresh ephemeral (ECC)
public-private key pair and signs the corresponding
@code{EphemeralKeyMessage} with its long-term key (which we usually call
the peer's identity; the hash of the public long term key is what
results in a @code{struct GNUNET_PeerIdentity} in all GNUnet APIs. The
ephemeral key is ONLY used for an ECDHE (Elliptic-curve Diffie---Hellman@footnote{http://en.wikipedia.org/wiki/Elliptic_curve_Diffie%E2%80%93Hellman})
exchange by the CORE service to establish symmetric session keys. A peer
will use the same @code{EphemeralKeyMessage} for all peers for
@code{REKEY_FREQUENCY}, which is usually 12 hours. After that time, it will
create a fresh ephemeral key (forgetting the old one) and broadcast the
new @code{EphemeralKeyMessage} to all connected peers, resulting in fresh
symmetric session keys. Note that peers independently decide on when to
discard ephemeral keys; it is not a protocol violation to discard keys
more often. Ephemeral keys are also never stored to disk; restarting a
peer will thus always create a fresh ephemeral key. The use of ephemeral
keys is what provides forward secrecy@footnote{http://en.wikipedia.org/wiki/Forward_secrecy}.

Just before transmission, the @code{EphemeralKeyMessage} is patched to
reflect the current sender_status, which specifies the current state of
the connection from the point of view of the sender. The possible values
are:


@itemize -

@item 
@code{KX_STATE_DOWN} Initial value, never used on the network

@item 
@code{KX_STATE_KEY_SENT} We sent our ephemeral key, do not know the key
of the other peer

@item 
@code{KX_STATE_KEY_RECEIVED} This peer has received a valid ephemeral
key of the other peer, but we are waiting for the other peer to
confirm it's authenticity (ability to decode) via challenge-response.

@item 
@code{KX_STATE_UP} The connection is fully up from the point of view of
the sender (now performing keep-alive)

@item 
@code{KX_STATE_REKEY_SENT} The sender has initiated a rekeying
operation; the other peer has so far failed to confirm a working
connection using the new ephemeral key
@end itemize

@node Establishing a connection,Encryption and Decryption,Creating the EphemeralKeyMessage,The CORE Peer-to-Peer Protocol
@anchor{developers/apis/core establishing-a-connection}@anchor{1c1}@anchor{developers/apis/core id4}@anchor{1c2}
@subsubsection Establishing a connection


Peers begin their interaction by sending a @code{EphemeralKeyMessage} to
the other peer once the TRANSPORT service notifies the CORE service
about the connection. A peer receiving an @code{EphemeralKeyMessage} with a
status indicating that the sender does not have the receiver's ephemeral
key, the receiver's @code{EphemeralKeyMessage} is sent in response.
Additionally, if the receiver has not yet confirmed the authenticity of
the sender, it also sends an (encrypted)@code{PingMessage} with a
challenge (and the identity of the target) to the other peer. Peers
receiving a @code{PingMessage} respond with an (encrypted) @code{PongMessage}
which includes the challenge. Peers receiving a @code{PongMessage} check
the challenge, and if it matches set the connection to @code{KX_STATE_UP}.

@node Encryption and Decryption,Type maps,Establishing a connection,The CORE Peer-to-Peer Protocol
@anchor{developers/apis/core encryption-and-decryption}@anchor{1c3}@anchor{developers/apis/core id5}@anchor{1c4}
@subsubsection Encryption and Decryption


All functions related to the key exchange and encryption/decryption of
messages can be found in @code{gnunet-service-core_kx.c} (except for the
cryptographic primitives, which are in @code{util/crypto*.c}). Given the
key material from ECDHE, a Key derivation function (Key derivation function@footnote{https://en.wikipedia.org/wiki/Key_derivation_function}) is
used to derive two pairs of encryption and decryption keys for AES-256
and TwoFish, as well as initialization vectors and authentication keys
(for HMAC (HMAC@footnote{https://en.wikipedia.org/wiki/HMAC})). The HMAC is
computed over the encrypted payload. Encrypted messages include an
iv_seed and the HMAC in the header.

Each encrypted message in the CORE service includes a sequence number
and a timestamp in the encrypted payload. The CORE service remembers the
largest observed sequence number and a bit-mask which represents which
of the previous 32 sequence numbers were already used. Messages with
sequence numbers lower than the largest observed sequence number minus
32 are discarded. Messages with a timestamp that is less than
@code{REKEY_TOLERANCE} off (5 minutes) are also discarded. This of course
means that system clocks need to be reasonably synchronized for peers to
be able to communicate. Additionally, as the ephemeral key changes every
12 hours, a peer would not even be able to decrypt messages older than
12 hours.

@node Type maps,,Encryption and Decryption,The CORE Peer-to-Peer Protocol
@anchor{developers/apis/core id6}@anchor{1c5}@anchor{developers/apis/core type-maps}@anchor{1c6}
@subsubsection Type maps


Once an encrypted connection has been established, peers begin to
exchange type maps. Type maps are used to allow the CORE service to
determine which (encrypted) connections should be shown to which
applications. A type map is an array of 65536 bits representing the
different types of messages understood by applications using the CORE
service. Each CORE service maintains this map, simply by setting the
respective bit for each message type supported by any of the
applications using the CORE service. Note that bits for message types
embedded in higher-level protocols (such as MESH) will not be included
in these type maps.

Typically, the type map of a peer will be sparse. Thus, the CORE service
attempts to compress its type map using @code{gzip}-style compression
("deflate") prior to transmission. However, if the compression fails
to compact the map, the map may also be transmitted without compression
(resulting in @code{GNUNET_MESSAGE_TYPE_CORE_COMPRESSED_TYPE_MAP} or
@code{GNUNET_MESSAGE_TYPE_CORE_BINARY_TYPE_MAP} messages respectively).
Upon receiving a type map, the respective CORE service notifies
applications about the connection to the other peer if they support any
message type indicated in the type map (or no message type at all). If
the CORE service experience a connect or disconnect event from an
application, it updates its type map (setting or unsetting the
respective bits) and notifies its neighbours about the change. The CORE
services of the neighbours then in turn generate connect and disconnect
events for the peer that sent the type map for their respective
applications. As CORE messages may be lost, the CORE service confirms
receiving a type map by sending back a
@code{GNUNET_MESSAGE_TYPE_CORE_CONFIRM_TYPE_MAP}. If such a confirmation
(with the correct hash of the type map) is not received, the sender will
retransmit the type map (with exponential back-off).

@geindex CONG; subsystem
@anchor{developers/apis/cong cong-subsystem-dev}@anchor{1c7}
@node CONG,DHT<2>,CORE,Service APIs
@anchor{developers/apis/cong doc}@anchor{1c8}@anchor{developers/apis/cong cong}@anchor{1c9}
@subsection CONG


CONG (COre Next Generation) is the name of the project redesigns the
@ref{1b5,,CORE} service. Here we document the design decisions and
parts that are changing.
The most notable change concerns peer ids: In order to avoid location tracking,
they are being made non-permanent - they change with each change of underlying
addresses. Next to that, the cryptographic primitives in use change, and the
interface to the (transport) layers below. The interface to the underlying
layers is made more generic so that libp2p can be switched in for gnunet's own
transport (layer 2 overlay/L2O). Finally, protocol-versioning above core will
be introduced.

@c TODO (from project plan)
@c - cryptographic protocol:
@c   - key exchange
@c   - key management

@c TODO write a short overview
@c - peer ids
@c - libp2p
@c - protocol versioning

@c Design goals
@c ------------
@c  - limit tracking

@menu
* Key exchange:: 
* Unified Address Format for L2O and libp2p:: 
* Peer IDs:: 
* libp2p Underlay:: 
* Protocol-Versioning:: 
* Open Design Questions:: 
* Requirements:: 
* Use Cases and Scenarios:: 

@end menu

@node Key exchange,Unified Address Format for L2O and libp2p,,CONG
@anchor{developers/apis/cong key-exchange}@anchor{1ca}
@subsubsection Key exchange


While we are at it we may as well improve the key exchange
(The-CORE-Peer_002dto_002dPeer-Protocol).
Currently, we are using our own ECDHE key exchange that derives 2x2 keys.
2 keys for each direction (sending/receiving).
Each direction uses two 256-bit symmetric encryption keys derived through the ECDH exchange.
Each payload is encrypted using AES(kA, Twofish(kB, payload)) both in CFB mode (!).

@c TODO Ephemeral key derivation, material sent, checks, ...
@c TODO Initiate handshake

@menu
* Next Steps:: 
* Handshake Protocol (Current): Handshake Protocol Current. 
* Draft; CORE Authenticated Key Exchange (CAKE): Draft CORE Authenticated Key Exchange CAKE. 

@end menu

@node Next Steps,Handshake Protocol Current,,Key exchange
@anchor{developers/apis/cong next-steps}@anchor{1cb}
@subsubsection Next Steps


For CONG, we should double-check the security of your ECDHE construction.
We decided on 11/03/2024 to investigate XChaCha20-Poly1305:

Proposal:

@quotation


@itemize *

@item 
Use X25519 for the KX with ephemeral Curve25519 keys.

@item 
Use XChaCha20-Poly1305 and kTx,kRx := KDF(X25519(),senderPK,receiverPK) for symmetric encryption
@end itemize
@end quotation

We will have to replace the use of @code{GNUNET_CRYPTO_symmetric_encrypt} and
HMAC use in @code{gnunet-service-core_kx.c} including the respective keys and IVs.

@node Handshake Protocol Current,Draft CORE Authenticated Key Exchange CAKE,Next Steps,Key exchange
@anchor{developers/apis/cong handshake-protocol-current}@anchor{1cc}
@subsubsection Handshake Protocol (Current)


@cartouche
@quotation Warning 
This is incomplete as the protocol is very messy and has around 6 RTTs
@end quotation
@end cartouche

We assume that the peers have semi-`static' (as opposed to ephemeral) key pairs.
Let (pk@w{[A]},sk@w{[A]}) be the key pair of peer PID@w{[A]} and (pk@w{[B]},sk@w{[B]}) the
key pair of peer PID@w{[B]}.

For any secure handshake protocol, we have to dermine an initiator and a receiver in the protocol.
We use @cite{GNUNET_CRYPTO_hash_cmp} to determine which peer is the receiver @cite{R} and which peer the initiator @cite{I}:

@example
if (GNUNET_CRYPTO_hash_cmp (pk_A, pk_B))
@{
  pk_I = pk_A
  pk_R = pk_B
@}
else
@{
  pk_I = pk_B
  pk_R = pk_A
@}
@end example

It is possible that the designated initiator does not initiate the handshake. After a pre-determined timeout,
the respective other peer may initiate.

We assume that the initiator knows pk@w{[R]} (pre-distributed through HELLO, for example).

@code{I} and @code{R} calculate `before any connection attempt is made':


@itemize *

@item 
(pk@w{[e]},sk@w{[e]}) <- `KeyGen'()
@end itemize

@cartouche
@quotation Danger 
Yes, both peers calculate `ephemeral' keys that are used for a set period of time in `all' handshakes.
@end quotation
@end cartouche

@code{I} calculates:


@itemize *

@item 
@code{EphemeralKeyMessage} <- (pk@w{[I]}, pk@w{[e]}, creation_time, ...)

@item 
sig@w{[e]} <- `Sign'(sk@w{[I]}, @code{EphemeralKeyMessage})
@end itemize

@cartouche
@quotation I sends to R 
@code{EphemeralKeyMessage}, sig@w{[e]}
@end quotation
@end cartouche

@code{R} calculates:


@itemize *

@item 
assert `Verify'(pk@w{[R]}, @code{EphemeralKeyMessage}, sig@w{[e]})

@item 
Establish session keys through ECDH with `ephemeral' keys.

@item 
@code{EphemeralKeyMessage} <- (pk@w{[R]}, pk@w{[e]}, creation_time, ...)

@item 
sig@w{[e]} <- `Sign'(sk@w{[R]}, @code{EphemeralKeyMessage})
@end itemize

@cartouche
@quotation R sends to I 
@code{EphemeralKeyMessage}, sig@w{[e]}
@end quotation
@end cartouche

@code{I} calculates:


@itemize *

@item 
assert `Verify'(pk@w{[R]}, @code{EphemeralKeyMessage}, sig@w{[e]})

@item 
Establish session keys through ECDH with `ephemeral' keys.
@end itemize

@cartouche
@quotation I sends to R 
@code{PingMessage}
@end quotation
@end cartouche

@code{R} calculates:


@itemize *

@item 
Pong message
@end itemize

@cartouche
@quotation R sends to I 
@code{PongMessage}
@end quotation
@end cartouche

@node Draft CORE Authenticated Key Exchange CAKE,,Handshake Protocol Current,Key exchange
@anchor{developers/apis/cong draft-core-authenticated-key-exchange-cake}@anchor{1cd}
@subsubsection Draft: CORE Authenticated Key Exchange (CAKE)



@subsubheading See also


This protocol is derived from KEMTLS@footnote{https://thomwiggers.nl/publication/thesis/thesis.pdf} (page 81ff).


The initiator selection remains unchanged from the above protocol.

@menu
* First Message (RTT=0): First Message RTT=0. 
* Second Message (RTT=0.5): Second Message RTT=0 5. 
* Third Message (RTT=1.5): Third Message RTT=1 5. 
* Confirmation (RTT=1.5): Confirmation RTT=1 5. 
* Rekey / Service Status change:: 
* Edge Cases:: 
* Glossary:: 

@end menu

@node First Message RTT=0,Second Message RTT=0 5,,Draft CORE Authenticated Key Exchange CAKE
@anchor{developers/apis/cong first-message-rtt-0}@anchor{1ce}
@subsubsection First Message (RTT=0)


@code{I} sends to @code{R} the following message:

@code{MessageHeader||InitiatorHello||EncryptedInitiatorCert}

where:

@code{MessageHeader} is a @code{GNUNET_MessageHeader} of type `TBD' (GANA registration).

The @code{InitiatorHello} consists of:


@itemize *

@item 
(pk@w{[e]}, c@w{[R]}, SHA512(pk@w{[R]}), r@w{[I]}, [SupportedAlgs,Version])
@end itemize

r@w{[I]} is nonce value (256 or 512 bit TBD).
pk@w{[e]} is the public key from a freshly generated ephemeral key pair:


@itemize *

@item 
(pk@w{[e]},sk@w{[e]}) <- `KeyGen'()
@end itemize

In GNUnet this @code{KeyGen} corresponds to @code{GNUNET_CRYPTO_ecdhe_key_create()}. This is a X25519 key pair.
The @code{InitiatorHello} may contain our version and other metadata which
is indicated by the brackets. But, we may want to put this information into the @code{cert} below, as this is encrypted
(identity hiding).

The @code{EncryptedInitiatorCert} is created as


@itemize *

@item 
`Enc'(`ETS', cert [pk@w{[I]}])
@end itemize

We may encode the capabilities/supported class in @code{cert}.
We do not want to use X.509 here, probably. @code{cert} is just a placeholder for the
signed metadata (e.g. supported services string) and pk@w{[I]}.

@cartouche
@quotation Certificate definition 
The definition of certificate is incomplete.
For now, we only use self-signed certificates: The cert is signed using sk@w{[I]}. Use @code{GNUNET_CRYPTO_eddsa_sign} API for this.
@end quotation
@end cartouche

@cartouche
@quotation Supported services string 
Supported services are just a series of `service':`version' strings
separated by a separator. We may want to use libtool versioning?
@end quotation
@end cartouche

`Enc' is XChaCha20-Poly1305 (IETF version).
To derive `ETS' we create our first shared secret as:


@itemize *

@item 
(ss@w{[R]},c@w{[R]}) <- `Encaps'(pk@w{[R]})
@end itemize

pk@w{[R]} is the EdDSA public key of the peer we want to connect to.
We should have received this as part of the trigger for this message.
In GNUnet, @code{Encaps} corresponds to @code{GNUNET_CRYPTO_eddsa_kem_encaps}.

We then derive our encryption keys:


@itemize *

@item 
`ES' <- `HKDF-Extract'(ss@w{[R]}, 0)

@item 
`ETS' <- `HKDF-Expand'(`ES', @code{"early data"}, @code{InitiatorHello})
@end itemize

and encrypt the certificate giving us the @code{EncryptedInitiatorCert}.

@node Second Message RTT=0 5,Third Message RTT=1 5,First Message RTT=0,Draft CORE Authenticated Key Exchange CAKE
@anchor{developers/apis/cong second-message-rtt-0-5}@anchor{1cf}
@subsubsection Second Message (RTT=0.5)


@code{R} the receives:

@code{MessageHeader||InitiatorHello||EncryptedInitiatorCert}

and sends to @code{I} the following message:

@code{MessageHeader||ReceiverHello||EncryptedExtensions||ReceiverKemCiphertext||ReceiverFinished}

The message may have already application payload appended, but in our case this is unlikely.

First @code{R} processes the message received from @code{I}:


@itemize *

@item 
Verify that the message type is `TBD'

@item 
Decrypt @code{EncryptedInitiatorCert} (cf. encryption above):


@itemize *

@item 
(ss@w{[R]}) <- Decaps(sk@w{[R]}, c@w{[R]})

@item 
`ES' <- `HKDF-Extract'(ss@w{[R]}, 0)

@item 
`ETS' <- `HKDF-Expand'(`ES', "early data", @code{InitiatorHello})

@item 
cert [pk@w{[I]}] <- `Dec'(`ETS', @code{EncryptedInitiatorCert})
@end itemize
@end itemize

@cartouche
@quotation Encaps X25519 
Here, @code{Decaps} corresponds to @code{GNUNET_CRYPTO_eddsa_kem_decaps}.
@end quotation
@end cartouche


@itemize *

@item 
Setup Handshake and Master Secrets (these can also be done as-needed and not necessarily all at once here):


@itemize *

@item 
`dES' <- `HKDF-Expand'(`ES', "derived", @code{NULL})

@item 
(ss@w{[e]},c@w{[e]}) <- `Encaps'(pk@w{[e]})

@item 
`HS' <- `HKDF-Extract'(ss@w{[e]}, `dES')

@item 
`dHS' <-  `HKDF-Expand'(`HS', "derived", @code{NULL})

@item 
(ss@w{[I]},c@w{[I]}) <- @code{Encaps}(pk@w{[I]})

@item 
`MS' <- `HKDF-Extract'(ss@w{[I]}, `dHS')
@end itemize
@end itemize

@cartouche
@quotation Encaps X25519 
Here, @code{Encaps} corresponds to @code{GNUNET_CRYPTO_hpke_kem_encaps}.
@end quotation
@end cartouche


@itemize *

@item 
Derive Handshake Traffic Encryption Keys:


@itemize *

@item 
@code{ReceiverHello} <- (c@w{[e]}, r@w{[R]}, [SelectedAlgs])

@item 
`IHTS' <- `HKDF-Expand'(`HS', "i hs traffic", @code{InitiatorHello...ReceiverHello})

@item 
`RHTS' <- `HKDF-Expand'(`HS', "r hs traffic", @code{InitiatorHello...ReceiverHello})

@item 
@code{ReceiverKemCiphertext} <- `Enc'(`RHTS', c@w{[I]})

@item 
(Optional) @code{EncryptedExtensions} <- `Enc'(`RHTS', SupportedAlgs/Services?)
@end itemize

@item 
Build @code{ReceiverFinished} message:


@itemize *

@item 
fk@w{[R]} <- `HKDF-Expand'(`MS', "r finished", @code{NULL})

@item 
@code{RF} <- `HMAC'(fk@w{[R]}, @code{InitiatorHello...ReceiverKemCiphertext})

@item 
@code{ReceiverFinished} <- `Enc'(`RHTS', @code{RF}) (TLS1.3-style explicit authentication of receiver after 1RTT!)
@end itemize

@item 
Derive Application Traffic Encryption Key:


@itemize *

@item 
`RATS' <- `HKDF-Expand'(`MS', "r ap traffic", @code{InitiatorHello...ReceiverFinished})
@end itemize
@end itemize

@node Third Message RTT=1 5,Confirmation RTT=1 5,Second Message RTT=0 5,Draft CORE Authenticated Key Exchange CAKE
@anchor{developers/apis/cong third-message-rtt-1-5}@anchor{1d0}
@subsubsection Third Message (RTT=1.5)


@code{I} receives:

@code{MessageHeader||ReceiverHello||EncryptedExtensions||ReceiverKemCiphertext||ReceiverFinished}

and sends to @code{I} the following message:

@code{MessageHeader||IteratorFinished}

The message may have already application payload appended, but in our case this again is unlikely.

First @code{I} processes the message received from @code{R}:


@itemize *

@item 
Verify that the message type is `TBD'

@item 
Setup Master Secret (cf. derivation in Second Message):


@itemize *

@item 
(ss@w{[e]}) <- `Decaps'(sk@w{[e]}, c@w{[e]}) (X25519 KEM)

@item 
`dES' <- `HKDF-Expand'(`ES', @code{"derived"}, @code{NULL})

@item 
`HS' <- `HKDF-Extract'(ss@w{[e]}, `dES')

@item 
`dHS' <- `HKDF-Expand'(`HS', "derived", @code{NULL})

@item 
(ss@w{[I]}) <- `Decaps'(sk@w{[I]}, c@w{[I]}) (EdDSA KEM)

@item 
`MS' <- `HKDF-Extract'(ss@w{[I]}, `dHS')
@end itemize

@item 
Derive Traffic Encryption Keys (these can also be done as-needed and not necessarily all at once here):


@itemize *

@item 
`IHTS' <- `HKDF-Expand'(`HS', "i hs traffic", @code{InitiatorHello...ReceiverHello})

@item 
`RHTS' <- `HKDF-Expand'(`HS', "r hs traffic", @code{InitiatorHello...ReceiverHello})

@item 
`IATS' <- `HKDF-Expand'(`MS', "i ap traffic", @code{InitiatorHello...InitiatorFinished})

@item 
`RATS' <- `HKDF-Expand'(`MS', "r ap traffic", @code{InitiatorHello...ReceiverFinished})
@end itemize

@item 
Build @code{ReceiverFinished} and @code{InitiatorFinished} plain texts:


@itemize *

@item 
fk@w{[I]} <- `HKDF-Expand'(`MS', "i finished", NULL)

@item 
@code{IF} <- `HMAC'(fk@w{[I]}, @code{InitiatorHello...ReceiverFinished})

@item 
fk@w{[R]} <- `HKDF-Expand'(`MS', "r finished", NULL)

@item 
@code{RF} <- `Dec'(RHTS, @code{ReceiverFinished})

@item 
assert `HMAC'(fk@w{[R]}, @code{InitiatorHello...ReceiverKemCiphertext}) == @code{RF}
@end itemize

@item 
@code{InitiatorFinished} <- `Enc'(`IHTS', @code{IF})
@end itemize

@node Confirmation RTT=1 5,Rekey / Service Status change,Third Message RTT=1 5,Draft CORE Authenticated Key Exchange CAKE
@anchor{developers/apis/cong confirmation-rtt-1-5}@anchor{1d1}
@subsubsection Confirmation (RTT=1.5)


@code{R} receives @code{IteratorFinished} and computes:


@itemize *

@item 
@code{IF} <- `Dec'(IHTS, @code{InitiatorFinished})

@item 
fk@w{[I]} <- `HKDF-Expand'(`MS', "i finished", @code{NULL})

@item 
assert `HMAC'(fk@w{[I]}, @code{InitiatorHello...ReceiverFinished}) == @code{IF}

@item 
`IATS' <- `HKDF-Expand'(`MS', "i ap traffic", @code{InitiatorHello...InitiatorFinished})
@end itemize

At this point we have a secure channel and application payload can be en/decrypted using `IATS' and `RATS', respectively.

@node Rekey / Service Status change,Edge Cases,Confirmation RTT=1 5,Draft CORE Authenticated Key Exchange CAKE
@anchor{developers/apis/cong rekey-service-status-change}@anchor{1d2}
@subsubsection Rekey / Service Status change


TODO

@node Edge Cases,Glossary,Rekey / Service Status change,Draft CORE Authenticated Key Exchange CAKE
@anchor{developers/apis/cong edge-cases}@anchor{1d3}
@subsubsection Edge Cases



@itemize *

@item 
The Initiator/Receiver selection logic may require a timed fallback: The designates Initiator may never initiate (NAT, already has sufficient connections, learns about receiver later than receiver about initiator etc.)

@item 
This may result in edge cases where the Initiator initiates a handshake and the Receiver also initiates a handshake at the same time switching roles.

@item 
In such cases we may simply do both key exchanges. If both succeed, we drop the key exchange that was not initiated by the designated initiator on both peers. Otherwise we use the successful key exchange and the roles are swapped.
@end itemize

@node Glossary,,Edge Cases,Draft CORE Authenticated Key Exchange CAKE
@anchor{developers/apis/cong glossary}@anchor{1d4}
@subsubsection Glossary


@quotation


@itemize *

@item 
`IATS': Initiator Application Traffic Secret Key

@item 
`RATS': Receiver Application Traffic Secret Key

@item 
`dES': Derived Early Secret Key

@item 
`dHS': Derived Handshake Key

@item 
`ES': Early Secret Key

@item 
`ETS': Early Traffic Secret Key

@item 
`HS': Handshake Secret Key

@item 
`MS': Main Secret Key

@item 
`ES': Early Secret Key

@item 
`IHTS': Initiator Handshake Secret Key

@item 
`RHTS': Receiver Handshake Secret Key

@item 
@code{Foo...Bar} means the transcript of received/send messages from @code{Foo} until @code{Bar}
@end itemize
@end quotation

@node Unified Address Format for L2O and libp2p,Peer IDs,Key exchange,CONG
@anchor{developers/apis/cong unified-address-format-for-l2o-and-libp2p}@anchor{1d5}
@subsubsection Unified Address Format for L2O and libp2p


@c TODO
@c libp2p peer-id multiaddress?, gnunet-hello
@c https://github.com/libp2p/specs/blob/master/peer-ids/peer-ids.md
@c https://github.com/libp2p/specs/blob/master/addressing/README.md#multiaddr-basics
@c https://docs.gnunet.org/latest/users/subsystems.html#hellos
@c https://docs.gnunet.org/latest/users/subsystems.html#hello-ng
@c https://docs.libp2p.io/concepts/fundamentals/addressing/
@c https://github.com/multiformats/multiaddr/

As a unified address format for L2O and libp2p we will use a concatenation of
the string representations of gnunet's hellos and libp2p's multiaddress,
separated by @cite{;;}.
For example: @cite{gnunet://hello/XXPIDXX/XXPIDXX/1725622944?udp=%5B%3A%3A1%5D%3A2086&;;libp2p:///ip4/127.0.0.1/tcp/24915}

This is only for the time being. For the long run the integration within each
other's addressing schemes should be evaluated. Meaning: Integrate a
gnunet-hello address type in libp2p's multiaddress format and integrate the
multiaddress format with the gnunet-hello.

@node Peer IDs,libp2p Underlay,Unified Address Format for L2O and libp2p,CONG
@anchor{developers/apis/cong id1}@anchor{1d6}@anchor{developers/apis/cong peer-ids}@anchor{1d7}
@subsubsection Peer IDs


Peer ids stop to be unique for the lifetime of a peer, but change each time a
peer's addresses change. This includes gaining or losing an address.

It is important to note that this design choice only increases the cost of
network location tracking and does not fully prevent it. For this feature onion
routing on top of CADET is envisioned.

At this point it seems like one has to weigh privacy versus performance when it
comes to this design decision.

@menu
* Reasoning:: 
* Implications:: 
* Details on how:: 
* Open questions:: 
* Ownership of Peer IDs:: 
* Implications: Implications<2>. 
* Details on how: Details on how<2>. 

@end menu

@node Reasoning,Implications,,Peer IDs
@anchor{developers/apis/cong id2}@anchor{1d8}@anchor{developers/apis/cong reasoning}@anchor{1d9}
@subsubsection Reasoning


@c - why this decision (and not others?)
@c         - (scenarios to understand whether it makes sense)

This change was introduced in order to stop tracking of more mobile peers
across the network. For example a more mobile peer (laptop) that logs into the
network at different places can be easily tracked by everyone just by recording
the different addresses that are tied to that peer id over time.

@c TODO
@c  - why does this prevent tracking? (partially answered in the attacker
@c    model section below
@c  - tracking hellos of peer: visible
@c  - reverse mapping (address to peer?) not possible - there's no functionality
@c    for that
@c  - gaining/losing as criterium to change peer id makes sense?

@menu
* Attacker Model:: 

@end menu

@node Attacker Model,,,Reasoning
@anchor{developers/apis/cong attacker-model}@anchor{1da}@anchor{developers/apis/cong id3}@anchor{1db}
@subsubsection Attacker Model


An attacker observes the hellos (containing ip and other addresses) published
under a peer id and is thus capable of tracking locations and thus obtaining a
movement profile of this peer.

With the proposed changes to the peer id an attacker can only see a peer id and
its connected set of addresses. A movement profile can not be obtained in the
previous way.

@c TODO how much info can be gained by that? all an observing peer sees is the
@c ip addresses. how much can this tell?
@c - mobile provider (implicit: mobile connection)
@c - internet provider - home, company, public, ...
@c - region (country, city, building, ...)

Tracking of addresses/locations might still be possible in the scenario that a
mobile client uses mobile broadband and wifi uplinks and uses them in an
'overlapping' manner. (Switching on mobile broadband before leaving the range
of a wifi hotspot and vice versa Turning off mobile broadband after connecting
to a wifi.) For the 'overlapping' time the peer would publish a HELLO
containing the old and the new network addresses. After the overlapping time,
the peer's HELLO would just contain the new network address, which was already
in the HELLO from during the 'overlap'. That way the overlap can be used to
link the old and new address and - in extreme scenarios - obtain the full
movement profile again. Note that this does not work on all, not all the time
and requires work for the correlation. It should be highlighted that, although
this attack is possible, the new design still greatly reduces the attack
surface.

A way to circumvent this anti-tracking mechanism would be for an attacker to
exploit the means for consistently connect to the same peer.
.. TODO wording/conceptualisation - overlap with next sentence
For example with the means of higher-layer services like gns. With the
knowledge of the gns entry, its peer ids and thus its addresses can be fully
tracked. Until the existence of an onion-routing service and its decoupling of
identities and network addresses, this behavior is probably intended and maybe
the only way to connect endpoints. Although this leak is still existent the
new design still prevents a lot of other leakage. Also it is important to point
out that it is impossible to prevent leakage at higher layers, but necessary to
prevent leakage at this layer, because leakage at this level could not be fixed
at higher layers.

@c TODO review for wording/clarity

@node Implications,Details on how,Reasoning,Peer IDs
@anchor{developers/apis/cong implications}@anchor{1dc}@anchor{developers/apis/cong implications-peerids}@anchor{1dd}
@subsubsection Implications


Here we present the implications for different parts of the framework. This
should help get a grasp of the implications of the change.
.. TODO poor phrasing?

@menu
* DHT:: 
* Scope of Peer ID for higher Layers:: 
* Reconnects:: 
* Messenger: Messenger<2>. 

@end menu

@node DHT,Scope of Peer ID for higher Layers,,Implications
@anchor{developers/apis/cong dht}@anchor{1de}@anchor{developers/apis/cong id4}@anchor{1df}
@subsubsection DHT


The DHT uses the peer id such that it determines which buckets a peer is
responsible for. So each time the peer id changes, the peer becomes responsible
for different data. This means that each time a peer changes address, it leaves
and re-enters the DHT, changes its and other peer's neighbors and changes the
stored data.

@node Scope of Peer ID for higher Layers,Reconnects,DHT,Implications
@anchor{developers/apis/cong id5}@anchor{1e0}@anchor{developers/apis/cong scope-of-peer-id-for-higher-layers}@anchor{1e1}
@subsubsection Scope of Peer ID for higher Layers


When peer ids stop to be unique over time, the framework is in lack of a
globally unique identifier. Higher layers may rely (have relied) on the
uniqueness of the ids. This means gnunet has to use other means for this
purpose. The @ref{1e2,,Reconnects} section below is concerned with the specific impact on
reconnects for different higher-layer services. In general gns/identity offers
this functionality.

As peer ids cease to be unique over time, this might be a good point to review
the scope of its and other elements' usage and terminology. (See
@ref{1e3,,Open_Design_Questions})

@c FIXME This section overlaps in scope with the next section.

@node Reconnects,Messenger<2>,Scope of Peer ID for higher Layers,Implications
@anchor{developers/apis/cong id6}@anchor{1e4}@anchor{developers/apis/cong reconnects}@anchor{1e2}
@subsubsection Reconnects


When addresses (and with those the peer id) of a peer change, all core
connections need to be torn down and with them all higher-layer connections.
This affects the layers above CADET as follows:

@quotation


@itemize -

@item 
Revocation: Is not really affected as it is only connected to direct CADET
neighbors and makes no use of CADET's routing, only of its flow and
congestion control.

@item 
File sharing: Only the non-anonymous filesharing uses CADET connections.
This is not significantly affected by a reconnect as it only looks up peer
and port in the DHT, so in the meanwhile it's looking for other peers. TODO
this is very unclear to me!

@item 
Messenger: All CADET connections would break and the peer might assume that
all previously connected peers went offline. So it would require a
mechanism to reconnect to peers with a known peer identity which offer
routing capability (open port via CADET to connect to). In case the peer
itself is providing such capability, it would help to know about peer ID
changes ahead of time to communicate a switch between IDs to other peers.
For other reconnections via GNS lookups are required.

@item 
Conversation: The call would be interrupted until the new peer id of the
other has been found (potentially via GNS).
@end itemize
@end quotation

Open question: is gns needed for a reconnect? Could the peer with the new
id not simply 'call back' the other peer? Of course this would only work if
only one peer changes its peer id. If both change their peer id at the same
time, an external mechanism would be needed.

See @ref{1e3,,Open_Design_Questions} for thoughts on good designs to handle address
changes more smoothly.

@node Messenger<2>,,Reconnects,Implications
@anchor{developers/apis/cong messenger}@anchor{1e5}@anchor{developers/apis/cong messenger-implications}@anchor{1e6}
@subsubsection Messenger


The current implementation of messenger heavily relies on a globally unique
peer id. The change requires messenger to account for peer id changes.

@node Details on how,Open questions,Implications,Peer IDs
@anchor{developers/apis/cong details-on-how}@anchor{1e7}@anchor{developers/apis/cong details-on-how-peerids}@anchor{1e8}
@subsubsection Details on how


Peer ids will be generated from the set of a peer's current addresses. Once a
peer obtains the same set of addresses it shall be using the same peer id. To
achieve this behavior, the current string representation of addresses is
sorted, the sorted representation hashed and the peer id generated from the
hash. We call the hash 'network generation hash'. The hash can be used to
quickly identify and recognise a set of addresses that was used in the past.

Once a peer id changes, all communication via open channels shall immediately
cease. To signal this, the mq is to be used.
To identify all queued messages that are to be cancelled, the 'network
generation id' is used. (This was decided to be implemented as a simple counter
of 64 bit.) With each address and peer id change, the network generation id is
incremented. When enqueueing a message with the mq, the current generation id
is stored with is. This way, when the generation id changes, all enqueued
messages with the old id (which still might refer to the old peer id in some
way) can be identified and dequeued. It is important to manage the network
generation id as close to the communicators as possible to be able to stop the
actual outgoing messages as quick as possible.

To be more precise, the communication via open channels should not cease when
the new peer id is available, but already when there's a change to the
addresses in use. This means there is a small window, until the new peer id is
generated, in which the peer is without peer id and thus without means to
communicate. This might open a new attack window by trying to change the
addresses of a peer via for example opening and closing wifi hotspots and
sending out new addresses via dhcp.

@c 64 bit: similar size to pid/hash

@c - adress bundle (transport communicators implement decision?)
@c - I vaguely remember the talk of 'insignificant' address changes (within
@c   local network or such), that don't require a new peer id, but forgot the
@c   specifics.

@node Open questions,Ownership of Peer IDs,Details on how,Peer IDs
@anchor{developers/apis/cong id7}@anchor{1e9}@anchor{developers/apis/cong open-questions}@anchor{1ea}
@subsubsection Open questions


@c overall for this "peer id" section
@c Notes from the meeting:
@c  - higher-layer roll-over
@c        - problem in practice 'phone call not interrupted by change of
@c          location'
@c        - solved by gns entries? and re-connect?
@c        - cadet search for peer id - search in dht and via gns
@c        - caching-problem for gns
@c        - no signaling for roll-over
@c        - estimated reconnect within 15 minutes
@c        - change of address/peerid results in unclear state (down? lost
@c          connection, ....?)
@c        - cadet needs to regularily check gns entries
@c  - how signal the change of addresses?
@c        - do we signal trear-down?
@c        - scenario address change in internet cafe?
@c        - choice between privacy and connectivity
@c        - sense of changing ids for cadet?
@c                - -> rendez-vous peer (more stable)
@c        - tracking via gns?
@c                - only for people who are connected/known (tracking might be
@c                  ok for this)
@c                        - requires pir for gns
@c                - tracking impossible/hard for others

@node Ownership of Peer IDs,Implications<2>,Open questions,Peer IDs
@anchor{developers/apis/cong id8}@anchor{1eb}@anchor{developers/apis/cong ownership-of-peer-ids}@anchor{1ec}
@subsubsection Ownership of Peer IDs


When the peer id was static, all parts of gnunet had a simple way to interface
with it. Once it becomes dynamic, it makes a lot of sense that a single part
takes control/responsibility/ownership for it. A new service is created for
this purpose. A name suggestion was "peer id lifecycle service" - PILS.

The reasons for this new service:

@quotation


@itemize -

@item 
Good encapsulation (which is even more important as it is a component deals
with crucial cryptographic operation).

@item 
Avoidance of circular subscriptions of core and transport.

@item 
Avoidance of callback api hell between core and transport.

@item 
Using peerstore for this would be really messy.
@end itemize
@end quotation


@table @asis

@item The reasons against this new service:


@itemize -

@item 
Having yet another service.
@end itemize
@end table

Core will be the responsible service to provide addresses to pils for the
switching of peer ids.

Peer ids should not be stored in permanent memory (on disk), but kept in
working memory.
They will be (re-) computed from the new set of addresses. The advantage is
that it is not possible to prove that a peer id belonged to a peer by
examining its hard-drive. Keeping the respective HELLO in the peerstore is fine
as a peer also keeps other peer's hello in there and the existence of a hello
in the peerstore proves nothing.

@node Implications<2>,Details on how<2>,Ownership of Peer IDs,Peer IDs
@anchor{developers/apis/cong id9}@anchor{1ed}@anchor{developers/apis/cong implications-onwnership-peerids}@anchor{1ee}
@subsubsection Implications


Pils needs to take ownership. It is responsible for generating, changing of
peer ids, informing subscribers of changes and signing with the current peer
id. (To be gentle on the ipc, it should not sign big amounts of data - if
applicable rather hashes of data or such.)

@menu
* Transport:: 
* Other:: 

@end menu

@node Transport,Other,,Implications<2>
@anchor{developers/apis/cong id10}@anchor{1ef}@anchor{developers/apis/cong transport}@anchor{1f0}
@subsubsection Transport


Up until now transport still creates, signs and puts hellos into the peerstore.
When pils takes ownership of peer ids, transport will need to ask it for
signing the hellos before putting it into the peerstore. (With libp2p as
alternative/additional underlay in mind, everything related to hellos needs to
move to core eventually as libp2p is oblivious about that part of gnunet. Core
will have to create and provide the hellos. See @ref{1f1,,libp2p-Underlay} below.)

@node Other,,Transport,Implications<2>
@anchor{developers/apis/cong id11}@anchor{1f2}@anchor{developers/apis/cong other}@anchor{1f3}
@subsubsection Other


Generally, all parts that so far read the peer id from a file into local
memory, will need to ask pils for it. All services that sign/encrypt something
with the peer id now will need to ask pils to do it.

@node Details on how<2>,,Implications<2>,Peer IDs
@anchor{developers/apis/cong details-on-how-ownership-peerids}@anchor{1f4}@anchor{developers/apis/cong id12}@anchor{1f5}
@subsubsection Details on how


In order to provide the needed functionality, pils needs to expose it via its
api:

@quotation


@itemize -

@item 
a call to provide the current set of addresses, so the generation of the
peer id can be triggered.

@item 
a call to obtain the current peer id

@item 
registering a handler that informs about new peer ids

@item 
signing data with the peer id
@end itemize
@end quotation

@node libp2p Underlay,Protocol-Versioning,Peer IDs,CONG
@anchor{developers/apis/cong id13}@anchor{1f6}@anchor{developers/apis/cong libp2p-underlay}@anchor{1f1}
@subsubsection libp2p Underlay


Get gnunet to work on libp2p. This includes the FFI with rust (or binding to
implementations in other languages), converting address formats, signaling
metadata (traffic class and priority, as far as libp2p supports it) and
signalling connectivity changes.  This is a first attempt to technically link
the two projects. Therefore the feasibility is quite uncertain and the
milestones might have to be re-evaluated after the report on the needs and
feasibility. Should it turn out that the needed resources are beyond the
capabilities of this project, a detailed report on the requirements and roadmap
of and for the realisation shall be written and published.

For a lot of practical purposes it would be a lot easier to add a new
communicator for libp2p instead of integrating it as an underlay below core,
next to transport. The downside would be that there might be a lot of
duplication of functionality.

@c TODO this generally needs more love

@c - addresses from underlay in string representation go into core
@c - libp2p can't handle hellos/peerstore
@c - address signaling between transport and core
@c - hellos should be handled by core

@node Protocol-Versioning,Open Design Questions,libp2p Underlay,CONG
@anchor{developers/apis/cong protocol-002dversioning}@anchor{1f7}@anchor{developers/apis/cong protocol-versioning}@anchor{1f8}
@subsubsection Protocol-Versioning


Currently applications signal to core which message types they support. For
this milestone we will implement proper protocol versioning where higher-level
applications can signal a range of protocol versions which they support (min,
max) and exchange messages at the CONG layer between peers to determine common
protocols.

@node Open Design Questions,Requirements,Protocol-Versioning,CONG
@anchor{developers/apis/cong id14}@anchor{1f9}@anchor{developers/apis/cong open-design-questions}@anchor{1e3}
@subsubsection Open Design Questions


In this section we list design question that are not decided on, yet.

@c - scope of peer ids and other such elements (gns/identity, ...)
@c - resolve gns to pid on lower layer? (cadet or even core)
@c    - convenience api: cadet connection to domain name
@c    - -> not the scope of cong

@menu
* Peer ID changes and connectivity:: 
* Peer ID Terminology:: 

@end menu

@node Peer ID changes and connectivity,Peer ID Terminology,,Open Design Questions
@anchor{developers/apis/cong id15}@anchor{1fa}@anchor{developers/apis/cong peer-id-changes-and-connectivity}@anchor{1fb}
@subsubsection Peer ID changes and connectivity


@c - smooth handling of address/pid change possible?/signaling change of address
@c    - should be possible at least when we gain one address
@c    - construct that allows an additional peer id that is pre-computed (not
@c      based on addresses) and announced ahead of address change?
@c    - other terminology: provide tracking capability selectively
@c    - use peer ids as long as there are open connections with them?

In case a peer's addresses change, it gets a new peer id and therefor needs to
reconnect. The challenge is to reconnect as fast as possible. The main problem
is that a peer cannot know its next peer id in all cases. Connections that have
dedicated peers at its endpoints will probably look up the new peer id of the
other peer in a higher-layer service, most probably gns.

In the case in which a peer just gains an additional address, that peer can
pre-calculate its next peer id, signal it via still open connections on the old
peer id and finally switch to using the new peer id.

Other more evolved ideas include using multiple peer ids per peer: Either an
additional address-independent peer id that will 'survive' address changes and
serve as means to link to the address-based peer id after a change. It would
just be sent to connected peers and reset once all connections have been
re-established.

Alternatively (maybe in addition) peers could use multiple address-based peer
ids - one per address. Thus some peer ids might stay unchanged while others go
offline.

@c - tracking even harder
@c - comes closer to equivalent of ip addresses
@c - possibly helps with (privacy-preserving) roll-overs
@c - we ceased having unique peer ids over time, why not cease to have a
@c   single address at a time?
@c - might serve as an 'onion-address light'?
@c - cleaner, less messy abstraction

Another idea to address this challenge is to keep peer ids in use on
connections which are still in use, but don't publish those ids anymore.

Terminology-wise we might add another perspective and say that we selectively
and deliberately provide tracking capability to peers which we want to stay in
touch with.

@node Peer ID Terminology,,Peer ID changes and connectivity,Open Design Questions
@anchor{developers/apis/cong id16}@anchor{1fc}@anchor{developers/apis/cong peer-id-terminology}@anchor{1fd}
@subsubsection Peer ID Terminology


Once peer ids cease to be unique over time the question raises whether they
should actually be called identities. (In my intuition an identity is something
more persistent.)
As we discuss peer ids as an analogue to ip addresses, a natural close idea
would be to use "peer address".

@node Requirements,Use Cases and Scenarios,Open Design Questions,CONG
@anchor{developers/apis/cong id17}@anchor{1fe}@anchor{developers/apis/cong requirements}@anchor{1ff}
@subsubsection Requirements


In the discussions we seem to have lost partial oversight over things.
In this section we figure out the requirements for core (and possibly other
components) it's mechanisms.

TODO: what requirements exactly did we want to document here?

@node Use Cases and Scenarios,,Requirements,CONG
@anchor{developers/apis/cong id18}@anchor{200}@anchor{developers/apis/cong use-cases-and-scenarios}@anchor{201}
@subsubsection Use Cases and Scenarios


This section is supposed to help with the understanding of the use of elements
and structures by imagined examples.

@menu
* Peer ID:: 
* GNS:: 
* CADET: CADET<2>. 
* Messenger: Messenger<3>. 

@end menu

@node Peer ID,GNS,,Use Cases and Scenarios
@anchor{developers/apis/cong id19}@anchor{202}@anchor{developers/apis/cong peer-id}@anchor{203}
@subsubsection Peer ID


The peer id has been used throughout gnunet as a very convenient means for many
purposes. It was used as globally unique and stable identifier of a peer. From
now on it should rather be treated as something more volatile and fitting for
its layer: an ip address. Below we collect the intended purposes and use cases
and also point out some uses for which it was not intended and point to other
means to achieve it.

@c - address change: shutdown vs. standby vs. ...

@node GNS,CADET<2>,Peer ID,Use Cases and Scenarios
@anchor{developers/apis/cong gns}@anchor{204}@anchor{developers/apis/cong id20}@anchor{205}
@subsubsection GNS


TODO

@c - identify on user level
@c - vpn record: globally unique id

@node CADET<2>,Messenger<3>,GNS,Use Cases and Scenarios
@anchor{developers/apis/cong cadet}@anchor{206}@anchor{developers/apis/cong id21}@anchor{207}
@subsubsection CADET


TODO

@node Messenger<3>,,CADET<2>,Use Cases and Scenarios
@anchor{developers/apis/cong id22}@anchor{208}@anchor{developers/apis/cong messenger-scenarios}@anchor{209}
@subsubsection Messenger


TODO

@c TODO 'tracking capability' via gns: shared secret
@c  - this a good place?
@c  - understand how that would work
@c  - hide multiple devices behind one identity

@c TODO Overall: Look at the project milestones to check coverage

@geindex Distributed hash table; subsystem
@geindex DHT; Distributed hash table
@anchor{developers/apis/dht distributed-hash-table-dev}@anchor{20a}
@node DHT<2>,FS,CONG,Service APIs
@anchor{developers/apis/dht doc}@anchor{20b}@anchor{developers/apis/dht dht}@anchor{20c}
@subsection DHT


@menu
* The API of libgnunetblock:: 
* Queries:: 
* Sample Code:: 
* Conclusion2:: 
* GET:: 
* PUT:: 
* MONITOR:: 
* DHT Routing Options:: 
* PUTting data into the DHT:: 
* GETting data from the DHT:: 
* Monitoring the DHT:: 
* Routing GETs or PUTs:: 
* PUTting data into the DHT: PUTting data into the DHT<2>. 
* GETting data from the DHT: GETting data from the DHT<2>. 

@end menu

@node The API of libgnunetblock,Queries,,DHT<2>
@anchor{developers/apis/dht the-api-of-libgnunetblock}@anchor{20d}
@subsubsection The API of libgnunetblock


The block library requires for each (family of) block type(s) a block
plugin (implementing @code{gnunet_block_plugin.h}) that provides basic
functions that are needed by the DHT (and possibly other subsystems) to
manage the block. These block plugins are typically implemented within
their respective subsystems. The main block library is then used to
locate, load and query the appropriate block plugin. Which plugin is
appropriate is determined by the block type (which is just a 32-bit
integer). Block plugins contain code that specifies which block types
are supported by a given plugin. The block library loads all block
plugins that are installed at the local peer and forwards the
application request to the respective plugin.

The central functions of the block APIs (plugin and main library) are to
allow the mapping of blocks to their respective key (if possible) and
the ability to check that a block is well-formed and matches a given
request (again, if possible). This way, GNUnet can avoid storing invalid
blocks, storing blocks under the wrong key and forwarding blocks in
response to a query that they do not answer.

One key function of block plugins is that it allows GNUnet to detect
duplicate replies (via the Bloom filter). All plugins MUST support
detecting duplicate replies (by adding the current response to the Bloom
filter and rejecting it if it is encountered again). If a plugin fails
to do this, responses may loop in the network.

@node Queries,Sample Code,The API of libgnunetblock,DHT<2>
@anchor{developers/apis/dht id1}@anchor{20e}@anchor{developers/apis/dht queries}@anchor{20f}
@subsubsection Queries


The query format for any block in GNUnet consists of four main
components. First, the type of the desired block must be specified.
Second, the query must contain a hash code. The hash code is used for
lookups in hash tables and databases and must not be unique for the
block (however, if possible a unique hash should be used as this would
be best for performance). Third, an optional Bloom filter can be
specified to exclude known results; replies that hash to the bits set in
the Bloom filter are considered invalid. False-positives can be
eliminated by sending the same query again with a different Bloom filter
mutator value, which parametrizes the hash function that is used.
Finally, an optional application-specific "eXtended query" (xquery)
can be specified to further constrain the results. It is entirely up to
the type-specific plugin to determine whether or not a given block
matches a query (type, hash, Bloom filter, and xquery). Naturally, not
all xquery's are valid and some types of blocks may not support Bloom
filters either, so the plugin also needs to check if the query is valid
in the first place.

Depending on the results from the plugin, the DHT will then discard the
(invalid) query, forward the query, discard the (invalid) reply, cache
the (valid) reply, and/or forward the (valid and non-duplicate) reply.

@node Sample Code,Conclusion2,Queries,DHT<2>
@anchor{developers/apis/dht id2}@anchor{210}@anchor{developers/apis/dht sample-code}@anchor{211}
@subsubsection Sample Code


The source code in `plugin_block_test.c' is a good starting point for
new block plugins --- it does the minimal work by implementing a plugin
that performs no validation at all. The respective `Makefile.am' shows
how to build and install a block plugin.

@node Conclusion2,GET,Sample Code,DHT<2>
@anchor{developers/apis/dht conclusion2}@anchor{212}@anchor{developers/apis/dht id3}@anchor{213}
@subsubsection Conclusion2


In conclusion, GNUnet subsystems that want to use the DHT need to define
a block format and write a plugin to match queries and replies. For
testing, the @code{GNUNET_BLOCK_TYPE_TEST} block type can be used; it
accepts any query as valid and any reply as matching any query. This
type is also used for the DHT command line tools. However, it should NOT
be used for normal applications due to the lack of error checking that
results from this primitive implementation.

libgnunetdht
@geindex libgnunet; dht
libgnunetdht
libgnunetdht
------------

The DHT API itself is pretty simple and offers the usual GET and PUT
functions that work as expected. The specified block type refers to the
block library which allows the DHT to run application-specific logic for
data stored in the network.

@node GET,PUT,Conclusion2,DHT<2>
@anchor{developers/apis/dht get}@anchor{214}@anchor{developers/apis/dht id4}@anchor{215}
@subsubsection GET


When using GET, the main consideration for developers (other than the
block library) should be that after issuing a GET, the DHT will
continuously cause (small amounts of) network traffic until the
operation is explicitly canceled. So GET does not simply send out a
single network request once; instead, the DHT will continue to search
for data. This is needed to achieve good success rates and also handles
the case where the respective PUT operation happens after the GET
operation was started. Developers should not cancel an existing GET
operation and then explicitly re-start it to trigger a new round of
network requests; this is simply inefficient, especially as the internal
automated version can be more efficient, for example by filtering
results in the network that have already been returned.

If an application that performs a GET request has a set of replies that
it already knows and would like to filter, it can
call @code{GNUNET_DHT_get_filter_known_results} with an array of hashes
over the respective blocks to tell the DHT that these results are not
desired (any more). This way, the DHT will filter the respective blocks
using the block library in the network, which may result in a
significant reduction in bandwidth consumption.

@node PUT,MONITOR,GET,DHT<2>
@anchor{developers/apis/dht id5}@anchor{216}@anchor{developers/apis/dht put}@anchor{217}
@subsubsection PUT


@cartouche
@quotation Todo 
inconsistent use of "must" above it's written "MUST"
@end quotation
@end cartouche

In contrast to GET operations, developers `must' manually re-run PUT
operations periodically (if they intend the content to continue to be
available). Content stored in the DHT expires or might be lost due to
churn. Furthermore, GNUnet's DHT typically requires multiple rounds of
PUT operations before a key-value pair is consistently available to all
peers (the DHT randomizes paths and thus storage locations, and only
after multiple rounds of PUTs there will be a sufficient number of
replicas in large DHTs). An explicit PUT operation using the DHT API
will only cause network traffic once, so in order to ensure basic
availability and resistance to churn (and adversaries), PUTs must be
repeated. While the exact frequency depends on the application, a rule
of thumb is that there should be at least a dozen PUT operations within
the content lifetime. Content in the DHT typically expires after one
day, so DHT PUT operations should be repeated at least every 1-2 hours.

@node MONITOR,DHT Routing Options,PUT,DHT<2>
@anchor{developers/apis/dht id7}@anchor{218}@anchor{developers/apis/dht monitor}@anchor{219}
@subsubsection MONITOR


The DHT API also allows applications to monitor messages crossing the
local DHT service. The types of messages used by the DHT are GET, PUT
and RESULT messages. Using the monitoring API, applications can choose
to monitor these requests, possibly limiting themselves to requests for
a particular block type.

The monitoring API is not only useful for diagnostics, it can also be
used to trigger application operations based on PUT operations. For
example, an application may use PUTs to distribute work requests to
other peers. The workers would then monitor for PUTs that give them
work, instead of looking for work using GET operations. This can be
beneficial, especially if the workers have no good way to guess the keys
under which work would be stored. Naturally, additional protocols might
be needed to ensure that the desired number of workers will process the
distributed workload.

@node DHT Routing Options,PUTting data into the DHT,MONITOR,DHT<2>
@anchor{developers/apis/dht dht-routing-options}@anchor{21a}@anchor{developers/apis/dht id8}@anchor{21b}
@subsubsection DHT Routing Options


There are two important options for GET and PUT requests:


@table @asis

@item GNUNET_DHT_RO_DEMULITPLEX_EVERYWHERE This option means that all

peers should process the request, even if their peer ID is not
closest to the key. For a PUT request, this means that all peers that
a request traverses may make a copy of the data. Similarly for a GET
request, all peers will check their local database for a result.
Setting this option can thus significantly improve caching and reduce
bandwidth consumption --- at the expense of a larger DHT database. If
in doubt, we recommend that this option should be used.

@item GNUNET_DHT_RO_RECORD_ROUTE This option instructs the DHT to record

the path that a GET or a PUT request is taking through the overlay
network. The resulting paths are then returned to the application
with the respective result. This allows the receiver of a result to
construct a path to the originator of the data, which might then be
used for routing. Naturally, setting this option requires additional
bandwidth and disk space, so applications should only set this if the
paths are needed by the application logic.

@item GNUNET_DHT_RO_FIND_PEER This option is an internal option used by

the DHT's peer discovery mechanism and should not be used by
applications.

@item GNUNET_DHT_RO_BART This option is currently not implemented. It may

in the future offer performance improvements for clique topologies.
@end table

@menu
* The DHT Client-Service Protocol:: 

@end menu

@node The DHT Client-Service Protocol,,,DHT Routing Options
@anchor{developers/apis/dht the-dht-client-002dservice-protocol}@anchor{21c}@anchor{developers/apis/dht the-dht-client-service-protocol}@anchor{21d}
@subsubsection The DHT Client-Service Protocol


@node PUTting data into the DHT,GETting data from the DHT,DHT Routing Options,DHT<2>
@anchor{developers/apis/dht id9}@anchor{21e}@anchor{developers/apis/dht putting-data-into-the-dht}@anchor{21f}
@subsubsection PUTting data into the DHT


To store (PUT) data into the DHT, the client sends a
@code{struct GNUNET_DHT_ClientPutMessage} to the service. This message
specifies the block type, routing options, the desired replication
level, the expiration time, key, value and a 64-bit unique ID for the
operation. The service responds with a
@code{struct GNUNET_DHT_ClientPutConfirmationMessage} with the same 64-bit
unique ID. Note that the service sends the confirmation as soon as it
has locally processed the PUT request. The PUT may still be propagating
through the network at this time.

In the future, we may want to change this to provide (limited) feedback
to the client, for example if we detect that the PUT operation had no
effect because the same key-value pair was already stored in the DHT.
However, changing this would also require additional state and messages
in the P2P interaction.

@node GETting data from the DHT,Monitoring the DHT,PUTting data into the DHT,DHT<2>
@anchor{developers/apis/dht getting-data-from-the-dht}@anchor{220}@anchor{developers/apis/dht id10}@anchor{221}
@subsubsection GETting data from the DHT


To retrieve (GET) data from the DHT, the client sends a
@code{struct GNUNET_DHT_ClientGetMessage} to the service. The message
specifies routing options, a replication level (for replicating the GET,
not the content), the desired block type, the key, the (optional)
extended query and unique 64-bit request ID.

Additionally, the client may send any number of
@code{struct GNUNET_DHT_ClientGetResultSeenMessage}s to notify the
service about results that the client is already aware of. These
messages consist of the key, the unique 64-bit ID of the request, and an
arbitrary number of hash codes over the blocks that the client is
already aware of. As messages are restricted to 64k, a client that
already knows more than about a thousand blocks may need to send several
of these messages. Naturally, the client should transmit these messages
as quickly as possible after the original GET request such that the DHT
can filter those results in the network early on. Naturally, as these
messages are sent after the original request, it is conceivable that the
DHT service may return blocks that match those already known to the
client anyway.

In response to a GET request, the service will send @code{struct
GNUNET_DHT_ClientResultMessage}s to the client. These messages
contain the block type, expiration, key, unique ID of the request and of
course the value (a block). Depending on the options set for the
respective operations, the replies may also contain the path the GET
and/or the PUT took through the network.

A client can stop receiving replies either by disconnecting or by
sending a @code{struct GNUNET_DHT_ClientGetStopMessage} which must contain
the key and the 64-bit unique ID of the original request. Using an
explicit "stop" message is more common as this allows a client to run
many concurrent GET operations over the same connection with the DHT
service --- and to stop them individually.

@node Monitoring the DHT,Routing GETs or PUTs,GETting data from the DHT,DHT<2>
@anchor{developers/apis/dht monitoring-dht}@anchor{222}@anchor{developers/apis/dht monitoring-the-dht}@anchor{223}
@subsubsection Monitoring the DHT


To begin monitoring, the client sends a
@code{struct GNUNET_DHT_MonitorStartStop} message to the DHT service. In
this message, flags can be set to enable (or disable) monitoring of GET,
PUT and RESULT messages that pass through a peer. The message can also
restrict monitoring to a particular block type or a particular key. Once
monitoring is enabled, the DHT service will notify the client about any
matching event using @code{struct GNUNET_DHT_MonitorGetMessage}s for GET
events, @code{struct GNUNET_DHT_MonitorPutMessage} for PUT events and
@code{struct GNUNET_DHT_MonitorGetRespMessage} for RESULTs. Each of these
messages contains all of the information about the event.

@menu
* The DHT Peer-to-Peer Protocol:: 

@end menu

@node The DHT Peer-to-Peer Protocol,,,Monitoring the DHT
@anchor{developers/apis/dht the-dht-peer-002dto-002dpeer-protocol}@anchor{224}@anchor{developers/apis/dht the-dht-peer-to-peer-protocol}@anchor{225}
@subsubsection The DHT Peer-to-Peer Protocol


@node Routing GETs or PUTs,PUTting data into the DHT<2>,Monitoring the DHT,DHT<2>
@anchor{developers/apis/dht id11}@anchor{226}@anchor{developers/apis/dht routing-gets-or-puts}@anchor{227}
@subsubsection Routing GETs or PUTs


When routing GETs or PUTs, the DHT service selects a suitable subset of
neighbours for forwarding. The exact number of neighbours can be zero or
more and depends on the hop counter of the query (initially zero) in
relation to the (log of) the network size estimate, the desired
replication level and the peer's connectivity. Depending on the hop
counter and our network size estimate, the selection of the peers maybe
randomized or by proximity to the key. Furthermore, requests include a
set of peers that a request has already traversed; those peers are also
excluded from the selection.

@node PUTting data into the DHT<2>,GETting data from the DHT<2>,Routing GETs or PUTs,DHT<2>
@anchor{developers/apis/dht id12}@anchor{228}@anchor{developers/apis/dht putting-data-into-the-dht2}@anchor{229}
@subsubsection PUTting data into the DHT


To PUT data into the DHT, the service sends a @code{struct PeerPutMessage}
of type @code{GNUNET_MESSAGE_TYPE_DHT_P2P_PUT} to the respective neighbour.
In addition to the usual information about the content (type, routing
options, desired replication level for the content, expiration time, key
and value), the message contains a fixed-size Bloom filter with
information about which peers (may) have already seen this request. This
Bloom filter is used to ensure that DHT messages never loop back to a
peer that has already processed the request. Additionally, the message
includes the current hop counter and, depending on the routing options,
the message may include the full path that the message has taken so far.
The Bloom filter should already contain the identity of the previous
hop; however, the path should not include the identity of the previous
hop and the receiver should append the identity of the sender to the
path, not its own identity (this is done to reduce bandwidth).

@node GETting data from the DHT<2>,,PUTting data into the DHT<2>,DHT<2>
@anchor{developers/apis/dht getting-data-from-the-dht2}@anchor{22a}@anchor{developers/apis/dht id13}@anchor{22b}
@subsubsection GETting data from the DHT


A peer can search the DHT by sending @code{struct PeerGetMessage}s of
type @code{GNUNET_MESSAGE_TYPE_DHT_P2P_GET} to other peers. In addition to
the usual information about the request (type, routing options, desired
replication level for the request, the key and the extended query), a
GET request also contains a hop counter, a Bloom filter over the peers
that have processed the request already and depending on the routing
options the full path traversed by the GET. Finally, a GET request
includes a variable-size second Bloom filter and a so-called Bloom
filter mutator value which together indicate which replies the sender
has already seen. During the lookup, each block that matches they block
type, key and extended query is additionally subjected to a test against
this Bloom filter. The block plugin is expected to take the hash of the
block and combine it with the mutator value and check if the result is
not yet in the Bloom filter. The originator of the query will from time
to time modify the mutator to (eventually) allow false-positives
filtered by the Bloom filter to be returned.

Peers that receive a GET request perform a local lookup (depending on
their proximity to the key and the query options) and forward the
request to other peers. They then remember the request (including the
Bloom filter for blocking duplicate results) and when they obtain a
matching, non-filtered response a @code{struct PeerResultMessage} of type
@code{GNUNET_MESSAGE_TYPE_DHT_P2P_RESULT} is forwarded to the previous hop.
Whenever a result is forwarded, the block plugin is used to update the
Bloom filter accordingly, to ensure that the same result is never
forwarded more than once. The DHT service may also cache forwarded
results locally if the "CACHE_RESULTS" option is set to "YES" in the
configuration.

@geindex File sharing; subsystem
@geindex FS; File sharing

@node FS,GNS<2>,DHT<2>,Service APIs
@anchor{developers/apis/fs doc}@anchor{22c}@anchor{developers/apis/fs fs}@anchor{22d}@anchor{developers/apis/fs fs-subsystem-dev}@anchor{22e}
@subsection FS


@menu
* Namespace Advertisements:: 
* KSBlocks:: 

@end menu

@node Namespace Advertisements,KSBlocks,,FS
@anchor{developers/apis/fs id1}@anchor{22f}@anchor{developers/apis/fs namespace-advertisements}@anchor{230}
@subsubsection Namespace Advertisements


@cartouche
@quotation Todo 
FIXME: all zeroses -> ?
@end quotation
@end cartouche

An @code{SBlock} with identifier all zeros is a signed advertisement for a
namespace. This special @code{SBlock} contains metadata describing the
content of the namespace. Instead of the name of the identifier for a
potential update, it contains the identifier for the root of the
namespace. The URI should always be empty. The @code{SBlock} is signed with
the content provider's RSA private key (just like any other SBlock).
Peers can search for @code{SBlock}s in order to find out more about a
namespace.

@node KSBlocks,,Namespace Advertisements,FS
@anchor{developers/apis/fs id3}@anchor{231}@anchor{developers/apis/fs ksblocks}@anchor{232}
@subsubsection KSBlocks


GNUnet implements @code{KSBlocks} which are @code{KBlocks} that, instead of
encrypting a CHK and metadata, encrypt an @code{SBlock} instead. In other
words, @code{KSBlocks} enable GNUnet to find @code{SBlocks} using the global
keyword search. Usually the encrypted @code{SBlock} is a namespace
advertisement. The rationale behind @code{KSBlock}s and @code{SBlock}s is
to enable peers to discover namespaces via keyword searches, and, to
associate useful information with namespaces. When GNUnet finds
@code{KSBlocks} during a normal keyword search, it adds the information to
an internal list of discovered namespaces. Users looking for interesting
namespaces can then inspect this list, reducing the need for out-of-band
discovery of namespaces. Naturally, namespaces (or more specifically,
namespace advertisements) can also be referenced from directories, but
@code{KSBlock}s should make it easier to advertise namespaces for the
owner of the pseudonym since they eliminate the need to first create a
directory.

Collections are also advertised using @code{KSBlock}s.

@c https://old.gnunet.org/sites/default/files/ecrs.pdf

@c What is this? - WGL

@menu
* File-sharing persistence directory structure:: 

@end menu

@node File-sharing persistence directory structure,,,KSBlocks
@anchor{developers/apis/fs file-002dsharing-persistence-directory-structure}@anchor{233}@anchor{developers/apis/fs file-sharing-persistence-directory-structure}@anchor{234}
@subsubsection File-sharing persistence directory structure


This section documents how the file-sharing library implements
persistence of file-sharing operations and specifically the resulting
directory structure. This code is only active if the
@code{GNUNET_FS_FLAGS_PERSISTENCE} flag was set when calling
@code{GNUNET_FS_start}. In this case, the file-sharing library will try
hard to ensure that all major operations (searching, downloading,
publishing, unindexing) are persistent, that is, can live longer than
the process itself. More specifically, an operation is supposed to live
until it is explicitly stopped.

If @code{GNUNET_FS_stop} is called before an operation has been stopped, a
@code{SUSPEND} event is generated and then when the process calls
@code{GNUNET_FS_start} next time, a @code{RESUME} event is generated.
Additionally, even if an application crashes (segfault, SIGKILL, system
crash) and hence @code{GNUNET_FS_stop} is never called and no @code{SUSPEND}
events are generated, operations are still resumed (with @code{RESUME}
events). This is implemented by constantly writing the current state of
the file-sharing operations to disk. Specifically, the current state is
always written to disk whenever anything significant changes (the
exception are block-wise progress in publishing and unindexing, since
those operations would be slowed down significantly and can be resumed
cheaply even without detailed accounting). Note that if the process
crashes (or is killed) during a serialization operation, FS does not
guarantee that this specific operation is recoverable (no strict
transactional semantics, again for performance reasons). However, all
other unrelated operations should resume nicely.

Since we need to serialize the state continuously and want to recover as
much as possible even after crashing during a serialization operation,
we do not use one large file for serialization. Instead, several
directories are used for the various operations. When
@code{GNUNET_FS_start} executes, the master directories are scanned for
files describing operations to resume. Sometimes, these operations can
refer to related operations in child directories which may also be
resumed at this point. Note that corrupted files are cleaned up
automatically. However, dangling files in child directories (those that
are not referenced by files from the master directories) are not
automatically removed.

Persistence data is kept in a directory that begins with the
"STATE_DIR" prefix from the configuration file (by default,
"$SERVICEHOME/persistence/") followed by the name of the client as
given to @code{GNUNET_FS_start} (for example, "gnunet-gtk") followed by
the actual name of the master or child directory.

The names for the master directories follow the names of the operations:


@itemize -

@item 
"search"

@item 
"download"

@item 
"publish"

@item 
"unindex"
@end itemize

Each of the master directories contains names (chosen at random) for
each active top-level (master) operation. Note that a download that is
associated with a search result is not a top-level operation.

In contrast to the master directories, the child directories are only
consulted when another operation refers to them. For each search, a
subdirectory (named after the master search synchronization file)
contains the search results. Search results can have an associated
download, which is then stored in the general "download-child"
directory. Downloads can be recursive, in which case children are stored
in subdirectories mirroring the structure of the recursive download
(either starting in the master "download" directory or in the
"download-child" directory depending on how the download was
initiated). For publishing operations, the "publish-file" directory
contains information about the individual files and directories that are
part of the publication. However, this directory structure is flat and
does not mirror the structure of the publishing operation. Note that
unindex operations cannot have associated child operations.

@geindex GNU Name System; subsystem
@geindex GNS; GNU Name System

@node GNS<2>,IDENTITY,FS,Service APIs
@anchor{developers/apis/gns doc}@anchor{235}@anchor{developers/apis/gns gns}@anchor{236}@anchor{developers/apis/gns gnu-name-system-dev}@anchor{237}
@subsection GNS


The GNS API itself is extremely simple. Clients first connect to the GNS
service using @code{GNUNET_GNS_connect}. They can then perform lookups
using @code{GNUNET_GNS_lookup} or cancel pending lookups using
@code{GNUNET_GNS_lookup_cancel}. Once finished, clients disconnect using
@code{GNUNET_GNS_disconnect}.

@menu
* Looking up records:: 
* Accessing the records:: 
* Creating records:: 
* Future work:: 
* Value handling:: 
* Type handling:: 
* Network Setup Details:: 
* Conversions between DNS and GNS:: 
* DNS Zone Size:: 
* Performance:: 

@end menu

@node Looking up records,Accessing the records,,GNS<2>
@anchor{developers/apis/gns id1}@anchor{238}@anchor{developers/apis/gns looking-up-records}@anchor{239}
@subsubsection Looking up records


@code{GNUNET_GNS_lookup} takes a number of arguments:


@table @asis

@item handle This is simply the GNS connection handle from

@code{GNUNET_GNS_connect}.

@item name The client needs to specify the name to

be resolved. This can be any valid DNS or GNS hostname.

@item zone The client

needs to specify the public key of the GNS zone against which the
resolution should be done. Note that a key must be provided, the
client should look up plausible values using its configuration, the
identity service and by attempting to interpret the TLD as a
base32-encoded public key.

@item type This is the desired GNS or DNS record type

to look for. While all records for the given name will be returned,
this can be important if the client wants to resolve record types
that themselves delegate resolution, such as CNAME, PKEY or GNS2DNS.
Resolving a record of any of these types will only work if the
respective record type is specified in the request, as the GNS
resolver will otherwise follow the delegation and return the records
from the respective destination, instead of the delegating record.

@item only_cached This argument should typically be set to

@code{GNUNET_NO}. Setting it to @code{GNUNET_YES} disables resolution via
the overlay network.

@item shorten_zone_key If GNS encounters new names during resolution,

their respective zones can automatically be learned and added to the
"shorten zone". If this is desired, clients must pass the private
key of the shorten zone. If NULL is passed, shortening is disabled.

@item proc This argument identifies

the function to call with the result. It is given proc_cls, the
number of records found (possibly zero) and the array of the records
as arguments. proc will only be called once. After proc,> has been
called, the lookup must no longer be canceled.
@end table

proc_cls The closure for proc.

@node Accessing the records,Creating records,Looking up records,GNS<2>
@anchor{developers/apis/gns accessing-the-records}@anchor{23a}@anchor{developers/apis/gns id2}@anchor{23b}
@subsubsection Accessing the records


The @code{libgnunetgnsrecord} library provides an API to manipulate the GNS
record array that is given to proc. In particular, it offers functions
such as converting record values to human-readable strings (and back).
However, most @code{libgnunetgnsrecord} functions are not interesting to
GNS client applications.

For DNS records, the @code{libgnunetdnsparser} library provides functions
for parsing (and serializing) common types of DNS records.

@node Creating records,Future work,Accessing the records,GNS<2>
@anchor{developers/apis/gns creating-records}@anchor{23c}@anchor{developers/apis/gns id3}@anchor{23d}
@subsubsection Creating records


Creating GNS records is typically done by building the respective record
information (possibly with the help of @code{libgnunetgnsrecord} and
@code{libgnunetdnsparser}) and then using the @code{libgnunetnamestore} to
publish the information. The GNS API is not involved in this operation.

@node Future work,Value handling,Creating records,GNS<2>
@anchor{developers/apis/gns future-work}@anchor{23e}@anchor{developers/apis/gns id4}@anchor{23f}
@subsubsection Future work


In the future, we want to expand @code{libgnunetgns} to allow applications
to observe shortening operations performed during GNS resolution, for
example so that users can receive visual feedback when this happens.

@menu
* libgnunetgnsrecord:: 

@end menu

@node libgnunetgnsrecord,,,Future work
@anchor{developers/apis/gns libgnunetgnsrecord}@anchor{240}
@subsubsection libgnunetgnsrecord


The @code{libgnunetgnsrecord} library is used to manipulate GNS records (in
plaintext or in their encrypted format). Applications mostly interact
with @code{libgnunetgnsrecord} by using the functions to convert GNS record
values to strings or vice-versa, or to lookup a GNS record type number
by name (or vice-versa). The library also provides various other
functions that are mostly used internally within GNS, such as converting
keys to names, checking for expiration, encrypting GNS records to GNS
blocks, verifying GNS block signatures and decrypting GNS records from
GNS blocks.

We will now discuss the four commonly used functions of the
API. @code{libgnunetgnsrecord} does not perform these operations itself,
but instead uses plugins to perform the operation. GNUnet includes
plugins to support common DNS record types as well as standard GNS
record types.

@node Value handling,Type handling,Future work,GNS<2>
@anchor{developers/apis/gns id5}@anchor{241}@anchor{developers/apis/gns value-handling}@anchor{242}
@subsubsection Value handling


@code{GNUNET_GNSRECORD_value_to_string} can be used to convert the (binary)
representation of a GNS record value to a human readable, 0-terminated
UTF-8 string. NULL is returned if the specified record type is not
supported by any available plugin.

@code{GNUNET_GNSRECORD_string_to_value} can be used to try to convert a
human readable string to the respective (binary) representation of a GNS
record value.

@node Type handling,Network Setup Details,Value handling,GNS<2>
@anchor{developers/apis/gns id6}@anchor{243}@anchor{developers/apis/gns type-handling}@anchor{244}
@subsubsection Type handling


@code{GNUNET_GNSRECORD_typename_to_number} can be used to obtain the
numeric value associated with a given typename. For example, given the
typename "A" (for DNS A reocrds), the function will return the number
1. A list of common DNS record types is
here@footnote{http://en.wikipedia.org/wiki/List_of_DNS_record_types}. Note
that not all DNS record types are supported by GNUnet GNSRECORD plugins
at this time.

@code{GNUNET_GNSRECORD_number_to_typename} can be used to obtain the
typename associated with a given numeric value. For example, given the
type number 1, the function will return the typename "A".

@menu
* GNS plugins:: 
* The GNS Client-Service Protocol:: 
* Hijacking the DNS-Traffic using gnunet-service-dns:: 

@end menu

@node GNS plugins,The GNS Client-Service Protocol,,Type handling
@anchor{developers/apis/gns gns-plugins}@anchor{245}@anchor{developers/apis/gns id7}@anchor{246}
@subsubsection GNS plugins


Adding a new GNS record type typically involves writing (or extending) a
GNSRECORD plugin. The plugin needs to implement the
@code{gnunet_gnsrecord_plugin.h} API which provides basic functions that
are needed by GNSRECORD to convert typenames and values of the
respective record type to strings (and back). These gnsrecord plugins
are typically implemented within their respective subsystems. Examples
for such plugins can be found in the GNSRECORD, GNS and CONVERSATION
subsystems.

The @code{libgnunetgnsrecord} library is then used to locate, load and
query the appropriate gnsrecord plugin. Which plugin is appropriate is
determined by the record type (which is just a 32-bit integer). The
@code{libgnunetgnsrecord} library loads all block plugins that are
installed at the local peer and forwards the application request to the
plugins. If the record type is not supported by the plugin, it should
simply return an error code.

The central functions of the block APIs (plugin and main library) are
the same four functions for converting between values and strings, and
typenames and numbers documented in the previous subsection.

@node The GNS Client-Service Protocol,Hijacking the DNS-Traffic using gnunet-service-dns,GNS plugins,Type handling
@anchor{developers/apis/gns the-gns-client-002dservice-protocol}@anchor{247}@anchor{developers/apis/gns the-gns-client-service-protocol}@anchor{248}
@subsubsection The GNS Client-Service Protocol


The GNS client-service protocol consists of two simple messages, the
@code{LOOKUP} message and the @code{LOOKUP_RESULT}. Each @code{LOOKUP} message
contains a unique 32-bit identifier, which will be included in the
corresponding response. Thus, clients can send many lookup requests in
parallel and receive responses out-of-order. A @code{LOOKUP} request also
includes the public key of the GNS zone, the desired record type and
fields specifying whether shortening is enabled or networking is
disabled. Finally, the @code{LOOKUP} message includes the name to be
resolved.

The response includes the number of records and the records themselves
in the format created by @code{GNUNET_GNSRECORD_records_serialize}. They
can thus be deserialized using @code{GNUNET_GNSRECORD_records_deserialize}.

@node Hijacking the DNS-Traffic using gnunet-service-dns,,The GNS Client-Service Protocol,Type handling
@anchor{developers/apis/gns hijacking-the-dns-002dtraffic-using-gnunet-002dservice-002ddns}@anchor{249}@anchor{developers/apis/gns hijacking-the-dns-traffic-using-gnunet-service-dns}@anchor{24a}
@subsubsection Hijacking the DNS-Traffic using gnunet-service-dns


This section documents how the gnunet-service-dns (and the
gnunet-helper-dns) intercepts DNS queries from the local system. This is
merely one method for how we can obtain GNS queries. It is also possible
to change @code{resolv.conf} to point to a machine running
@code{gnunet-dns2gns} or to modify libc's name system switch (NSS)
configuration to include a GNS resolution plugin. The method described
in this chapter is more of a last-ditch catch-all approach.

@code{gnunet-service-dns} enables intercepting DNS traffic using policy
based routing. We MARK every outgoing DNS-packet if it was not sent by
our application. Using a second routing table in the Linux kernel these
marked packets are then routed through our virtual network interface and
can thus be captured unchanged.

Our application then reads the query and decides how to handle it. If
the query can be addressed via GNS, it is passed to
@code{gnunet-service-gns} and resolved internally using GNS. In the future,
a reverse query for an address of the configured virtual network could
be answered with records kept about previous forward queries. Queries
that are not hijacked by some application using the DNS service will be
sent to the original recipient. The answer to the query will always be
sent back through the virtual interface with the original nameserver as
source address.

@node Network Setup Details,Conversions between DNS and GNS,Type handling,GNS<2>
@anchor{developers/apis/gns id8}@anchor{24b}@anchor{developers/apis/gns network-setup-details}@anchor{24c}
@subsubsection Network Setup Details


The DNS interceptor adds the following rules to the Linux kernel:

@example
iptables -t mangle -I OUTPUT 1 -p udp --sport $LOCALPORT --dport 53 \
-j ACCEPT iptables -t mangle -I OUTPUT 2 -p udp --dport 53 -j MARK \
--set-mark 3 ip rule add fwmark 3 table2 ip route add default via \
$VIRTUALDNS table2
@end example

@cartouche
@quotation Todo 
FIXME: Rewrite to reflect display which is no longer content
by line due to the < 74 characters limit.
@end quotation
@end cartouche

Line 1 makes sure that all packets coming from a port our application
opened beforehand (@code{$LOCALPORT}) will be routed normally. Line 2 marks
every other packet to a DNS-Server with mark 3 (chosen arbitrarily). The
third line adds a routing policy based on this mark 3 via the routing
table.

@menu
* Importing DNS Zones into GNS:: 

@end menu

@node Importing DNS Zones into GNS,,,Network Setup Details
@anchor{developers/apis/gns id10}@anchor{24d}@anchor{developers/apis/gns importing-dns-zones-into-gns}@anchor{24e}
@subsubsection Importing DNS Zones into GNS


This section discusses the challenges and problems faced when writing
the Ascension tool. It also takes a look at possible improvements in the
future.

Consider the following diagram that shows the workflow of Ascension:


@image{gnunet-figures/ascension_ssd,,,ascension,png}


Further the interaction between components of GNUnet are shown in the
diagram below:

DNS Conversion
.. _Conversions-between-DNS-and-GNS:

@node Conversions between DNS and GNS,DNS Zone Size,Network Setup Details,GNS<2>
@anchor{developers/apis/gns conversions-between-dns-and-gns}@anchor{24f}
@subsubsection Conversions between DNS and GNS


The differences between the two name systems lies in the details and is
not always transparent. For instance an SRV record is converted to a BOX
record which is unique to GNS.

This is done by converting to a BOX record from an existing SRV record:

@example
# SRV
# _service._proto.name. TTL class SRV priority weight port target
_sip._tcp.example.com. 14000 IN SRV     0 0 5060 www.example.com.
# BOX
# TTL BOX flags port protocol recordtype priority weight port target
14000 BOX n 5060 6 33 0 0 5060 www.example.com
@end example

Other records that need to undergo such transformation is the MX record
type, as well as the SOA record type.

Transformation of a SOA record into GNS works as described in the
following example. Very important to note are the rname and mname keys.

@example
# BIND syntax for a clean SOA record
   IN SOA master.example.com. hostmaster.example.com. (
    2017030300 ; serial
    3600       ; refresh
    1800       ; retry
    604800     ; expire
    600 )      ; ttl
# Recordline for adding the record
$ gnunet-namestore -z example.com -a -n  -t SOA -V \
    rname=master.example.com mname=hostmaster.example.com  \
    2017030300,3600,1800,604800,600 -e 7200s
@end example

The transformation of MX records is done in a simple way.

@example
# mail.example.com. 3600 IN MX 10 mail.example.com.
$ gnunet-namestore -z example.com -n mail -R 3600 MX n 10,mail
@end example

Finally, one of the biggest struggling points were the NS records that
are found in top level domain zones. The intended behaviour for those is
to add GNS2DNS records for those so that gnunet-gns can resolve records
for those domains on its own. Those require the values from DNS GLUE
records, provided they are within the same zone.

The following two examples show one record with a GLUE record and the
other one does not have a GLUE record. This takes place in the 'com'
TLD.

@example
# ns1.example.com 86400 IN A 127.0.0.1
# example.com 86400 IN NS ns1.example.com.
$ gnunet-namestore -z com -n example -R 86400 GNS2DNS n \
    example.com@@127.0.0.1

# example.com 86400 IN NS ns1.example.org.
$ gnunet-namestore -z com -n example -R 86400 GNS2DNS n \
    example.com@@ns1.example.org
@end example

As you can see, one of the GNS2DNS records has an IP address listed and
the other one a DNS name. For the first one there is a GLUE record to do
the translation directly and the second one will issue another DNS query
to figure out the IP of ns1.example.org.

A solution was found by creating a hierarchical zone structure in GNS
and linking the zones using PKEY records to one another. This allows the
resolution of the name servers to work within GNS while not taking
control over unwanted zones.

Currently the following record types are supported:


@itemize -

@item 
A

@item 
AAAA

@item 
CNAME

@item 
MX

@item 
NS

@item 
SRV

@item 
TXT
@end itemize

This is not due to technical limitations but rather a practical ones.
The problem occurs with DNSSEC enabled DNS zones. As records within
those zones are signed periodically, and every new signature is an
update to the zone, there are many revisions of zones. This results in a
problem with bigger zones as there are lots of records that have been
signed again but no major changes. Also trying to add records that are
unknown that require a different format take time as they cause a CLI
call of the namestore. Furthermore certain record types need
transformation into a GNS compatible format which, depending on the
record type, takes more time.

Further a blacklist was added to drop for instance DNSSEC related
records. Also if a record type is neither in the white list nor the
blacklist it is considered as a loss of data and a message is shown to
the user. This helps with transparency and also with contributing, as
the not supported record types can then be added accordingly.

@node DNS Zone Size,Performance,Conversions between DNS and GNS,GNS<2>
@anchor{developers/apis/gns dns-zone-size}@anchor{250}@anchor{developers/apis/gns id11}@anchor{251}
@subsubsection DNS Zone Size


Another very big problem exists with very large zones. When migrating a
small zone the delay between adding of records and their expiry is
negligible. However when working with big zones that easily have more
than a few million records this delay becomes a problem.

Records will start to expire well before the zone has finished
migrating. This is usually not a problem but can cause a high CPU load
when a peer is restarted and the records have expired.

A good solution has not been found yet. One of the idea that floated
around was that the records should be added with the s (shadow) flag to
keep the records resolvable even if they expired. However this would
introduce the problem of how to detect if a record has been removed from
the zone and would require deletion of said record(s).

Another problem that still persists is how to refresh records. Expired
records are still displayed when calling gnunet-namestore but do not
resolve with gnunet-gns. Zonemaster will sign the expired records again
and make sure that the records are still valid. With a recent change
this was fixed as gnunet-gns to improve the suffix lookup which allows
for a fast lookup even with thousands of local egos.

Currently the pace of adding records in general is around 10 records per
second. Crypto is the upper limit for adding of records. The performance
of your machine can be tested with the perf_crypto_* tools. There is
still a big discrepancy between the pace of Ascension and the
theoretical limit.

A performance metric for measuring improvements has not yet been
implemented in Ascension.

@node Performance,,DNS Zone Size,GNS<2>
@anchor{developers/apis/gns id12}@anchor{252}@anchor{developers/apis/gns performance}@anchor{253}
@subsubsection Performance


The performance when migrating a zone using the Ascension tool is
limited by a handful of factors. First of all ascension is written in
Python3 and calls the CLI tools of GNUnet. This is comparable to a fork
and exec call which costs a few CPU cycles. Furthermore all the records
that are added to the same label are signed using the zones private key.
This signing operation is very resource heavy and was optimized during
development by adding the '-R' (Recordline) option to gnunet-namestore
which allows to specify multiple records using the CLI tool. Assuming
that in a TLD zone every domain has at least two name servers this
halves the amount of signatures needed.

Another improvement that could be made is with the addition of multiple
threads or using asynchronous subprocesses when opening the GNUnet CLI
tools. This could be implemented by simply creating more workers in the
program but performance improvements were not tested.

Ascension was tested using different hardware and database backends.
Performance differences between SQLite and postgresql are marginal and
almost non existent. What did make a huge impact on record adding
performance was the storage medium. On a traditional mechanical hard
drive adding of records were slow compared to a solid state disk.

In conclusion there are many bottlenecks still around in the program,
namely the single threaded implementation and inefficient, sequential
calls of gnunet-namestore. In the future a solution that uses the C API
would be cleaner and better.

@geindex IDENTITY; subsystem

@node IDENTITY,MESSENGER,GNS<2>,Service APIs
@anchor{developers/apis/identity doc}@anchor{254}@anchor{developers/apis/identity identity}@anchor{255}@anchor{developers/apis/identity identity-subsystem-dev}@anchor{256}
@subsection IDENTITY


@menu
* Connecting to the service:: 
* Operations on Egos:: 
* The anonymous Ego:: 
* Convenience API to lookup a single ego:: 

@end menu

@node Connecting to the service,Operations on Egos,,IDENTITY
@anchor{developers/apis/identity connecting-to-the-identity-service}@anchor{257}@anchor{developers/apis/identity connecting-to-the-service}@anchor{258}
@subsubsection Connecting to the service


First, typical clients connect to the identity service using
@code{GNUNET_IDENTITY_connect}. This function takes a callback as a
parameter. If the given callback parameter is non-null, it will be
invoked to notify the application about the current state of the
identities in the system.


@itemize -

@item 
First, it will be invoked on all known egos at the time of the
connection. For each ego, a handle to the ego and the user's name for
the ego will be passed to the callback. Furthermore, a @code{void **}
context argument will be provided which gives the client the
opportunity to associate some state with the ego.

@item 
Second, the callback will be invoked with NULL for the ego, the name
and the context. This signals that the (initial) iteration over all
egos has completed.

@item 
Then, the callback will be invoked whenever something changes about
an ego. If an ego is renamed, the callback is invoked with the ego
handle of the ego that was renamed, and the new name. If an ego is
deleted, the callback is invoked with the ego handle and a name of
NULL. In the deletion case, the application should also release
resources stored in the context.

@item 
When the application destroys the connection to the identity service
using @code{GNUNET_IDENTITY_disconnect}, the callback is again invoked
with the ego and a name of NULL (equivalent to deletion of the egos).
This should again be used to clean up the per-ego context.
@end itemize

The ego handle passed to the callback remains valid until the callback
is invoked with a name of NULL, so it is safe to store a reference to
the ego's handle.

@node Operations on Egos,The anonymous Ego,Connecting to the service,IDENTITY
@anchor{developers/apis/identity id1}@anchor{259}@anchor{developers/apis/identity operations-on-egos}@anchor{25a}
@subsubsection Operations on Egos


Given an ego handle, the main operations are to get its associated
private key using @code{GNUNET_IDENTITY_ego_get_private_key} or its
associated public key using @code{GNUNET_IDENTITY_ego_get_public_key}.

The other operations on egos are pretty straightforward. Using
@code{GNUNET_IDENTITY_create}, an application can request the creation of
an ego by specifying the desired name. The operation will fail if that
name is already in use. Using @code{GNUNET_IDENTITY_rename} the name of an
existing ego can be changed. Finally, egos can be deleted using
@code{GNUNET_IDENTITY_delete}. All of these operations will trigger updates
to the callback given to the @code{GNUNET_IDENTITY_connect} function of all
applications that are connected with the identity service at the time.
@code{GNUNET_IDENTITY_cancel} can be used to cancel the operations before
the respective continuations would be called. It is not guaranteed that
the operation will not be completed anyway, only the continuation will
no longer be called.

@node The anonymous Ego,Convenience API to lookup a single ego,Operations on Egos,IDENTITY
@anchor{developers/apis/identity id2}@anchor{25b}@anchor{developers/apis/identity the-anonymous-ego}@anchor{25c}
@subsubsection The anonymous Ego


A special way to obtain an ego handle is to call
@code{GNUNET_IDENTITY_ego_get_anonymous}, which returns an ego for the
"anonymous" user --- anyone knows and can get the private key for this
user, so it is suitable for operations that are supposed to be anonymous
but require signatures (for example, to avoid a special path in the
code). The anonymous ego is always valid and accessing it does not
require a connection to the identity service.

@node Convenience API to lookup a single ego,,The anonymous Ego,IDENTITY
@anchor{developers/apis/identity convenience-api-to-lookup-a-single-ego}@anchor{25d}@anchor{developers/apis/identity id3}@anchor{25e}
@subsubsection Convenience API to lookup a single ego


As applications commonly simply have to lookup a single ego, there is a
convenience API to do just that. Use @code{GNUNET_IDENTITY_ego_lookup} to
lookup a single ego by name. Note that this is the user's name for the
ego, not the service function. The resulting ego will be returned via a
callback and will only be valid during that callback. The operation can
be canceled via @code{GNUNET_IDENTITY_ego_lookup_cancel} (cancellation is
only legal before the callback is invoked).

@menu
* The IDENTITY Client-Service Protocol:: 

@end menu

@node The IDENTITY Client-Service Protocol,,,Convenience API to lookup a single ego
@anchor{developers/apis/identity the-identity-client-002dservice-protocol}@anchor{25f}@anchor{developers/apis/identity the-identity-client-service-protocol}@anchor{260}
@subsubsection The IDENTITY Client-Service Protocol


A client connecting to the identity service first sends a message with
type @code{GNUNET_MESSAGE_TYPE_IDENTITY_START} to the service. After that,
the client will receive information about changes to the egos by
receiving messages of type @code{GNUNET_MESSAGE_TYPE_IDENTITY_UPDATE}.
Those messages contain the private key of the ego and the user's name of
the ego (or zero bytes for the name to indicate that the ego was
deleted). A special bit @code{end_of_list} is used to indicate the end of
the initial iteration over the identity service's egos.

The client can trigger changes to the egos by sending @code{CREATE},
@code{RENAME} or @code{DELETE} messages. The CREATE message contains the
private key and the desired name. The RENAME message contains the old
name and the new name. The DELETE message only needs to include the name
of the ego to delete. The service responds to each of these messages
with a @code{RESULT_CODE} message which indicates success or error of the
operation, and possibly a human-readable error message.

Finally, the client can bind the name of a service function to an ego by
sending a @code{SET_DEFAULT} message with the name of the service function
and the private key of the ego. Such bindings can then be resolved using
a @code{GET_DEFAULT} message, which includes the name of the service
function. The identity service will respond to a GET_DEFAULT request
with a SET_DEFAULT message containing the respective information, or
with a RESULT_CODE to indicate an error.

@geindex subsystem; MESSENGER

@node MESSENGER,NAMECACHE,IDENTITY,Service APIs
@anchor{developers/apis/messenger doc}@anchor{261}@anchor{developers/apis/messenger messenger}@anchor{262}@anchor{developers/apis/messenger messenger-subsystem-dev}@anchor{263}
@subsection MESSENGER


The MESSENGER API (defined in @code{gnunet_messenger_service.h}) allows P2P
applications built using GNUnet to communicate with specified kinds of
messages in a group. It provides applications the ability to send and
receive encrypted messages to any group of peers participating in GNUnet
in a decentralized way ( without even knowing all peers's identities).

MESSENGER delivers messages to other peers in "rooms". A room uses a
variable amount of CADET "channels" which will all be used for message
distribution. Each channel can represent an outgoing connection opened
by entering a room with @code{GNUNET_MESSENGER_enter_room} or an incoming
connection if the room was opened before via
@code{GNUNET_MESSENGER_open_room}.


@image{gnunet-figures/messenger_room,,,messenger_room,png}


To enter a room you have to specify the "door" (peer's identity of a
peer which has opened the room) and the key of the room (which is
identical to a CADET "port"). To open a room you have to specify only
the key to use. When opening a room you automatically distribute a
PEER-message sharing your peer's identity in the room.

Entering or opening a room can also be combined in any order. In any
case you will automatically get a unique member ID and send a
JOIN-message notifying others about your entry and your public key
derived from your selected private key.

The private key can be selected in combination with a name using
@code{GNUNET_MESSENGER_connect} besides setting a (message-)callback which
gets called every time a message gets sent or received in the room. Once
the handle is initialized you can check your used key pair with
@code{GNUNET_MESSENGER_get_key} providing only its public key. The function
returns NULL if the anonymous key pair is used. If the key pair should
be replaced with a different one, you can use
@code{GNUNET_MESSENGER_set_key} to ensure proper chaining of used private
keys.

This will automatically cause the handle to send a KEY-message which
introduces the change of key pair to all other members in the rooms you
have entered or opened. Your sessions will therefore stay valid while
your old key pair gets replaced, signing the exchange.

Also once the handle is initialized you can check your current name
with @code{GNUNET_MESSENGER_get_name} and potentially change or set a name
via @code{GNUNET_MESSENGER_set_name}. Any change in name will automatically
be distributed in all entered or opened rooms with a NAME-message.

In case you have adjusted your name separately in a specific room of
choice by sending a NAME-message manually, that room will not be
affected by the change of your handle's name.

To send a message a message inside of a room you can use
@code{GNUNET_MESSENGER_send_message}. If you specify a selected contact as
receiver, the message gets encrypted automatically and will be sent as
PRIVATE- message instead.

To request a potentially missed message or to get a specific message
after its original call of the message-callback, you can use
@code{GNUNET_MESSENGER_get_message}. Additionally once a message was
distributed to application level and the message-callback got called,
you can get the contact respresenting a message's sender respectively
with @code{GNUNET_MESSENGER_get_sender}. This allows getting name and the
public key of any sender currently in use with
@code{GNUNET_MESSENGER_contact_get_name} and
@code{GNUNET_MESSENGER_contact_get_key}. It is also possible to iterate
through all current members of a room with
@code{GNUNET_MESSENGER_iterate_members} using a callback.

To leave a room you can use @code{GNUNET_MESSENGER_close_room} which will
also close the rooms connections once all applications on the same peer
have left the room. Leaving a room will also send a LEAVE-message
closing a member session on all connected peers before any connection
will be closed. Leaving a room is however not required for any
application to keep your member session open between multiple sessions
of the actual application.

Finally, when an application no longer wants to use CADET, it should
call @code{GNUNET_MESSENGER_disconnect}. You don't have to explicitly close
the used rooms or leave them.

@menu
* Messages:: 
* Member sessions:: 

@end menu

@node Messages,Member sessions,,MESSENGER
@anchor{developers/apis/messenger messages}@anchor{264}
@subsubsection Messages


Here is a little summary to the kinds of messages you can send manually:

@menu
* NAME-message:: 
* INVITE-message:: 
* TEXT-message:: 
* FILE-message:: 
* DELETE-message:: 
* TICKET-message:: 
* TAG-message:: 
* SUBSCRIBE-message:: 
* TALK-message:: 

@end menu

@node NAME-message,INVITE-message,,Messages
@anchor{developers/apis/messenger name-002dmessage}@anchor{265}@anchor{developers/apis/messenger name-message}@anchor{266}
@subsubsection NAME-message


NAME-messages can be used to change the name (or nick) of your identity
inside a room. The selected name can differ from the identifier used to
select your private key for signing and decrypting messages.

@node INVITE-message,TEXT-message,NAME-message,Messages
@anchor{developers/apis/messenger invite-002dmessage}@anchor{267}@anchor{developers/apis/messenger invite-message}@anchor{268}
@subsubsection INVITE-message


INVITE-messages can be used to invite other members in a room to a
different room, sharing one potential door and the required key to enter
the room. This kind of message is typically sent as encrypted
PRIVATE-message to selected members because it doesn't make much sense
to invite all members from one room to another considering a rooms key
doesn't specify its usage.

@node TEXT-message,FILE-message,INVITE-message,Messages
@anchor{developers/apis/messenger text-002dmessage}@anchor{269}@anchor{developers/apis/messenger text-message}@anchor{26a}
@subsubsection TEXT-message


TEXT-messages can be used to send simple text-based messages and should
be considered as being in readable form without complex decoding. The
text has to end with a NULL-terminator character and should be in UTF-8
encoding for most compatibility.

@node FILE-message,DELETE-message,TEXT-message,Messages
@anchor{developers/apis/messenger file-002dmessage}@anchor{26b}@anchor{developers/apis/messenger file-message}@anchor{26c}
@subsubsection FILE-message


FILE-messages can be used to share files inside of a room. They do not
contain the actual file being shared but its original hash, filename,
URI to download the file and a symmetric key to decrypt the downloaded
file.

It is recommended to use the FS subsystem and the FILE-messages in
combination.

@node DELETE-message,TICKET-message,FILE-message,Messages
@anchor{developers/apis/messenger delete-002dmessage}@anchor{26d}@anchor{developers/apis/messenger delete-message}@anchor{26e}
@subsubsection DELETE-message


DELETE-messages can be send via the separate function
@code{GNUNET_MESSENGER_delete_message} which will handle linked deletions
of messages automatically. Messages can be linked in cases the content
of one message requires another message to co-exist.

DELETE-messages can be used to delete messages selected with its hash.
You can also select any custom delay relative to the time of sending the
DELETE-message. Deletion will only be processed on each peer in a room
if the sender is authorized.

The only information of a deleted message which being kept will be the
chained hashes connecting the message graph for potential traversion.
For example the check for completion of a member session requires this
information.

@node TICKET-message,TAG-message,DELETE-message,Messages
@anchor{developers/apis/messenger ticket-002dmessage}@anchor{26f}@anchor{developers/apis/messenger ticket-message}@anchor{270}
@subsubsection TICKET-message


TICKET-messages can be send privately to other members in the room. The
member will be able to consume the received ticket via
@code{GNUNET_RECLAIM_ticket_consume} to gain access to selected attributes
and their stored values.

TICKET-messages require the usage of the RECLAIM service of GNUnet to
issue, revoke and consume tickets. Revoking tickets is independant of
deletions inside the MESSENGER API.

@node TAG-message,SUBSCRIBE-message,TICKET-message,Messages
@anchor{developers/apis/messenger tag-002dmessage}@anchor{271}@anchor{developers/apis/messenger tag-message}@anchor{272}
@subsubsection TAG-message


TAG-messages can be used to tag or reject other messages in a
communicative way. Depending on the level of publication the message
fulfills different terms of functionality. For example if sent in a
private way using your own public key as audience, it might be used to
tag messages in a sharable way between different devices.

Public tagging or private tagging shared with only one contact might
reflect the communication of an own assignment or decision. For example
it can be used to reject an invitation from another contact.

Rejection can be expressed via a TAG-message that does not contain a
tag or in other words the tag is an empty string. This is a special case
for applications because it might translate into the rejection of
contacts or blocking of them.

Tagging or blocking of contacts depends on the message which is the
target of a TAG-message. If the tagged message is the latest
JOIN-message of another contact, the tag will be interpreted as tagging
of the regarding contact. The same way a rejection of such a
JOIN-message will be interpreted as block of the contact.

Unblocking or untagging can be done via deletion of the selected
TAG-message.

@node SUBSCRIBE-message,TALK-message,TAG-message,Messages
@anchor{developers/apis/messenger subscribe-002dmessage}@anchor{273}@anchor{developers/apis/messenger subscribe-message}@anchor{274}
@subsubsection SUBSCRIBE-message


SUBSCRIBE-messages can be used to subscribe to a selected discourse in
a room. With the @code{GNUNET_MESSENGER_FLAG_SUBSCRIPTION_KEEP_ALIVE} flag
this subscription will be extended automatically by the client API of
the MESSENGER service.

A subscription can be ended manually via SUBSCRIBE-message with the flag
@code{GNUNET_MESSENGER_FLAG_SUBSCRIPTION_UNSUBSCRIBE} or it will end
automatically when closing the room or disconnecting the client, letting
the subscription run out.

Such an active subscription is necessary to receive TALK-messages from
a discourse. SUBSCRIBE-messages will not be stored locally.

@node TALK-message,,SUBSCRIBE-message,Messages
@anchor{developers/apis/messenger talk-002dmessage}@anchor{275}@anchor{developers/apis/messenger talk-message}@anchor{276}
@subsubsection TALK-message


TALK-messages can be send while subscribed to a selected discourse. The
messages can contain any sort of binary data. Applications need to work
out format compatibility on their own. It is possible to link formats
to the discourse id used to identify a given discourse inside a room.

Only members that have an active subscription to the given discourse
will receive the TALK-messages. These messages will not be stored
locally.

@node Member sessions,,Messages,MESSENGER
@anchor{developers/apis/messenger id1}@anchor{277}@anchor{developers/apis/messenger member-sessions}@anchor{278}
@subsubsection Member sessions


A member session is a triple of the room key, the member ID and the
public key of the member's key pair. Member sessions allow that a member
can change their ID or their private key once at a time without losing
the ability to delete old messages or identifying the original sender
of a message. On every change of ID or private key a session will be
marked as closed. So every session chain will only contain one open
session with the current ID and public key.

If a session is marked as closed the MESSENGER service will check from
the first message opening a session to its last one closing the session
for completion. If a the service can confirm that there is no message
still missing which was sent from the closed member session, it will be
marked as completed.

A completed member session is not able to verify any incoming message to
ensure forward secrecy preventing others from using old stolen private
keys.

@geindex GNS; name cache
@geindex subsystem; NAMECACHE

@node NAMECACHE,NAMESTORE,MESSENGER,Service APIs
@anchor{developers/apis/namecache doc}@anchor{279}@anchor{developers/apis/namecache gns-namecache-dev}@anchor{27a}@anchor{developers/apis/namecache namecache}@anchor{27b}
@subsection NAMECACHE


The NAMECACHE API consists of five simple functions. First, there is
@code{GNUNET_NAMECACHE_connect} to connect to the NAMECACHE service. This
returns the handle required for all other operations on the NAMECACHE.
Using @code{GNUNET_NAMECACHE_block_cache} clients can insert a block into
the cache. @code{GNUNET_NAMECACHE_lookup_block} can be used to lookup
blocks that were stored in the NAMECACHE. Both operations can be
canceled using @code{GNUNET_NAMECACHE_cancel}. Note that canceling a
@code{GNUNET_NAMECACHE_block_cache} operation can result in the block being
stored in the NAMECACHE --- or not. Cancellation primarily ensures that
the continuation function with the result of the operation will no
longer be invoked. Finally, @code{GNUNET_NAMECACHE_disconnect} closes the
connection to the NAMECACHE.

The maximum size of a block that can be stored in the NAMECACHE is
@code{GNUNET_NAMECACHE_MAX_VALUE_SIZE}, which is defined to be 63 kB.

@menu
* The NAMECACHE Client-Service Protocol:: 
* The NAMECACHE Plugin API:: 

@end menu

@node The NAMECACHE Client-Service Protocol,The NAMECACHE Plugin API,,NAMECACHE
@anchor{developers/apis/namecache the-namecache-client-002dservice-protocol}@anchor{27c}@anchor{developers/apis/namecache the-namecache-client-service-protocol}@anchor{27d}
@subsubsection The NAMECACHE Client-Service Protocol


All messages in the NAMECACHE IPC protocol start with the
@code{struct GNUNET_NAMECACHE_Header} which adds a request ID (32-bit
integer) to the standard message header. The request ID is used to match
requests with the respective responses from the NAMECACHE, as they are
allowed to happen out-of-order.

@menu
* Lookup:: 
* Store:: 

@end menu

@node Lookup,Store,,The NAMECACHE Client-Service Protocol
@anchor{developers/apis/namecache id1}@anchor{27e}@anchor{developers/apis/namecache lookup}@anchor{27f}
@subsubsection Lookup


The @code{struct LookupBlockMessage} is used to lookup a block stored in
the cache. It contains the query hash. The NAMECACHE always responds
with a @code{struct LookupBlockResponseMessage}. If the NAMECACHE has no
response, it sets the expiration time in the response to zero.
Otherwise, the response is expected to contain the expiration time, the
ECDSA signature, the derived key and the (variable-size) encrypted data
of the block.

@node Store,,Lookup,The NAMECACHE Client-Service Protocol
@anchor{developers/apis/namecache id2}@anchor{280}@anchor{developers/apis/namecache store}@anchor{281}
@subsubsection Store


The @code{struct BlockCacheMessage} is used to cache a block in the
NAMECACHE. It has the same structure as the
@code{struct LookupBlockResponseMessage}. The service responds with a
@code{struct BlockCacheResponseMessage} which contains the result of the
operation (success or failure). In the future, we might want to make it
possible to provide an error message as well.

@node The NAMECACHE Plugin API,,The NAMECACHE Client-Service Protocol,NAMECACHE
@anchor{developers/apis/namecache id3}@anchor{282}@anchor{developers/apis/namecache the-namecache-plugin-api}@anchor{283}
@subsubsection The NAMECACHE Plugin API


The NAMECACHE plugin API consists of two functions, @code{cache_block} to
store a block in the database, and @code{lookup_block} to lookup a block in
the database.

@menu
* Lookup2:: 
* Store2:: 

@end menu

@node Lookup2,Store2,,The NAMECACHE Plugin API
@anchor{developers/apis/namecache id4}@anchor{284}@anchor{developers/apis/namecache lookup2}@anchor{285}
@subsubsection Lookup2


The @code{lookup_block} function is expected to return at most one block to
the iterator, and return @code{GNUNET_NO} if there were no non-expired
results. If there are multiple non-expired results in the cache, the
lookup is supposed to return the result with the largest expiration
time.

@node Store2,,Lookup2,The NAMECACHE Plugin API
@anchor{developers/apis/namecache id5}@anchor{286}@anchor{developers/apis/namecache store2}@anchor{287}
@subsubsection Store2


The @code{cache_block} function is expected to try to store the block in
the database, and return @code{GNUNET_SYSERR} if this was not possible for
any reason. Furthermore, @code{cache_block} is expected to implicitly
perform cache maintenance and purge blocks from the cache that have
expired. Note that @code{cache_block} might encounter the case where the
database already has another block stored under the same key. In this
case, the plugin must ensure that the block with the larger expiration
time is preserved. Obviously, this can done either by simply adding new
blocks and selecting for the most recent expiration time during lookup,
or by checking which block is more recent during the store operation.

@geindex subsystem; NAMESTORE

@node NAMESTORE,NSE,NAMECACHE,Service APIs
@anchor{developers/apis/namestore doc}@anchor{288}@anchor{developers/apis/namestore namestore}@anchor{289}@anchor{developers/apis/namestore namestore-subsystem-dev}@anchor{28a}
@subsection NAMESTORE


To interact with NAMESTORE clients first connect to the NAMESTORE
service using the @code{GNUNET_NAMESTORE_connect} passing a configuration
handle. As a result they obtain a NAMESTORE handle, they can use for
operations, or NULL is returned if the connection failed.

To disconnect from NAMESTORE, clients use
@code{GNUNET_NAMESTORE_disconnect} and specify the handle to disconnect.

NAMESTORE internally uses the private key to refer to zones. These
private keys can be obtained from the IDENTITY subsystem. Here `egos'
`can be used to refer to zones or the default ego assigned to the GNS
subsystem can be used to obtained the master zone's private key.'

@menu
* Editing Zone Information:: 
* Transactional operations:: 
* Iterating Zone Information:: 
* Monitoring Zone Information:: 

@end menu

@node Editing Zone Information,Transactional operations,,NAMESTORE
@anchor{developers/apis/namestore editing-zone-information}@anchor{28b}@anchor{developers/apis/namestore id1}@anchor{28c}
@subsubsection Editing Zone Information


NAMESTORE provides functions to lookup records stored under a label in a
zone and to store records under a label in a zone.

To store (and delete) records, the client uses the
@code{GNUNET_NAMESTORE_records_store} function and has to provide namestore
handle to use, the private key of the zone, the label to store the
records under, the records and number of records plus an callback
function. After the operation is performed NAMESTORE will call the
provided callback function with the result GNUNET_SYSERR on failure
(including timeout/queue drop/failure to validate), GNUNET_NO if content
was already there or not found GNUNET_YES (or other positive value) on
success plus an additional error message.

In addition, @code{GNUNET_NAMESTORE_records_store2} can be used to store multiple
record sets using a single API call. This allows the caller to import
a large number of (label, records) tuples in a single database transaction.
This is useful for large zone imports.

Records are deleted by using the store command with 0 records to store.
It is important to note, that records are not merged when records exist
with the label. So a client has first to retrieve records, merge with
existing records and then store the result.

To perform a lookup operation, the client uses the
@code{GNUNET_NAMESTORE_records_lookup} function. Here it has to pass the
namestore handle, the private key of the zone and the label. It also has
to provide a callback function which will be called with the result of
the lookup operation: the zone for the records, the label, and the
records including the number of records included.

A special operation is used to set the preferred nickname for a zone.
This nickname is stored with the zone and is automatically merged with
all labels and records stored in a zone. Here the client uses the
@code{GNUNET_NAMESTORE_set_nick} function and passes the private key of the
zone, the nickname as string plus a the callback with the result of the
operation.

@node Transactional operations,Iterating Zone Information,Editing Zone Information,NAMESTORE
@anchor{developers/apis/namestore transactional-namestore-api}@anchor{28d}@anchor{developers/apis/namestore transactional-operations}@anchor{28e}
@subsubsection Transactional operations


All API calls by default are mapped to implicit single transactions in the
database backends.
This happends automatically, as most databases support implicit transactions
including the databases supported by NAMESTORE.

However, implicit transactions have two drawbacks:

@quotation


@enumerate 

@item 
When storing or deleting a lot of records individual transactions cause
a significant overhead in the database.

@item 
Storage and deletion of records my multiple clients concurrently can lead
to inconsistencies.
@end enumerate
@end quotation

This is why NAMESTORE supports explicit transactions in order to efficiently
handle large amounds of zone data as well as keep then NAMESTORE consistent
when the client thinks this is necessary.

When the client wants to start a transaction, @code{GNUNET_NAMESTORE_transaction_begin}
is called.
After this call, @code{GNUNET_NAMESTORE_records_lookup} or @code{GNUNET_NAMESTORE_records_store(2)}
can be called successively.
The operations will only be commited to the database (and monitors such as ZONEMASTER
notified of the changes) when @code{GNUNET_NAMESTORE_transaction_commit} is used
to finalize the transaction.
Alternatively, the transaction can be aborted using @code{GNUNET_NAMESTORE_transaction_rollback}.
Should the client disconnect after calling @code{GNUNET_NAMESTORE_transaction_begin}
any running transaction will automatically be rolled-back.

@node Iterating Zone Information,Monitoring Zone Information,Transactional operations,NAMESTORE
@anchor{developers/apis/namestore id2}@anchor{28f}@anchor{developers/apis/namestore iterating-zone-information}@anchor{290}
@subsubsection Iterating Zone Information


A client can iterate over all information in a zone or all zones managed
by NAMESTORE. Here a client uses one of the
@code{GNUNET_NAMESTORE_zone_iteration_start(2)} functions and passes the
namestore handle, the zone to iterate over and a callback function to
call with the result. To iterate over all the zones, it is possible to
pass NULL for the zone. A @code{GNUNET_NAMESTORE_ZoneIterator} handle is
returned to be used to continue iteration.

NAMESTORE calls the callback for every result and expects the client to
call @code{GNUNET_NAMESTORE_zone_iterator_next} to continue to iterate or
@code{GNUNET_NAMESTORE_zone_iterator_stop} to interrupt the iteration. When
NAMESTORE reached the last item it will call the callback with a NULL
value to indicate.

@node Monitoring Zone Information,,Iterating Zone Information,NAMESTORE
@anchor{developers/apis/namestore id3}@anchor{291}@anchor{developers/apis/namestore monitoring-zone-information}@anchor{292}
@subsubsection Monitoring Zone Information


Clients can also monitor zones to be notified about changes. Here the
clients uses one of the @code{GNUNET_NAMESTORE_zone_monitor_start(2)} functions and
passes the private key of the zone and and a callback function to call
with updates for a zone. The client can specify to obtain zone
information first by iterating over the zone and specify a
synchronization callback to be called when the client and the namestore
are synced.

On an update, NAMESTORE will call the callback with the private key of
the zone, the label and the records and their number.

To stop monitoring, the client calls
@code{GNUNET_NAMESTORE_zone_monitor_stop} and passes the handle obtained
from the function to start the monitoring.

@geindex subsystem; Network size estimation
@geindex NSE; Network size estimation

@node NSE,PEERSTORE,NAMESTORE,Service APIs
@anchor{developers/apis/nse doc}@anchor{293}@anchor{developers/apis/nse nse}@anchor{294}@anchor{developers/apis/nse nse-subsystem-dev}@anchor{295}
@subsection NSE


The NSE subsystem has the simplest API of all services, with only two
calls: @code{GNUNET_NSE_connect} and @code{GNUNET_NSE_disconnect}.

The connect call gets a callback function as a parameter and this
function is called each time the network agrees on an estimate. This
usually is once per round, with some exceptions: if the closest peer has
a late local clock and starts spreading its ID after everyone else
agreed on a value, the callback might be activated twice in a round, the
second value being always bigger than the first. The default round time
is set to 1 hour.

The disconnect call disconnects from the NSE subsystem and the callback
is no longer called with new estimates.

@menu
* Results:: 
* Examples:: 

@end menu

@node Results,Examples,,NSE
@anchor{developers/apis/nse id1}@anchor{296}@anchor{developers/apis/nse results}@anchor{297}
@subsubsection Results


The callback provides two values: the average and the standard deviation@footnote{http://en.wikipedia.org/wiki/Standard_deviation} of the
last 64 rounds. The values provided by the callback function are
logarithmic, this means that the real estimate numbers can be obtained
by calculating 2 to the power of the given value (2average). From a
statistics point of view this means that:


@itemize -

@item 
68% of the time the real size is included in the interval
[(2average-stddev), 2]

@item 
95% of the time the real size is included in the interval
[(2average-2*stddev, 2^average+2*stddev]

@item 
99.7% of the time the real size is included in the interval
[(2average-3*stddev, 2average+3*stddev]
@end itemize

The expected standard variation for 64 rounds in a network of stable
size is 0.2. Thus, we can say that normally:


@itemize -

@item 
68% of the time the real size is in the range [-13%, +15%]

@item 
95% of the time the real size is in the range [-24%, +32%]

@item 
99.7% of the time the real size is in the range [-34%, +52%]
@end itemize

As said in the introduction, we can be quite sure that usually the real
size is between one third and three times the estimate. This can of
course vary with network conditions. Thus, applications may want to also
consider the provided standard deviation value, not only the average (in
particular, if the standard variation is very high, the average maybe
meaningless: the network size is changing rapidly).

@node Examples,,Results,NSE
@anchor{developers/apis/nse examples}@anchor{298}@anchor{developers/apis/nse libgnunetnse-002d-examples}@anchor{299}
@subsubsection Examples


Let's close with a couple examples.


@table @asis

@item Average: 10, std dev: 1 Here the estimate would be

2^10 = 1024 peers. (The range in which we can be 95% sure is: [2^8,
2^12] = [256, 4096]. We can be very (>99.7%) sure that the network is
not a hundred peers and absolutely sure that it is not a million
peers, but somewhere around a thousand.)

@item Average 22, std dev: 0.2 Here the estimate would be

2^22 = 4 Million peers. (The range in which we can be 99.7% sure is:
[2^21.4, 2^22.6] = [2.8M, 6.3M]. We can be sure that the network size
is around four million, with absolutely way of it being 1 million.)
@end table

To put this in perspective, if someone remembers the LHC Higgs boson
results, were announced with "5 sigma" and "6 sigma" certainties. In
this case a 5 sigma minimum would be 2 million and a 6 sigma minimum,
1.8 million.

@menu
* The NSE Client-Service Protocol:: 
* The NSE Peer-to-Peer Protocol:: 

@end menu

@node The NSE Client-Service Protocol,The NSE Peer-to-Peer Protocol,,Examples
@anchor{developers/apis/nse the-nse-client-002dservice-protocol}@anchor{29a}@anchor{developers/apis/nse the-nse-client-service-protocol}@anchor{29b}
@subsubsection The NSE Client-Service Protocol


As with the API, the client-service protocol is very simple, only has 2
different messages, defined in @code{src/nse/nse.h}:


@itemize -

@item 
@code{GNUNET_MESSAGE_TYPE_NSE_START} This message has no parameters and
is sent from the client to the service upon connection.

@item 
@code{GNUNET_MESSAGE_TYPE_NSE_ESTIMATE} This message is sent from the
service to the client for every new estimate and upon connection.
Contains a timestamp for the estimate, the average and the standard
deviation for the respective round.
@end itemize

When the @code{GNUNET_NSE_disconnect} API call is executed, the client
simply disconnects from the service, with no message involved.

NSE Peer-to-Peer Protocol
.. _The-NSE-Peer_002dto_002dPeer-Protocol:

@node The NSE Peer-to-Peer Protocol,,The NSE Client-Service Protocol,Examples
@anchor{developers/apis/nse the-nse-peer-to-peer-protocol}@anchor{29c}
@subsubsection The NSE Peer-to-Peer Protocol


GNUNET_MESSAGE_TYPE_NSE_P2P_FLOOD
The NSE subsystem only has one message in the P2P protocol, the
@code{GNUNET_MESSAGE_TYPE_NSE_P2P_FLOOD} message.

This message key contents are the timestamp to identify the round
(differences in system clocks may cause some peers to send messages way
too early or way too late, so the timestamp allows other peers to
identify such messages easily), the proof of work@footnote{http://en.wikipedia.org/wiki/Proof-of-work_system} used to make
it difficult to mount a Sybil attack@footnote{http://en.wikipedia.org/wiki/Sybil_attack}, and the public
key, which is used to verify the signature on the message.

Every peer stores a message for the previous, current and next round.
The messages for the previous and current round are given to peers that
connect to us. The message for the next round is simply stored until our
system clock advances to the next round. The message for the current
round is what we are flooding the network with right now. At the
beginning of each round the peer does the following:


@itemize -

@item 
calculates its own distance to the target value

@item 
creates, signs and stores the message for the current round (unless
it has a better message in the "next round" slot which came early
in the previous round)

@item 
calculates, based on the stored round message (own or received) when
to start flooding it to its neighbors
@end itemize

Upon receiving a message the peer checks the validity of the message
(round, proof of work, signature). The next action depends on the
contents of the incoming message:


@itemize -

@item 
if the message is worse than the current stored message, the peer
sends the current message back immediately, to stop the other peer
from spreading suboptimal results

@item 
if the message is better than the current stored message, the peer
stores the new message and calculates the new target time to start
spreading it to its neighbors (excluding the one the message came
from)

@item 
if the message is for the previous round, it is compared to the
message stored in the "previous round slot", which may then be
updated

@item 
if the message is for the next round, it is compared to the message
stored in the "next round slot", which again may then be updated
@end itemize

Finally, when it comes to send the stored message for the current round
to the neighbors there is a random delay added for each neighbor, to
avoid traffic spikes and minimize cross-messages.

@geindex subsystem; PEERSTORE

@node PEERSTORE,REST,NSE,Service APIs
@anchor{developers/apis/peerstore doc}@anchor{29d}@anchor{developers/apis/peerstore peerstore}@anchor{29e}@anchor{developers/apis/peerstore peerstore-subsystem-dev}@anchor{29f}
@subsection PEERSTORE


@menu
* libgnunetpeerstore:: 

@end menu

@node libgnunetpeerstore,,,PEERSTORE
@anchor{developers/apis/peerstore libgnunetpeerstore}@anchor{2a0}
@subsubsection libgnunetpeerstore


libgnunetpeerstore is the library containing the PEERSTORE API.
Subsystems wishing to communicate with the PEERSTORE service use this
API to open a connection to PEERSTORE. This is done by calling
@code{GNUNET_PEERSTORE_connect} which returns a handle to the newly created
connection. This handle has to be used with any further calls to the
API.

To store a new record, the function @code{GNUNET_PEERSTORE_store} is to be
used which requires the record fields and a continuation function that
will be called by the API after the STORE request is sent to the
PEERSTORE service. Note that calling the continuation function does not
mean that the record is successfully stored, only that the STORE request
has been successfully sent to the PEERSTORE service.
@code{GNUNET_PEERSTORE_store_cancel} can be called to cancel the STORE
request only before the continuation function has been called.

To iterate over stored records, the function
@code{GNUNET_PEERSTORE_iterate} is to be used. `peerid' and `key' can be
set to NULL. An iterator callback function will be called with each
matching record found and a NULL record at the end to signal the end of
result set. @code{GNUNET_PEERSTORE_iterate_cancel} can be used to cancel
the ITERATE request before the iterator callback is called with a NULL
record.

To be notified with new values stored under a (subsystem, peerid, key)
combination, the function @code{GNUNET_PEERSTORE_watch} is to be used. This
will register the watcher with the PEERSTORE service, any new records
matching the given combination will trigger the callback function passed
to @code{GNUNET_PEERSTORE_watch}. This continues until
@code{GNUNET_PEERSTORE_watch_cancel} is called or the connection to the
service is destroyed.

After the connection is no longer needed, the function
@code{GNUNET_PEERSTORE_disconnect} can be called to disconnect from the
PEERSTORE service. Any pending ITERATE or WATCH requests will be
destroyed. If the @code{sync_first} flag is set to @code{GNUNET_YES}, the API
will delay the disconnection until all pending STORE requests are sent
to the PEERSTORE service, otherwise, the pending STORE requests will be
destroyed as well.

@geindex subsystem; REST

@node REST,REVOCATION,PEERSTORE,Service APIs
@anchor{developers/apis/rest doc}@anchor{2a1}@anchor{developers/apis/rest rest}@anchor{2a2}@anchor{developers/apis/rest rest-subsystem-dev}@anchor{2a3}
@subsection REST


@cartouche
@quotation Todo 
Define REST
@end quotation
@end cartouche

Using the REST subsystem, you can expose REST-based APIs or services.
The REST service is designed as a pluggable architecture. To create a
new REST endpoint, simply add a library in the form "plugin_rest_*". The
REST service will automatically load all REST plugins on startup.

@menu
* Namespace considerations:: 
* Endpoint documentation:: 

@end menu

@node Namespace considerations,Endpoint documentation,,REST
@anchor{developers/apis/rest id2}@anchor{2a4}@anchor{developers/apis/rest namespace-considerations}@anchor{2a5}
@subsubsection Namespace considerations


The @code{gnunet-rest-service} will load all plugins that are installed. As
such it is important that the endpoint namespaces do not clash.

For example, plugin X might expose the endpoint "/xxx" while plugin Y
exposes endpoint "/xxx/yyy". This is a problem if plugin X is also
supposed to handle a call to "/xxx/yyy". Currently the REST service will
not complain or warn about such clashes, so please make sure that
endpoints are unambiguous.

@node Endpoint documentation,,Namespace considerations,REST
@anchor{developers/apis/rest endpoint-documentation}@anchor{2a6}@anchor{developers/apis/rest id3}@anchor{2a7}
@subsubsection Endpoint documentation


This is WIP. Endpoints should be documented appropriately. Preferably
using annotations.

@geindex subsystem; REVOCATION

@node REVOCATION,Peer-to-Peer Set Operations,REST,Service APIs
@anchor{developers/apis/revocation doc}@anchor{2a8}@anchor{developers/apis/revocation revocation}@anchor{2a9}@anchor{developers/apis/revocation revocation-subsystem-dev}@anchor{2aa}
@subsection REVOCATION


The REVOCATION API consists of two parts, to query and to issue
revocations.

@menu
* Querying for revoked keys:: 
* Preparing revocations:: 
* Issuing revocations:: 

@end menu

@node Querying for revoked keys,Preparing revocations,,REVOCATION
@anchor{developers/apis/revocation id1}@anchor{2ab}@anchor{developers/apis/revocation querying-for-revoked-keys}@anchor{2ac}
@subsubsection Querying for revoked keys


@code{GNUNET_REVOCATION_query} is used to check if a given ECDSA public key
has been revoked. The given callback will be invoked with the result of
the check. The query can be canceled using
@code{GNUNET_REVOCATION_query_cancel} on the return value.

@node Preparing revocations,Issuing revocations,Querying for revoked keys,REVOCATION
@anchor{developers/apis/revocation id2}@anchor{2ad}@anchor{developers/apis/revocation preparing-revocations}@anchor{2ae}
@subsubsection Preparing revocations


It is often desirable to create a revocation record ahead-of-time and
store it in an off-line location to be used later in an emergency. This
is particularly true for GNUnet revocations, where performing the
revocation operation itself is computationally expensive and thus is
likely to take some time. Thus, if users want the ability to perform
revocations quickly in an emergency, they must pre-compute the
revocation message. The revocation API enables this with two functions
that are used to compute the revocation message, but not trigger the
actual revocation operation.

@code{GNUNET_REVOCATION_check_pow} should be used to calculate the
proof-of-work required in the revocation message. This function takes
the public key, the required number of bits for the proof of work (which
in GNUnet is a network-wide constant) and finally a proof-of-work number
as arguments. The function then checks if the given proof-of-work number
is a valid proof of work for the given public key. Clients preparing a
revocation are expected to call this function repeatedly (typically with
a monotonically increasing sequence of numbers of the proof-of-work
number) until a given number satisfies the check. That number should
then be saved for later use in the revocation operation.

@code{GNUNET_REVOCATION_sign_revocation} is used to generate the signature
that is required in a revocation message. It takes the private key that
(possibly in the future) is to be revoked and returns the signature. The
signature can again be saved to disk for later use, which will then
allow performing a revocation even without access to the private key.

@node Issuing revocations,,Preparing revocations,REVOCATION
@anchor{developers/apis/revocation id3}@anchor{2af}@anchor{developers/apis/revocation issuing-revocations}@anchor{2b0}
@subsubsection Issuing revocations


Given a ECDSA public key, the signature from @code{GNUNET_REVOCATION_sign}
and the proof-of-work, @code{GNUNET_REVOCATION_revoke} can be used to
perform the actual revocation. The given callback is called upon
completion of the operation. @code{GNUNET_REVOCATION_revoke_cancel} can be
used to stop the library from calling the continuation; however, in that
case it is undefined whether or not the revocation operation will be
executed.

@menu
* The REVOCATION Client-Service Protocol:: 
* The REVOCATION Peer-to-Peer Protocol:: 

@end menu

@node The REVOCATION Client-Service Protocol,The REVOCATION Peer-to-Peer Protocol,,Issuing revocations
@anchor{developers/apis/revocation the-revocation-client-002dservice-protocol}@anchor{2b1}@anchor{developers/apis/revocation the-revocation-client-service-protocol}@anchor{2b2}
@subsubsection The REVOCATION Client-Service Protocol


The REVOCATION protocol consists of four simple messages.

A @code{QueryMessage} containing a public ECDSA key is used to check if a
particular key has been revoked. The service responds with a
@code{QueryResponseMessage} which simply contains a bit that says if the
given public key is still valid, or if it has been revoked.

The second possible interaction is for a client to revoke a key by
passing a @code{RevokeMessage} to the service. The @code{RevokeMessage}
contains the ECDSA public key to be revoked, a signature by the
corresponding private key and the proof-of-work. The service responds
with a @code{RevocationResponseMessage} which can be used to indicate that
the @code{RevokeMessage} was invalid (e.g. the proof of work is incorrect),
or otherwise to indicate that the revocation has been processed
successfully.

@node The REVOCATION Peer-to-Peer Protocol,,The REVOCATION Client-Service Protocol,Issuing revocations
@anchor{developers/apis/revocation the-revocation-peer-002dto-002dpeer-protocol}@anchor{2b3}@anchor{developers/apis/revocation the-revocation-peer-to-peer-protocol}@anchor{2b4}
@subsubsection The REVOCATION Peer-to-Peer Protocol


Revocation uses two disjoint ways to spread revocation information among
peers. First of all, P2P gossip exchanged via CORE-level neighbours is
used to quickly spread revocations to all connected peers. Second,
whenever two peers (that both support revocations) connect, the SET
service is used to compute the union of the respective revocation sets.

In both cases, the exchanged messages are @code{RevokeMessage}s which
contain the public key that is being revoked, a matching ECDSA
signature, and a proof-of-work. Whenever a peer learns about a new
revocation this way, it first validates the signature and the
proof-of-work, then stores it to disk (typically to a file
$GNUNET_DATA_HOME/revocation.dat) and finally spreads the information to
all directly connected neighbours.

For computing the union using the SET service, the peer with the smaller
hashed peer identity will connect (as a "client" in the two-party set
protocol) to the other peer after one second (to reduce traffic spikes
on connect) and initiate the computation of the set union. All
revocation services use a common hash to identify the SET operation over
revocation sets.

The current implementation accepts revocation set union operations from
all peers at any time; however, well-behaved peers should only initiate
this operation once after establishing a connection to a peer with a
larger hashed peer identity.

@node Peer-to-Peer Set Operations,STATISTICS,REVOCATION,Service APIs
@anchor{developers/apis/setops doc}@anchor{2b5}@anchor{developers/apis/setops peer-to-peer-set-operations}@anchor{2b6}
@subsection Peer-to-Peer Set Operations


Many applications

@geindex subsystem; SET

@menu
* SET:: 
* SETI:: 
* SETU:: 

@end menu

@node SET,SETI,,Peer-to-Peer Set Operations
@anchor{developers/apis/set/set doc}@anchor{2b7}@anchor{developers/apis/set/set set}@anchor{2b8}@anchor{developers/apis/set/set set-subsystem-dev}@anchor{2b9}
@subsubsection SET


@menu
* Sets:: 
* Listeners:: 
* Operations:: 
* Supplying a Set:: 
* The Result Callback:: 
* Creating Sets:: 
* Listeners: Listeners<2>. 
* Initiating Operations:: 
* Modifying Sets:: 
* Results and Operation Status:: 
* Iterating Sets:: 
* The Bloom filter exchange:: 
* Salt:: 

@end menu

@node Sets,Listeners,,SET
@anchor{developers/apis/set/set id1}@anchor{2ba}@anchor{developers/apis/set/set sets}@anchor{2bb}
@subsubsection Sets


New sets are created with @code{GNUNET_SET_create}. Both the local peer's
configuration (as each set has its own client connection) and the
operation type must be specified. The set exists until either the client
calls @code{GNUNET_SET_destroy} or the client's connection to the service
is disrupted. In the latter case, the client is notified by the return
value of functions dealing with sets. This return value must always be
checked.

Elements are added and removed with @code{GNUNET_SET_add_element} and
@code{GNUNET_SET_remove_element}.

@node Listeners,Operations,Sets,SET
@anchor{developers/apis/set/set id2}@anchor{2bc}@anchor{developers/apis/set/set listeners}@anchor{2bd}
@subsubsection Listeners


Listeners are created with @code{GNUNET_SET_listen}. Each time time a
remote peer suggests a set operation with an application id and
operation type matching a listener, the listener's callback is invoked.
The client then must synchronously call either @code{GNUNET_SET_accept} or
@code{GNUNET_SET_reject}. Note that the operation will not be started until
the client calls @code{GNUNET_SET_commit} (see Section "Supplying a
Set").

@node Operations,Supplying a Set,Listeners,SET
@anchor{developers/apis/set/set id3}@anchor{2be}@anchor{developers/apis/set/set operations}@anchor{2bf}
@subsubsection Operations


Operations to be initiated by the local peer are created with
@code{GNUNET_SET_prepare}. Note that the operation will not be started
until the client calls @code{GNUNET_SET_commit} (see Section "Supplying a
Set").

@node Supplying a Set,The Result Callback,Operations,SET
@anchor{developers/apis/set/set id4}@anchor{2c0}@anchor{developers/apis/set/set supplying-a-set}@anchor{2c1}
@subsubsection Supplying a Set


To create symmetry between the two ways of starting a set operation
(accepting and initiating it), the operation handles returned by
@code{GNUNET_SET_accept} and @code{GNUNET_SET_prepare} do not yet have a set
to operate on, thus they can not do any work yet.

The client must call @code{GNUNET_SET_commit} to specify a set to use for
an operation. @code{GNUNET_SET_commit} may only be called once per set
operation.

@node The Result Callback,Creating Sets,Supplying a Set,SET
@anchor{developers/apis/set/set id5}@anchor{2c2}@anchor{developers/apis/set/set the-result-callback}@anchor{2c3}
@subsubsection The Result Callback


Clients must specify both a result mode and a result callback with
@code{GNUNET_SET_accept} and @code{GNUNET_SET_prepare}. The result callback
with a status indicating either that an element was received, or the
operation failed or succeeded. The interpretation of the received
element depends on the result mode. The callback needs to know which
result mode it is used in, as the arguments do not indicate if an
element is part of the full result set, or if it is in the difference
between the original set and the final set.

@menu
* The SET Client-Service Protocol:: 

@end menu

@node The SET Client-Service Protocol,,,The Result Callback
@anchor{developers/apis/set/set the-set-client-002dservice-protocol}@anchor{2c4}@anchor{developers/apis/set/set the-set-client-service-protocol}@anchor{2c5}
@subsubsection The SET Client-Service Protocol


@node Creating Sets,Listeners<2>,The Result Callback,SET
@anchor{developers/apis/set/set creating-sets}@anchor{2c6}@anchor{developers/apis/set/set id6}@anchor{2c7}
@subsubsection Creating Sets


For each set of a client, there exists a client connection to the
service. Sets are created by sending the @code{GNUNET_SERVICE_SET_CREATE}
message over a new client connection. Multiple operations for one set
are multiplexed over one client connection, using a request id supplied
by the client.

@node Listeners<2>,Initiating Operations,Creating Sets,SET
@anchor{developers/apis/set/set id7}@anchor{2c8}@anchor{developers/apis/set/set listeners2}@anchor{2c9}
@subsubsection Listeners


Each listener also requires a separate client connection. By sending the
@code{GNUNET_SERVICE_SET_LISTEN} message, the client notifies the service
of the application id and operation type it is interested in. A client
rejects an incoming request by sending @code{GNUNET_SERVICE_SET_REJECT} on
the listener's client connection. In contrast, when accepting an
incoming request, a @code{GNUNET_SERVICE_SET_ACCEPT} message must be sent
over the set that is supplied for the set operation.

@node Initiating Operations,Modifying Sets,Listeners<2>,SET
@anchor{developers/apis/set/set id8}@anchor{2ca}@anchor{developers/apis/set/set initiating-operations}@anchor{2cb}
@subsubsection Initiating Operations


Operations with remote peers are initiated by sending a
@code{GNUNET_SERVICE_SET_EVALUATE} message to the service. The client
connection that this message is sent by determines the set to use.

@node Modifying Sets,Results and Operation Status,Initiating Operations,SET
@anchor{developers/apis/set/set id9}@anchor{2cc}@anchor{developers/apis/set/set modifying-sets}@anchor{2cd}
@subsubsection Modifying Sets


Sets are modified with the @code{GNUNET_SERVICE_SET_ADD} and
@code{GNUNET_SERVICE_SET_REMOVE} messages.

@node Results and Operation Status,Iterating Sets,Modifying Sets,SET
@anchor{developers/apis/set/set id10}@anchor{2ce}@anchor{developers/apis/set/set results-and-operation-status}@anchor{2cf}
@subsubsection Results and Operation Status


The service notifies the client of result elements and success/failure
of a set operation with the @code{GNUNET_SERVICE_SET_RESULT} message.

@node Iterating Sets,The Bloom filter exchange,Results and Operation Status,SET
@anchor{developers/apis/set/set id11}@anchor{2d0}@anchor{developers/apis/set/set iterating-sets}@anchor{2d1}
@subsubsection Iterating Sets


All elements of a set can be requested by sending
@code{GNUNET_SERVICE_SET_ITER_REQUEST}. The server responds with
@code{GNUNET_SERVICE_SET_ITER_ELEMENT} and eventually terminates the
iteration with @code{GNUNET_SERVICE_SET_ITER_DONE}. After each received
element, the client must send @code{GNUNET_SERVICE_SET_ITER_ACK}. Note that
only one set iteration may be active for a set at any given time.

@menu
* The SET Intersection Peer-to-Peer Protocol:: 

@end menu

@node The SET Intersection Peer-to-Peer Protocol,,,Iterating Sets
@anchor{developers/apis/set/set the-set-intersection-peer-002dto-002dpeer-protocol}@anchor{2d2}@anchor{developers/apis/set/set the-set-intersection-peer-to-peer-protocol}@anchor{2d3}
@subsubsection The SET Intersection Peer-to-Peer Protocol


The intersection protocol operates over CADET and starts with a
GNUNET_MESSAGE_TYPE_SET_P2P_OPERATION_REQUEST being sent by the peer
initiating the operation to the peer listening for inbound requests. It
includes the number of elements of the initiating peer, which is used to
decide which side will send a Bloom filter first.

The listening peer checks if the operation type and application
identifier are acceptable for its current state. If not, it responds
with a GNUNET_MESSAGE_TYPE_SET_RESULT and a status of
GNUNET_SET_STATUS_FAILURE (and terminates the CADET channel).

If the application accepts the request, the listener sends back a
@code{GNUNET_MESSAGE_TYPE_SET_INTERSECTION_P2P_ELEMENT_INFO} if it has more
elements in the set than the client. Otherwise, it immediately starts
with the Bloom filter exchange. If the initiator receives a
@code{GNUNET_MESSAGE_TYPE_SET_INTERSECTION_P2P_ELEMENT_INFO} response, it
beings the Bloom filter exchange, unless the set size is indicated to be
zero, in which case the intersection is considered finished after just
the initial handshake.

@node The Bloom filter exchange,Salt,Iterating Sets,SET
@anchor{developers/apis/set/set id12}@anchor{2d4}@anchor{developers/apis/set/set the-bloom-filter-exchange}@anchor{2d5}
@subsubsection The Bloom filter exchange


In this phase, each peer transmits a Bloom filter over the remaining
keys of the local set to the other peer using a
@code{GNUNET_MESSAGE_TYPE_SET_INTERSECTION_P2P_BF} message. This message
additionally includes the number of elements left in the sender's set,
as well as the XOR over all of the keys in that set.

The number of bits 'k' set per element in the Bloom filter is calculated
based on the relative size of the two sets. Furthermore, the size of the
Bloom filter is calculated based on 'k' and the number of elements in
the set to maximize the amount of data filtered per byte transmitted on
the wire (while avoiding an excessively high number of iterations).

The receiver of the message removes all elements from its local set that
do not pass the Bloom filter test. It then checks if the set size of the
sender and the XOR over the keys match what is left of its own set. If
they do, it sends a @code{GNUNET_MESSAGE_TYPE_SET_INTERSECTION_P2P_DONE}
back to indicate that the latest set is the final result. Otherwise, the
receiver starts another Bloom filter exchange, except this time as the
sender.

@node Salt,,The Bloom filter exchange,SET
@anchor{developers/apis/set/set id13}@anchor{2d6}@anchor{developers/apis/set/set salt}@anchor{2d7}
@subsubsection Salt


Bloomfilter operations are probabilistic: With some non-zero probability
the test may incorrectly say an element is in the set, even though it is
not.

To mitigate this problem, the intersection protocol iterates exchanging
Bloom filters using a different random 32-bit salt in each iteration
(the salt is also included in the message). With different salts, set
operations may fail for different elements. Merging the results from the
executions, the probability of failure drops to zero.

The iterations terminate once both peers have established that they have
sets of the same size, and where the XOR over all keys computes the same
512-bit value (leaving a failure probability of 2@w{^-511}).

@menu
* The SET Union Peer-to-Peer Protocol:: 

@end menu

@node The SET Union Peer-to-Peer Protocol,,,Salt
@anchor{developers/apis/set/set the-set-union-peer-002dto-002dpeer-protocol}@anchor{2d8}@anchor{developers/apis/set/set the-set-union-peer-to-peer-protocol}@anchor{2d9}
@subsubsection The SET Union Peer-to-Peer Protocol


The SET union protocol is based on Eppstein's efficient set
reconciliation without prior context. You should read this paper first
if you want to understand the protocol.

@cartouche
@quotation Todo 
Link to Eppstein's paper!
@end quotation
@end cartouche

The union protocol operates over CADET and starts with a
GNUNET_MESSAGE_TYPE_SET_P2P_OPERATION_REQUEST being sent by the peer
initiating the operation to the peer listening for inbound requests. It
includes the number of elements of the initiating peer, which is
currently not used.

The listening peer checks if the operation type and application
identifier are acceptable for its current state. If not, it responds
with a @code{GNUNET_MESSAGE_TYPE_SET_RESULT} and a status of
@code{GNUNET_SET_STATUS_FAILURE} (and terminates the CADET channel).

If the application accepts the request, it sends back a strata estimator
using a message of type GNUNET_MESSAGE_TYPE_SET_UNION_P2P_SE. The
initiator evaluates the strata estimator and initiates the exchange of
invertible Bloom filters, sending a
GNUNET_MESSAGE_TYPE_SET_UNION_P2P_IBF.

During the IBF exchange, if the receiver cannot invert the Bloom filter
or detects a cycle, it sends a larger IBF in response (up to a defined
maximum limit; if that limit is reached, the operation fails). Elements
decoded while processing the IBF are transmitted to the other peer using
GNUNET_MESSAGE_TYPE_SET_P2P_ELEMENTS, or requested from the other peer
using GNUNET_MESSAGE_TYPE_SET_P2P_ELEMENT_REQUESTS messages, depending
on the sign observed during decoding of the IBF. Peers respond to a
GNUNET_MESSAGE_TYPE_SET_P2P_ELEMENT_REQUESTS message with the respective
element in a GNUNET_MESSAGE_TYPE_SET_P2P_ELEMENTS message. If the IBF
fully decodes, the peer responds with a
GNUNET_MESSAGE_TYPE_SET_UNION_P2P_DONE message instead of another
GNUNET_MESSAGE_TYPE_SET_UNION_P2P_IBF.

All Bloom filter operations use a salt to mingle keys before hashing
them into buckets, such that future iterations have a fresh chance of
succeeding if they failed due to collisions before.

@geindex subsystem; SETI

@node SETI,SETU,SET,Peer-to-Peer Set Operations
@anchor{developers/apis/seti/seti doc}@anchor{2da}@anchor{developers/apis/seti/seti seti}@anchor{2db}@anchor{developers/apis/seti/seti seti-subsystem-dev}@anchor{2dc}
@subsubsection SETI


@menu
* Intersection Set API:: 
* Intersection Listeners:: 
* Intersection Operations:: 
* Supplying a Set for Intersection:: 
* The Intersection Result Callback:: 
* Creating Intersection Sets:: 
* Listeners for Intersection:: 
* Initiating Intersection Operations:: 
* Modifying Intersection Sets:: 
* Intersection Results and Operation Status:: 
* The Bloom filter exchange in SETI:: 
* Intersection Salt:: 

@end menu

@node Intersection Set API,Intersection Listeners,,SETI
@anchor{developers/apis/seti/seti id1}@anchor{2dd}@anchor{developers/apis/seti/seti intersection-set-api}@anchor{2de}
@subsubsection Intersection Set API


New sets are created with @code{GNUNET_SETI_create}. Only the local peer's
configuration (as each set has its own client connection) must be
provided. The set exists until either the client calls
@code{GNUNET_SET_destroy} or the client's connection to the service is
disrupted. In the latter case, the client is notified by the return
value of functions dealing with sets. This return value must always be
checked.

Elements are added with @code{GNUNET_SET_add_element}.

@node Intersection Listeners,Intersection Operations,Intersection Set API,SETI
@anchor{developers/apis/seti/seti id2}@anchor{2df}@anchor{developers/apis/seti/seti intersection-listeners}@anchor{2e0}
@subsubsection Intersection Listeners


Listeners are created with @code{GNUNET_SET_listen}. Each time time a
remote peer suggests a set operation with an application id and
operation type matching a listener, the listener's callback is invoked.
The client then must synchronously call either @code{GNUNET_SET_accept} or
@code{GNUNET_SET_reject}. Note that the operation will not be started until
the client calls @code{GNUNET_SET_commit} (see Section "Supplying a
Set").

@node Intersection Operations,Supplying a Set for Intersection,Intersection Listeners,SETI
@anchor{developers/apis/seti/seti id3}@anchor{2e1}@anchor{developers/apis/seti/seti intersection-operations}@anchor{2e2}
@subsubsection Intersection Operations


Operations to be initiated by the local peer are created with
@code{GNUNET_SET_prepare}. Note that the operation will not be started
until the client calls @code{GNUNET_SET_commit} (see Section "Supplying a
Set").

@node Supplying a Set for Intersection,The Intersection Result Callback,Intersection Operations,SETI
@anchor{developers/apis/seti/seti id4}@anchor{2e3}@anchor{developers/apis/seti/seti supplying-a-set-for-intersection}@anchor{2e4}
@subsubsection Supplying a Set for Intersection


To create symmetry between the two ways of starting a set operation
(accepting and initiating it), the operation handles returned by
@code{GNUNET_SET_accept} and @code{GNUNET_SET_prepare} do not yet have a set
to operate on, thus they can not do any work yet.

The client must call @code{GNUNET_SET_commit} to specify a set to use for
an operation. @code{GNUNET_SET_commit} may only be called once per set
operation.

@node The Intersection Result Callback,Creating Intersection Sets,Supplying a Set for Intersection,SETI
@anchor{developers/apis/seti/seti id5}@anchor{2e5}@anchor{developers/apis/seti/seti the-intersection-result-callback}@anchor{2e6}
@subsubsection The Intersection Result Callback


Clients must specify both a result mode and a result callback with
@code{GNUNET_SET_accept} and @code{GNUNET_SET_prepare}. The result callback
with a status indicating either that an element was received, or the
operation failed or succeeded. The interpretation of the received
element depends on the result mode. The callback needs to know which
result mode it is used in, as the arguments do not indicate if an
element is part of the full result set, or if it is in the difference
between the original set and the final set.

@menu
* The SETI Client-Service Protocol:: 

@end menu

@node The SETI Client-Service Protocol,,,The Intersection Result Callback
@anchor{developers/apis/seti/seti the-seti-client-002dservice-protocol}@anchor{2e7}@anchor{developers/apis/seti/seti the-seti-client-service-protocol}@anchor{2e8}
@subsubsection The SETI Client-Service Protocol


@node Creating Intersection Sets,Listeners for Intersection,The Intersection Result Callback,SETI
@anchor{developers/apis/seti/seti creating-intersection-sets}@anchor{2e9}@anchor{developers/apis/seti/seti id6}@anchor{2ea}
@subsubsection Creating Intersection Sets


For each set of a client, there exists a client connection to the
service. Sets are created by sending the @code{GNUNET_SERVICE_SETI_CREATE}
message over a new client connection. Multiple operations for one set
are multiplexed over one client connection, using a request id supplied
by the client.

@node Listeners for Intersection,Initiating Intersection Operations,Creating Intersection Sets,SETI
@anchor{developers/apis/seti/seti id7}@anchor{2eb}@anchor{developers/apis/seti/seti listeners-for-intersection}@anchor{2ec}
@subsubsection Listeners for Intersection


Each listener also requires a separate client connection. By sending the
@code{GNUNET_SERVICE_SETI_LISTEN} message, the client notifies the service
of the application id and operation type it is interested in. A client
rejects an incoming request by sending @code{GNUNET_SERVICE_SETI_REJECT} on
the listener's client connection. In contrast, when accepting an
incoming request, a @code{GNUNET_SERVICE_SETI_ACCEPT} message must be sent
over the set that is supplied for the set operation.

@node Initiating Intersection Operations,Modifying Intersection Sets,Listeners for Intersection,SETI
@anchor{developers/apis/seti/seti id8}@anchor{2ed}@anchor{developers/apis/seti/seti initiating-intersection-operations}@anchor{2ee}
@subsubsection Initiating Intersection Operations


Operations with remote peers are initiated by sending a
@code{GNUNET_SERVICE_SETI_EVALUATE} message to the service. The client
connection that this message is sent by determines the set to use.

@node Modifying Intersection Sets,Intersection Results and Operation Status,Initiating Intersection Operations,SETI
@anchor{developers/apis/seti/seti id9}@anchor{2ef}@anchor{developers/apis/seti/seti modifying-intersection-sets}@anchor{2f0}
@subsubsection Modifying Intersection Sets


Sets are modified with the @code{GNUNET_SERVICE_SETI_ADD} message.

@node Intersection Results and Operation Status,The Bloom filter exchange in SETI,Modifying Intersection Sets,SETI
@anchor{developers/apis/seti/seti id10}@anchor{2f1}@anchor{developers/apis/seti/seti intersection-results-and-operation-status}@anchor{2f2}
@subsubsection Intersection Results and Operation Status


The service notifies the client of result elements and success/failure
of a set operation with the @code{GNUNET_SERVICE_SETI_RESULT} message.

@menu
* The SETI Intersection Peer-to-Peer Protocol:: 

@end menu

@node The SETI Intersection Peer-to-Peer Protocol,,,Intersection Results and Operation Status
@anchor{developers/apis/seti/seti the-seti-intersection-peer-002dto-002dpeer-protocol}@anchor{2f3}@anchor{developers/apis/seti/seti the-seti-intersection-peer-to-peer-protocol}@anchor{2f4}
@subsubsection The SETI Intersection Peer-to-Peer Protocol


The intersection protocol operates over CADET and starts with a
GNUNET_MESSAGE_TYPE_SETI_P2P_OPERATION_REQUEST being sent by the peer
initiating the operation to the peer listening for inbound requests. It
includes the number of elements of the initiating peer, which is used to
decide which side will send a Bloom filter first.

The listening peer checks if the operation type and application
identifier are acceptable for its current state. If not, it responds
with a GNUNET_MESSAGE_TYPE_SETI_RESULT and a status of
GNUNET_SETI_STATUS_FAILURE (and terminates the CADET channel).

If the application accepts the request, the listener sends back a
@code{GNUNET_MESSAGE_TYPE_SETI_P2P_ELEMENT_INFO} if it has more elements in
the set than the client. Otherwise, it immediately starts with the Bloom
filter exchange. If the initiator receives a
@code{GNUNET_MESSAGE_TYPE_SETI_P2P_ELEMENT_INFO} response, it beings the
Bloom filter exchange, unless the set size is indicated to be zero, in
which case the intersection is considered finished after just the
initial handshake.

@node The Bloom filter exchange in SETI,Intersection Salt,Intersection Results and Operation Status,SETI
@anchor{developers/apis/seti/seti id11}@anchor{2f5}@anchor{developers/apis/seti/seti the-bloom-filter-exchange-in-seti}@anchor{2f6}
@subsubsection The Bloom filter exchange in SETI


In this phase, each peer transmits a Bloom filter over the remaining
keys of the local set to the other peer using a
@code{GNUNET_MESSAGE_TYPE_SETI_P2P_BF} message. This message additionally
includes the number of elements left in the sender's set, as well as the
XOR over all of the keys in that set.

The number of bits 'k' set per element in the Bloom filter is calculated
based on the relative size of the two sets. Furthermore, the size of the
Bloom filter is calculated based on 'k' and the number of elements in
the set to maximize the amount of data filtered per byte transmitted on
the wire (while avoiding an excessively high number of iterations).

The receiver of the message removes all elements from its local set that
do not pass the Bloom filter test. It then checks if the set size of the
sender and the XOR over the keys match what is left of its own set. If
they do, it sends a @code{GNUNET_MESSAGE_TYPE_SETI_P2P_DONE} back to
indicate that the latest set is the final result. Otherwise, the
receiver starts another Bloom filter exchange, except this time as the
sender.

@node Intersection Salt,,The Bloom filter exchange in SETI,SETI
@anchor{developers/apis/seti/seti id12}@anchor{2f7}@anchor{developers/apis/seti/seti intersection-salt}@anchor{2f8}
@subsubsection Intersection Salt


Bloom filter operations are probabilistic: With some non-zero
probability the test may incorrectly say an element is in the set, even
though it is not.

To mitigate this problem, the intersection protocol iterates exchanging
Bloom filters using a different random 32-bit salt in each iteration
(the salt is also included in the message). With different salts, set
operations may fail for different elements. Merging the results from the
executions, the probability of failure drops to zero.

The iterations terminate once both peers have established that they have
sets of the same size, and where the XOR over all keys computes the same
512-bit value (leaving a failure probability of 2-511).

@geindex SETU; subsystem

@node SETU,,SETI,Peer-to-Peer Set Operations
@anchor{developers/apis/setu/setu doc}@anchor{2f9}@anchor{developers/apis/setu/setu setu}@anchor{2fa}@anchor{developers/apis/setu/setu setu-subsystem-dev}@anchor{2fb}
@subsubsection SETU


@menu
* Union Set API:: 
* Union Listeners:: 
* Union Operations:: 
* Supplying a Set for Union:: 
* The Union Result Callback:: 
* Creating Union Sets:: 
* Listeners for Union:: 
* Initiating Union Operations:: 
* Modifying Union Sets:: 
* Union Results and Operation Status:: 
* The SETU Union Peer-to-Peer Protocol:: 

@end menu

@node Union Set API,Union Listeners,,SETU
@anchor{developers/apis/setu/setu id1}@anchor{2fc}@anchor{developers/apis/setu/setu union-set-api}@anchor{2fd}
@subsubsection Union Set API


New sets are created with @code{GNUNET_SETU_create}. Only the local peer's
configuration (as each set has its own client connection) must be
provided. The set exists until either the client calls
@code{GNUNET_SETU_destroy} or the client's connection to the service is
disrupted. In the latter case, the client is notified by the return
value of functions dealing with sets. This return value must always be
checked.

Elements are added with @code{GNUNET_SETU_add_element}.

@node Union Listeners,Union Operations,Union Set API,SETU
@anchor{developers/apis/setu/setu id2}@anchor{2fe}@anchor{developers/apis/setu/setu union-listeners}@anchor{2ff}
@subsubsection Union Listeners


Listeners are created with @code{GNUNET_SETU_listen}. Each time time a
remote peer suggests a set operation with an application id and
operation type matching a listener, the listener's callback is invoked.
The client then must synchronously call either @code{GNUNET_SETU_accept} or
@code{GNUNET_SETU_reject}. Note that the operation will not be started
until the client calls @code{GNUNET_SETU_commit} (see Section "Supplying a
Set").

@node Union Operations,Supplying a Set for Union,Union Listeners,SETU
@anchor{developers/apis/setu/setu id3}@anchor{300}@anchor{developers/apis/setu/setu union-operations}@anchor{301}
@subsubsection Union Operations


Operations to be initiated by the local peer are created with
@code{GNUNET_SETU_prepare}. Note that the operation will not be started
until the client calls @code{GNUNET_SETU_commit} (see Section "Supplying a
Set").

@node Supplying a Set for Union,The Union Result Callback,Union Operations,SETU
@anchor{developers/apis/setu/setu id4}@anchor{302}@anchor{developers/apis/setu/setu supplying-a-set-for-union}@anchor{303}
@subsubsection Supplying a Set for Union


To create symmetry between the two ways of starting a set operation
(accepting and initiating it), the operation handles returned by
@code{GNUNET_SETU_accept} and @code{GNUNET_SETU_prepare} do not yet have a set
to operate on, thus they can not do any work yet.

The client must call @code{GNUNET_SETU_commit} to specify a set to use for
an operation. @code{GNUNET_SETU_commit} may only be called once per set
operation.

@node The Union Result Callback,Creating Union Sets,Supplying a Set for Union,SETU
@anchor{developers/apis/setu/setu id5}@anchor{304}@anchor{developers/apis/setu/setu the-union-result-callback}@anchor{305}
@subsubsection The Union Result Callback


Clients must specify both a result mode and a result callback with
@code{GNUNET_SETU_accept} and @code{GNUNET_SETU_prepare}. The result callback
with a status indicating either that an element was received,
transmitted to the other peer (if this information was requested), or if
the operation failed or ultimately succeeded.

@menu
* The SETU Client-Service Protocol:: 

@end menu

@node The SETU Client-Service Protocol,,,The Union Result Callback
@anchor{developers/apis/setu/setu the-setu-client-002dservice-protocol}@anchor{306}@anchor{developers/apis/setu/setu the-setu-client-service-protocol}@anchor{307}
@subsubsection The SETU Client-Service Protocol


@node Creating Union Sets,Listeners for Union,The Union Result Callback,SETU
@anchor{developers/apis/setu/setu creating-union-sets}@anchor{308}@anchor{developers/apis/setu/setu id6}@anchor{309}
@subsubsection Creating Union Sets


For each set of a client, there exists a client connection to the
service. Sets are created by sending the @code{GNUNET_SERVICE_SETU_CREATE}
message over a new client connection. Multiple operations for one set
are multiplexed over one client connection, using a request id supplied
by the client.

@node Listeners for Union,Initiating Union Operations,Creating Union Sets,SETU
@anchor{developers/apis/setu/setu id7}@anchor{30a}@anchor{developers/apis/setu/setu listeners-for-union}@anchor{30b}
@subsubsection Listeners for Union


Each listener also requires a separate client connection. By sending the
@code{GNUNET_SERVICE_SETU_LISTEN} message, the client notifies the service
of the application id and operation type it is interested in. A client
rejects an incoming request by sending @code{GNUNET_SERVICE_SETU_REJECT} on
the listener's client connection. In contrast, when accepting an
incoming request, a @code{GNUNET_SERVICE_SETU_ACCEPT} message must be sent
over the set that is supplied for the set operation.

@node Initiating Union Operations,Modifying Union Sets,Listeners for Union,SETU
@anchor{developers/apis/setu/setu id8}@anchor{30c}@anchor{developers/apis/setu/setu initiating-union-operations}@anchor{30d}
@subsubsection Initiating Union Operations


Operations with remote peers are initiated by sending a
@code{GNUNET_SERVICE_SETU_EVALUATE} message to the service. The client
connection that this message is sent by determines the set to use.

@node Modifying Union Sets,Union Results and Operation Status,Initiating Union Operations,SETU
@anchor{developers/apis/setu/setu id9}@anchor{30e}@anchor{developers/apis/setu/setu modifying-union-sets}@anchor{30f}
@subsubsection Modifying Union Sets


Sets are modified with the @code{GNUNET_SERVICE_SETU_ADD} message.

@node Union Results and Operation Status,The SETU Union Peer-to-Peer Protocol,Modifying Union Sets,SETU
@anchor{developers/apis/setu/setu id10}@anchor{310}@anchor{developers/apis/setu/setu union-results-and-operation-status}@anchor{311}
@subsubsection Union Results and Operation Status


The service notifies the client of result elements and success/failure
of a set operation with the @code{GNUNET_SERVICE_SETU_RESULT} message.

@node The SETU Union Peer-to-Peer Protocol,,Union Results and Operation Status,SETU
@anchor{developers/apis/setu/setu the-setu-union-peer-002dto-002dpeer-protocol}@anchor{312}@anchor{developers/apis/setu/setu the-setu-union-peer-to-peer-protocol}@anchor{313}
@subsubsection The SETU Union Peer-to-Peer Protocol


The SET union protocol is based on Eppstein's efficient set
reconciliation without prior context. You should read this paper first
if you want to understand the protocol.

@cartouche
@quotation Todo 
Link to Eppstein's paper!
@end quotation
@end cartouche

The union protocol operates over CADET and starts with a
GNUNET_MESSAGE_TYPE_SETU_P2P_OPERATION_REQUEST being sent by the peer
initiating the operation to the peer listening for inbound requests. It
includes the number of elements of the initiating peer, which is
currently not used.

The listening peer checks if the operation type and application
identifier are acceptable for its current state. If not, it responds
with a @code{GNUNET_MESSAGE_TYPE_SETU_RESULT} and a status of
@code{GNUNET_SETU_STATUS_FAILURE} (and terminates the CADET channel).

If the application accepts the request, it sends back a strata estimator
using a message of type GNUNET_MESSAGE_TYPE_SETU_P2P_SE. The initiator
evaluates the strata estimator and initiates the exchange of invertible
Bloom filters, sending a GNUNET_MESSAGE_TYPE_SETU_P2P_IBF.

During the IBF exchange, if the receiver cannot invert the Bloom filter
or detects a cycle, it sends a larger IBF in response (up to a defined
maximum limit; if that limit is reached, the operation fails). Elements
decoded while processing the IBF are transmitted to the other peer using
GNUNET_MESSAGE_TYPE_SETU_P2P_ELEMENTS, or requested from the other peer
using GNUNET_MESSAGE_TYPE_SETU_P2P_ELEMENT_REQUESTS messages, depending
on the sign observed during decoding of the IBF. Peers respond to a
GNUNET_MESSAGE_TYPE_SETU_P2P_ELEMENT_REQUESTS message with the
respective element in a GNUNET_MESSAGE_TYPE_SETU_P2P_ELEMENTS message.
If the IBF fully decodes, the peer responds with a
GNUNET_MESSAGE_TYPE_SETU_P2P_DONE message instead of another
GNUNET_MESSAGE_TYPE_SETU_P2P_IBF.

All Bloom filter operations use a salt to mingle keys before hashing
them into buckets, such that future iterations have a fresh chance of
succeeding if they failed due to collisions before.

@geindex STATISTICS; subsystem

@node STATISTICS,TRANSPORT,Peer-to-Peer Set Operations,Service APIs
@anchor{developers/apis/statistics doc}@anchor{314}@anchor{developers/apis/statistics statistics}@anchor{315}@anchor{developers/apis/statistics statistics-subsystem-dev}@anchor{316}
@subsection STATISTICS


`libgnunetstatistics' is the library containing the API for the
STATISTICS subsystem. Any process requiring to use STATISTICS should use
this API by to open a connection to the STATISTICS service. This is done
by calling the function @code{GNUNET_STATISTICS_create()}. This function
takes the subsystem's name which is trying to use STATISTICS and a
configuration. All values written to STATISTICS with this connection
will be placed in the section corresponding to the given subsystem's
name. The connection to STATISTICS can be destroyed with the function
@code{GNUNET_STATISTICS_destroy()}. This function allows for the connection
to be destroyed immediately or upon transferring all pending write
requests to the service.

Note: STATISTICS subsystem can be disabled by setting @code{DISABLE = YES}
under the @code{[STATISTICS]} section in the configuration. With such a
configuration all calls to @code{GNUNET_STATISTICS_create()} return
@code{NULL} as the STATISTICS subsystem is unavailable and no other
functions from the API can be used.

@menu
* Statistics retrieval:: 
* Setting statistics and updating them:: 
* Watches:: 
* Statistics retrieval: Statistics retrieval<2>. 
* Setting and updating statistics:: 
* Watching for updates:: 

@end menu

@node Statistics retrieval,Setting statistics and updating them,,STATISTICS
@anchor{developers/apis/statistics id1}@anchor{317}@anchor{developers/apis/statistics statistics-retrieval}@anchor{318}
@subsubsection Statistics retrieval


Once a connection to the statistics service is obtained, information
about any other system which uses statistics can be retrieved with the
function GNUNET_STATISTICS_get(). This function takes the connection
handle, the name of the subsystem whose information we are interested in
(a @code{NULL} value will retrieve information of all available subsystems
using STATISTICS), the name of the statistic we are interested in (a
@code{NULL} value will retrieve all available statistics), a continuation
callback which is called when all of requested information is retrieved,
an iterator callback which is called for each parameter in the retrieved
information and a closure for the aforementioned callbacks. The library
then invokes the iterator callback for each value matching the request.

Call to @code{GNUNET_STATISTICS_get()} is asynchronous and can be canceled
with the function @code{GNUNET_STATISTICS_get_cancel()}. This is helpful
when retrieving statistics takes too long and especially when we want to
shutdown and cleanup everything.

@node Setting statistics and updating them,Watches,Statistics retrieval,STATISTICS
@anchor{developers/apis/statistics id2}@anchor{319}@anchor{developers/apis/statistics setting-statistics-and-updating-them}@anchor{31a}
@subsubsection Setting statistics and updating them


So far we have seen how to retrieve statistics, here we will learn how
we can set statistics and update them so that other subsystems can
retrieve them.

A new statistic can be set using the function
@code{GNUNET_STATISTICS_set()}. This function takes the name of the
statistic and its value and a flag to make the statistic persistent. The
value of the statistic should be of the type @code{uint64_t}. The function
does not take the name of the subsystem; it is determined from the
previous @code{GNUNET_STATISTICS_create()} invocation. If the given
statistic is already present, its value is overwritten.

An existing statistics can be updated, i.e its value can be increased or
decreased by an amount with the function @code{GNUNET_STATISTICS_update()}.
The parameters to this function are similar to
@code{GNUNET_STATISTICS_set()}, except that it takes the amount to be
changed as a type @code{int64_t} instead of the value.

The library will combine multiple set or update operations into one
message if the client performs requests at a rate that is faster than
the available IPC with the STATISTICS service. Thus, the client does not
have to worry about sending requests too quickly.

@node Watches,Statistics retrieval<2>,Setting statistics and updating them,STATISTICS
@anchor{developers/apis/statistics id3}@anchor{31b}@anchor{developers/apis/statistics watches}@anchor{31c}
@subsubsection Watches


As interesting feature of STATISTICS lies in serving notifications
whenever a statistic of our interest is modified. This is achieved by
registering a watch through the function @code{GNUNET_STATISTICS_watch()}.
The parameters of this function are similar to those of
@code{GNUNET_STATISTICS_get()}. Changes to the respective statistic's value
will then cause the given iterator callback to be called. Note: A watch
can only be registered for a specific statistic. Hence the subsystem
name and the parameter name cannot be @code{NULL} in a call to
@code{GNUNET_STATISTICS_watch()}.

A registered watch will keep notifying any value changes until
@code{GNUNET_STATISTICS_watch_cancel()} is called with the same parameters
that are used for registering the watch.

@menu
* The STATISTICS Client-Service Protocol:: 

@end menu

@node The STATISTICS Client-Service Protocol,,,Watches
@anchor{developers/apis/statistics the-statistics-client-002dservice-protocol}@anchor{31d}@anchor{developers/apis/statistics the-statistics-client-service-protocol}@anchor{31e}
@subsubsection The STATISTICS Client-Service Protocol


@node Statistics retrieval<2>,Setting and updating statistics,Watches,STATISTICS
@anchor{developers/apis/statistics id4}@anchor{31f}@anchor{developers/apis/statistics statistics-retrieval2}@anchor{320}
@subsubsection Statistics retrieval


To retrieve statistics, the client transmits a message of type
@code{GNUNET_MESSAGE_TYPE_STATISTICS_GET} containing the given subsystem
name and statistic parameter to the STATISTICS service. The service
responds with a message of type @code{GNUNET_MESSAGE_TYPE_STATISTICS_VALUE}
for each of the statistics parameters that match the client request for
the client. The end of information retrieved is signaled by the service
by sending a message of type @code{GNUNET_MESSAGE_TYPE_STATISTICS_END}.

@node Setting and updating statistics,Watching for updates,Statistics retrieval<2>,STATISTICS
@anchor{developers/apis/statistics id5}@anchor{321}@anchor{developers/apis/statistics setting-and-updating-statistics}@anchor{322}
@subsubsection Setting and updating statistics


The subsystem name, parameter name, its value and the persistence flag
are communicated to the service through the message
@code{GNUNET_MESSAGE_TYPE_STATISTICS_SET}.

When the service receives a message of type
@code{GNUNET_MESSAGE_TYPE_STATISTICS_SET}, it retrieves the subsystem name
and checks for a statistic parameter with matching the name given in the
message. If a statistic parameter is found, the value is overwritten by
the new value from the message; if not found then a new statistic
parameter is created with the given name and value.

In addition to just setting an absolute value, it is possible to perform
a relative update by sending a message of type
@code{GNUNET_MESSAGE_TYPE_STATISTICS_SET} with an update flag
(@code{GNUNET_STATISTICS_SETFLAG_RELATIVE}) signifying that the value in
the message should be treated as an update value.

@node Watching for updates,,Setting and updating statistics,STATISTICS
@anchor{developers/apis/statistics id6}@anchor{323}@anchor{developers/apis/statistics watching-for-updates}@anchor{324}
@subsubsection Watching for updates


The function registers the watch at the service by sending a message of
type @code{GNUNET_MESSAGE_TYPE_STATISTICS_WATCH}. The service then sends
notifications through messages of type
@code{GNUNET_MESSAGE_TYPE_STATISTICS_WATCH_VALUE} whenever the statistic
parameter's value is changed.

@geindex TRANSPORT; subsystem

@node TRANSPORT,,STATISTICS,Service APIs
@anchor{developers/apis/transport doc}@anchor{325}@anchor{developers/apis/transport transport}@anchor{326}@anchor{developers/apis/transport transport-subsystem-dev}@anchor{327}
@subsection TRANSPORT


@menu
* Address validation protocol:: 
* NAT library:: 
* Distance-Vector plugin:: 
* SMTP plugin:: 
* Why use SMTP for a peer-to-peer transport?:: 
* How does it work?:: 
* How do I configure my peer?:: 
* How do I test if it works?:: 
* How fast is it?:: 
* Bluetooth plugin:: 
* What do I need to use the Bluetooth plugin transport?:: 
* How does it work2?:: 
* What possible errors should I be aware of?:: 
* How do I configure my peer2?:: 
* How can I test it?:: 
* The implementation of the Bluetooth transport plugin:: 
* WLAN plugin:: 

@end menu

@node Address validation protocol,NAT library,,TRANSPORT
@anchor{developers/apis/transport address-validation-protocol}@anchor{328}@anchor{developers/apis/transport id1}@anchor{329}
@subsubsection Address validation protocol


This section documents how the GNUnet transport service validates
connections with other peers. It is a high-level description of the
protocol necessary to understand the details of the implementation. It
should be noted that when we talk about PING and PONG messages in this
section, we refer to transport-level PING and PONG messages, which are
different from core-level PING and PONG messages (both in implementation
and function).

The goal of transport-level address validation is to minimize the
chances of a successful man-in-the-middle attack against GNUnet peers on
the transport level. Such an attack would not allow the adversary to
decrypt the P2P transmissions, but a successful attacker could at least
measure traffic volumes and latencies (raising the adversaries
capabilities by those of a global passive adversary in the worst case).
The scenarios we are concerned about is an attacker, Mallory, giving a
@code{HELLO} to Alice that claims to be for Bob, but contains Mallory's IP
address instead of Bobs (for some transport). Mallory would then forward
the traffic to Bob (by initiating a connection to Bob and claiming to be
Alice). As a further complication, the scheme has to work even if say
Alice is behind a NAT without traversal support and hence has no address
of her own (and thus Alice must always initiate the connection to Bob).

An additional constraint is that @code{HELLO} messages do not contain a
cryptographic signature since other peers must be able to edit (i.e.
remove) addresses from the @code{HELLO} at any time (this was not true in
GNUnet 0.8.x). A basic `assumption' is that each peer knows the set of
possible network addresses that it `might' be reachable under (so for
example, the external IP address of the NAT plus the LAN address(es)
with the respective ports).

The solution is the following. If Alice wants to validate that a given
address for Bob is valid (i.e. is actually established `directly' with
the intended target), she sends a PING message over that connection to
Bob. Note that in this case, Alice initiated the connection so only
Alice knows which address was used for sure (Alice may be behind NAT, so
whatever address Bob sees may not be an address Alice knows she has).
Bob checks that the address given in the @code{PING} is actually one of
Bob's addresses (ie: does not belong to Mallory), and if it is, sends
back a @code{PONG} (with a signature that says that Bob owns/uses the
address from the @code{PING}). Alice checks the signature and is happy if
it is valid and the address in the @code{PONG} is the address Alice used.
This is similar to the 0.8.x protocol where the @code{HELLO} contained a
signature from Bob for each address used by Bob. Here, the purpose code
for the signature is @code{GNUNET_SIGNATURE_PURPOSE_TRANSPORT_PONG_OWN}.
After this, Alice will remember Bob's address and consider the address
valid for a while (12h in the current implementation). Note that after
this exchange, Alice only considers Bob's address to be valid, the
connection itself is not considered 'established'. In particular, Alice
may have many addresses for Bob that Alice considers valid.

The @code{PONG} message is protected with a nonce/challenge against replay
attacks (replay@footnote{http://en.wikipedia.org/wiki/Replay_attack}) and
uses an expiration time for the signature (but those are almost
implementation details).

NAT library
.. _NAT-library:

@node NAT library,Distance-Vector plugin,Address validation protocol,TRANSPORT
@anchor{developers/apis/transport nat-library}@anchor{32a}
@subsubsection NAT library


The goal of the GNUnet NAT library is to provide a general-purpose API
for NAT traversal `without' third-party support. So protocols that
involve contacting a third peer to help establish a connection between
two peers are outside of the scope of this API. That does not mean that
GNUnet doesn't support involving a third peer (we can do this with the
distance-vector transport or using application-level protocols), it just
means that the NAT API is not concerned with this possibility. The API
is written so that it will work for IPv6-NAT in the future as well as
current IPv4-NAT. Furthermore, the NAT API is always used, even for
peers that are not behind NAT --- in that case, the mapping provided is
simply the identity.

NAT traversal is initiated by calling @code{GNUNET_NAT_register}. Given a
set of addresses that the peer has locally bound to (TCP or UDP), the
NAT library will return (via callback) a (possibly longer) list of
addresses the peer `might' be reachable under. Internally, depending
on the configuration, the NAT library will try to punch a hole (using
UPnP) or just "know" that the NAT was manually punched and generate
the respective external IP address (the one that should be globally
visible) based on the given information.

The NAT library also supports ICMP-based NAT traversal. Here, the other
peer can request connection-reversal by this peer (in this special case,
the peer is even allowed to configure a port number of zero). If the NAT
library detects a connection-reversal request, it returns the respective
target address to the client as well. It should be noted that
connection-reversal is currently only intended for TCP, so other plugins
`must' pass @code{NULL} for the reversal callback. Naturally, the NAT
library also supports requesting connection reversal from a remote peer
(@code{GNUNET_NAT_run_client}).

Once initialized, the NAT handle can be used to test if a given address
is possibly a valid address for this peer (@code{GNUNET_NAT_test_address}).
This is used for validating our addresses when generating PONGs.

Finally, the NAT library contains an API to test if our NAT
configuration is correct. Using @code{GNUNET_NAT_test_start} `before'
binding to the respective port, the NAT library can be used to test if
the configuration works. The test function act as a local client,
initialize the NAT traversal and then contact a @code{gnunet-nat-server}
(running by default on @code{gnunet.org}) and ask for a connection to be
established. This way, it is easy to test if the current NAT
configuration is valid.

@node Distance-Vector plugin,SMTP plugin,NAT library,TRANSPORT
@anchor{developers/apis/transport distance-002dvector-plugin}@anchor{32b}@anchor{developers/apis/transport distance-vector-plugin}@anchor{32c}
@subsubsection Distance-Vector plugin


The Distance Vector (DV) transport is a transport mechanism that allows
peers to act as relays for each other, thereby connecting peers that
would otherwise be unable to connect. This gives a larger connection set
to applications that may work better with more peers to choose from (for
example, File Sharing and/or DHT).

The Distance Vector transport essentially has two functions. The first
is "gossiping" connection information about more distant peers to
directly connected peers. The second is taking messages intended for
non-directly connected peers and encapsulating them in a DV wrapper that
contains the required information for routing the message through
forwarding peers. Via gossiping, optimal routes through the known DV
neighborhood are discovered and utilized and the message encapsulation
provides some benefits in addition to simply getting the message from
the correct source to the proper destination.

The gossiping function of DV provides an up to date routing table of
peers that are available up to some number of hops. We call this a
fisheye view of the network (like a fish, nearby objects are known while
more distant ones unknown). Gossip messages are sent only to directly
connected peers, but they are sent about other knowns peers within the
"fisheye distance". Whenever two peers connect, they immediately
gossip to each other about their appropriate other neighbors. They also
gossip about the newly connected peer to previously connected neighbors.
In order to keep the routing tables up to date, disconnect notifications
are propagated as gossip as well (because disconnects may not be
sent/received, timeouts are also used remove stagnant routing table
entries).

Routing of messages via DV is straightforward. When the DV transport is
notified of a message destined for a non-direct neighbor, the
appropriate forwarding peer is selected, and the base message is
encapsulated in a DV message which contains information about the
initial peer and the intended recipient. At each forwarding hop, the
initial peer is validated (the forwarding peer ensures that it has the
initial peer in its neighborhood, otherwise the message is dropped).
Next the base message is re-encapsulated in a new DV message for the
next hop in the forwarding chain (or delivered to the current peer, if
it has arrived at the destination).

Assume a three peer network with peers Alice, Bob and Carol. Assume that

@example
Alice <-> Bob and Bob <-> Carol
@end example

are direct (e.g. over TCP or UDP transports) connections, but that Alice
cannot directly connect to Carol. This may be the case due to NAT or
firewall restrictions, or perhaps based on one of the peers respective
configurations. If the Distance Vector transport is enabled on all three
peers, it will automatically discover (from the gossip protocol) that
Alice and Carol can connect via Bob and provide a "virtual" Alice <->
Carol connection. Routing between Alice and Carol happens as follows;
Alice creates a message destined for Carol and notifies the DV transport
about it. The DV transport at Alice looks up Carol in the routing table
and finds that the message must be sent through Bob for Carol. The
message is encapsulated setting Alice as the initiator and Carol as the
destination and sent to Bob. Bob receives the messages, verifies that
both Alice and Carol are known to Bob, and re-wraps the message in a new
DV message for Carol. The DV transport at Carol receives this message,
unwraps the original message, and delivers it to Carol as though it came
directly from Alice.

SMTP plugin
.. _SMTP-plugin:

@node SMTP plugin,Why use SMTP for a peer-to-peer transport?,Distance-Vector plugin,TRANSPORT
@anchor{developers/apis/transport smtp-plugin}@anchor{32d}
@subsubsection SMTP plugin


@cartouche
@quotation Todo 
Update?
@end quotation
@end cartouche

This section describes the new SMTP transport plugin for GNUnet as it
exists in the 0.7.x and 0.8.x branch. SMTP support is currently not
available in GNUnet 0.9.x. This page also describes the transport layer
abstraction (as it existed in 0.7.x and 0.8.x) in more detail and gives
some benchmarking results. The performance results presented are quite
old and maybe outdated at this point. For the readers in the year 2019,
you will notice by the mention of version 0.7, 0.8, and 0.9 that this
section has to be taken with your usual grain of salt and be updated
eventually.


@itemize -

@item 
Why use SMTP for a peer-to-peer transport?

@item 
SMTPHow does it work?

@item 
How do I configure my peer?

@item 
How do I test if it works?

@item 
How fast is it?

@item 
Is there any additional documentation?
@end itemize

@node Why use SMTP for a peer-to-peer transport?,How does it work?,SMTP plugin,TRANSPORT
@anchor{developers/apis/transport why-use-smtp-for-a-peer-002dto-002dpeer-transport-003f}@anchor{32e}@anchor{developers/apis/transport why-use-smtp-for-a-peer-to-peer-transport}@anchor{32f}
@subsubsection Why use SMTP for a peer-to-peer transport?


There are many reasons why one would not want to use SMTP:


@itemize -

@item 
SMTP is using more bandwidth than TCP, UDP or HTTP

@item 
SMTP has a much higher latency.

@item 
SMTP requires significantly more computation (encoding and decoding
time) for the peers.

@item 
SMTP is significantly more complicated to configure.

@item 
SMTP may be abused by tricking GNUnet into sending mail
to non-participating third parties.
@end itemize

So why would anybody want to use SMTP?


@itemize -

@item 
SMTP can be used to contact peers behind NAT boxes (in virtual
private networks).

@item 
SMTP can be used to circumvent policies that limit or prohibit
peer-to-peer traffic by masking as "legitimate" traffic.

@item 
SMTP uses E-mail addresses which are independent of a specific IP,
which can be useful to address peers that use dynamic IP addresses.

@item 
SMTP can be used to initiate a connection (e.g. initial address
exchange) and peers can then negotiate the use of a more efficient
protocol (e.g. TCP) for the actual communication.
@end itemize

In summary, SMTP can for example be used to send a message to a peer
behind a NAT box that has a dynamic IP to tell the peer to establish a
TCP connection to a peer outside of the private network. Even an
extraordinary overhead for this first message would be irrelevant in
this type of situation.

@node How does it work?,How do I configure my peer?,Why use SMTP for a peer-to-peer transport?,TRANSPORT
@anchor{developers/apis/transport how-does-it-work}@anchor{330}@anchor{developers/apis/transport how-does-it-work-003f}@anchor{331}
@subsubsection How does it work?


When a GNUnet peer needs to send a message to another GNUnet peer that
has advertised (only) an SMTP transport address, GNUnet base64-encodes
the message and sends it in an E-mail to the advertised address. The
advertisement contains a filter which is placed in the E-mail header,
such that the receiving host can filter the tagged E-mails and forward
it to the GNUnet peer process. The filter can be specified individually
by each peer and be changed over time. This makes it impossible to
censor GNUnet E-mail messages by searching for a generic filter.

@node How do I configure my peer?,How do I test if it works?,How does it work?,TRANSPORT
@anchor{developers/apis/transport how-do-i-configure-my-peer}@anchor{332}@anchor{developers/apis/transport how-do-i-configure-my-peer-003f}@anchor{333}
@subsubsection How do I configure my peer?


First, you need to configure @code{procmail} to filter your inbound E-mail
for GNUnet traffic. The GNUnet messages must be delivered into a pipe,
for example @code{/tmp/gnunet.smtp}. You also need to define a filter that
is used by @code{procmail} to detect GNUnet messages. You are free to
choose whichever filter you like, but you should make sure that it does
not occur in your other E-mail. In our example, we will use
@code{X-mailer: GNUnet}. The @code{~/.procmailrc} configuration file then
looks like this:

@example
:0:
* ^X-mailer: GNUnet
/tmp/gnunet.smtp
# where do you want your other e-mail delivered to
# (default: /var/spool/mail/)
:0: /var/spool/mail/
@end example

After adding this file, first make sure that your regular E-mail still
works (e.g. by sending an E-mail to yourself). Then edit the GNUnet
configuration. In the section @code{SMTP} you need to specify your E-mail
address under @code{EMAIL}, your mail server (for outgoing mail) under
@code{SERVER}, the filter (X-mailer: GNUnet in the example) under
@code{FILTER} and the name of the pipe under @code{PIPE}. The completed
section could then look like this:

@example
EMAIL = me@@mail.gnu.org MTU = 65000 SERVER = mail.gnu.org:25 FILTER =
"X-mailer: GNUnet" PIPE = /tmp/gnunet.smtp
@end example

@cartouche
@quotation Todo 
set highlighting for this code block properly.
@end quotation
@end cartouche

Finally, you need to add @code{smtp} to the list of @code{TRANSPORTS} in the
@code{GNUNETD} section. GNUnet peers will use the E-mail address that you
specified to contact your peer until the advertisement times out. Thus,
if you are not sure if everything works properly or if you are not
planning to be online for a long time, you may want to configure this
timeout to be short, e.g. just one hour. For this, set @code{HELLOEXPIRES}
to @code{1} in the @code{GNUNETD} section.

This should be it, but you may probably want to test it first.

@node How do I test if it works?,How fast is it?,How do I configure my peer?,TRANSPORT
@anchor{developers/apis/transport how-do-i-test-if-it-works}@anchor{334}@anchor{developers/apis/transport how-do-i-test-if-it-works-003f}@anchor{335}
@subsubsection How do I test if it works?


Any transport can be subjected to some rudimentary tests using the
@code{gnunet-transport-check} tool. The tool sends a message to the local
node via the transport and checks that a valid message is received.
While this test does not involve other peers and can not check if
firewalls or other network obstacles prohibit proper operation, this is
a great testcase for the SMTP transport since it tests pretty much
nearly all of the functionality.

@code{gnunet-transport-check} should only be used without running
@code{gnunetd} at the same time. By default, @code{gnunet-transport-check}
tests all transports that are specified in the configuration file. But
you can specifically test SMTP by giving the option
@code{--transport=smtp}.

Note that this test always checks if a transport can receive and send.
While you can configure most transports to only receive or only send
messages, this test will only work if you have configured the transport
to send and receive messages.

@node How fast is it?,Bluetooth plugin,How do I test if it works?,TRANSPORT
@anchor{developers/apis/transport how-fast-is-it}@anchor{336}@anchor{developers/apis/transport how-fast-is-it-003f}@anchor{337}
@subsubsection How fast is it?


We have measured the performance of the UDP, TCP and SMTP transport
layer directly and when used from an application using the GNUnet core.
Measuring just the transport layer gives the better view of the actual
overhead of the protocol, whereas evaluating the transport from the
application puts the overhead into perspective from a practical point of
view.

The loopback measurements of the SMTP transport were performed on three
different machines spanning a range of modern SMTP configurations. We
used a PIII-800 running RedHat 7.3 with the Purdue Computer Science
configuration which includes filters for spam. We also used a Xenon 2
GHZ with a vanilla RedHat 8.0 sendmail configuration. Furthermore, we
used qmail on a PIII-1000 running Sorcerer GNU Linux (SGL). The numbers
for UDP and TCP are provided using the SGL configuration. The qmail
benchmark uses qmail's internal filtering whereas the sendmail
benchmarks relies on procmail to filter and deliver the mail. We used
the transport layer to send a message of b bytes (excluding transport
protocol headers) directly to the local machine. This way, network
latency and packet loss on the wire have no impact on the timings. n
messages were sent sequentially over the transport layer, sending
message i+1 after the i-th message was received. All messages were sent
over the same connection and the time to establish the connection was
not taken into account since this overhead is minuscule in practice ---
as long as a connection is used for a significant number of messages.


@multitable {xxxxxxxxxxxxxxxx} {xxxxxxxxxxxx} {xxxxxxxxxxxx} {xxxxxxxxxxxx} {xxxxxxxxxxxx} {xxxxxxxxxxxx} 
@headitem

Transport

@tab

UDP

@tab

TCP

@tab

SMTP
(Purdue
s
endmail)

@tab

SMTP (RH
8.0)

@tab

SMTP
(SGL
qmail)

@item

11 bytes

@tab

31 ms

@tab

55 ms

@tab

781 s

@tab

77 s

@tab

24 s

@item

407 bytes

@tab

37 ms

@tab

62 ms

@tab

789 s

@tab

78 s

@tab

25 s

@item

1,221 bytes

@tab

46 ms

@tab

73 ms

@tab

804 s

@tab

78 s

@tab

25 s

@end multitable


The benchmarks show that UDP and TCP are, as expected, both
significantly faster compared with any of the SMTP services. Among the
SMTP implementations, there can be significant differences depending on
the SMTP configuration. Filtering with an external tool like procmail
that needs to re-parse its configuration for each mail can be very
expensive. Applying spam filters can also significantly impact the
performance of the underlying SMTP implementation. The microbenchmark
shows that SMTP can be a viable solution for initiating peer-to-peer
sessions: a couple of seconds to connect to a peer are probably not even
going to be noticed by users. The next benchmark measures the possible
throughput for a transport. Throughput can be measured by sending
multiple messages in parallel and measuring packet loss. Note that not
only UDP but also the TCP transport can actually loose messages since
the TCP implementation drops messages if the @code{write} to the socket
would block. While the SMTP protocol never drops messages itself, it is
often so slow that only a fraction of the messages can be sent and
received in the given time-bounds. For this benchmark we report the
message loss after allowing t time for sending m messages. If messages
were not sent (or received) after an overall timeout of t, they were
considered lost. The benchmark was performed using two Xeon 2 GHZ
machines running RedHat 8.0 with sendmail. The machines were connected
with a direct 100 MBit Ethernet connection. Figures udp1200, tcp1200 and
smtp-MTUs show that the throughput for messages of size 1,200 octets is
2,343 kbps, 3,310 kbps and 6 kbps for UDP, TCP and SMTP respectively.
The high per-message overhead of SMTP can be improved by increasing the
MTU, for example, an MTU of 12,000 octets improves the throughput to 13
kbps as figure smtp-MTUs shows. Our research paper @ref{338,,[Transport2014]} has
some more details on the benchmarking results.

Bluetooth plugin
.. _Bluetooth-plugin:

@node Bluetooth plugin,What do I need to use the Bluetooth plugin transport?,How fast is it?,TRANSPORT
@anchor{developers/apis/transport bluetooth-plugin}@anchor{339}
@subsubsection Bluetooth plugin


This page describes the new Bluetooth transport plugin for GNUnet. The
plugin is still in the testing stage so don't expect it to work
perfectly. If you have any questions or problems just post them here or
ask on the IRC channel.


@itemize -

@item 
What do I need to use the Bluetooth plugin transport?

@item 
BluetoothHow does it work?

@item 
What possible errors should I be aware of?

@item 
How do I configure my peer?

@item 
How can I test it?
@end itemize

@node What do I need to use the Bluetooth plugin transport?,How does it work2?,Bluetooth plugin,TRANSPORT
@anchor{developers/apis/transport what-do-i-need-to-use-the-bluetooth-plugin-transport}@anchor{33a}@anchor{developers/apis/transport what-do-i-need-to-use-the-bluetooth-plugin-transport-003f}@anchor{33b}
@subsubsection What do I need to use the Bluetooth plugin transport?


If you are a GNU/Linux user and you want to use the Bluetooth transport
plugin you should install the @code{BlueZ} development libraries (if they
aren't already installed). For instructions about how to install the
libraries you should check out the BlueZ site
(http://www.bluez.org@footnote{http://www.bluez.org/}). If you don't know if
you have the necessary libraries, don't worry, just run the GNUnet
configure script and you will be able to see a notification at the end
which will warn you if you don't have the necessary libraries.
@anchor{developers/apis/transport how-does-it-work2-003f}@anchor{33c}
@cartouche
@quotation Todo 
Change to unique title?
@end quotation
@end cartouche

@node How does it work2?,What possible errors should I be aware of?,What do I need to use the Bluetooth plugin transport?,TRANSPORT
@anchor{developers/apis/transport how-does-it-work2}@anchor{33d}
@subsubsection How does it work2?


The Bluetooth transport plugin uses virtually the same code as the WLAN
plugin and only the helper binary is different. The helper takes a
single argument, which represents the interface name and is specified in
the configuration file. Here are the basic steps that are followed by
the helper binary used on GNU/Linux:


@itemize -

@item 
it verifies if the name corresponds to a Bluetooth interface name

@item 
it verifies if the interface is up (if it is not, it tries to bring
it up)

@item 
it tries to enable the page and inquiry scan in order to make the
device discoverable and to accept incoming connection requests `The
above operations require root access so you should start the
transport plugin with root privileges.'

@item 
it finds an available port number and registers a SDP service which
will be used to find out on which port number is the server listening
on and switch the socket in listening mode

@item 
it sends a HELLO message with its address

@item 
finally it forwards traffic from the reading sockets to the STDOUT
and from the STDIN to the writing socket
@end itemize

Once in a while the device will make an inquiry scan to discover the
nearby devices and it will send them randomly HELLO messages for peer
discovery.

@node What possible errors should I be aware of?,How do I configure my peer2?,How does it work2?,TRANSPORT
@anchor{developers/apis/transport what-possible-errors-should-i-be-aware-of}@anchor{33e}@anchor{developers/apis/transport what-possible-errors-should-i-be-aware-of-003f}@anchor{33f}
@subsubsection What possible errors should I be aware of?


`This section is dedicated for GNU/Linux users'

Well there are many ways in which things could go wrong but I will try
to present some tools that you could use to debug and some scenarios.


@itemize -

@item 
@code{bluetoothd -n -d} : use this command to enable logging in the
foreground and to print the logging messages

@item 
@code{hciconfig}: can be used to configure the Bluetooth devices. If you
run it without any arguments it will print information about the
state of the interfaces. So if you receive an error that the device
couldn't be brought up you should try to bring it manually and to see
if it works (use @code{hciconfig -a hciX up}). If you can't and the
Bluetooth address has the form 00:00:00:00:00:00 it means that there
is something wrong with the D-Bus daemon or with the Bluetooth
daemon. Use @code{bluetoothd} tool to see the logs

@item 
@code{sdptool} can be used to control and interrogate SDP servers. If
you encounter problems regarding the SDP server (like the SDP server
is down) you should check out if the D-Bus daemon is running
correctly and to see if the Bluetooth daemon started correctly(use
@code{bluetoothd} tool). Also, sometimes the SDP service could work but
somehow the device couldn't register its service. Use
@code{sdptool browse [dev-address]} to see if the service is registered.
There should be a service with the name of the interface and GNUnet
as provider.

@item 
@code{hcitool} : another useful tool which can be used to configure the
device and to send some particular commands to it.

@item 
@code{hcidump} : could be used for low level debugging
@end itemize
@anchor{developers/apis/transport how-do-i-configure-my-peer2-003f}@anchor{340}
@cartouche
@quotation Todo 
Fix name/referencing now that we're using Sphinx.
@end quotation
@end cartouche

@node How do I configure my peer2?,How can I test it?,What possible errors should I be aware of?,TRANSPORT
@anchor{developers/apis/transport how-do-i-configure-my-peer2}@anchor{341}
@subsubsection How do I configure my peer2?


On GNU/Linux, you just have to be sure that the interface name
corresponds to the one that you want to use. Use the @code{hciconfig} tool
to check that. By default it is set to hci0 but you can change it.

A basic configuration looks like this:

@example
[transport-bluetooth]
# Name of the interface (typically hciX)
INTERFACE = hci0
# Real hardware, no testing
TESTMODE = 0 TESTING_IGNORE_KEYS = ACCEPT_FROM;
@end example

In order to use the Bluetooth transport plugin when the transport
service is started, you must add the plugin name to the default
transport service plugins list. For example:

@example
[transport] ...  PLUGINS = dns bluetooth ...
@end example

If you want to use only the Bluetooth plugin set `PLUGINS = bluetooth'

On Windows, you cannot specify which device to use. The only thing that
you should do is to add `bluetooth' on the plugins list of the transport
service.

@node How can I test it?,The implementation of the Bluetooth transport plugin,How do I configure my peer2?,TRANSPORT
@anchor{developers/apis/transport how-can-i-test-it}@anchor{342}@anchor{developers/apis/transport how-can-i-test-it-003f}@anchor{343}
@subsubsection How can I test it?


If you have two Bluetooth devices on the same machine and you are using
GNU/Linux you must:


@itemize -

@item 
create two different file configuration (one which will use the first
interface (`hci0') and the other which will use the second interface
(`hci1')). Let's name them `peer1.conf' and `peer2.conf'.

@item 
run `gnunet-core -c peerX.conf -i' in order to generate the peers
private keys. The `X' must be replace with 1 or 2.

@item 
run `gnunet-arm -c peerX.conf -s -i=transport' in order to start the
transport service. (Make sure that you have "bluetooth" on the
transport plugins list if the Bluetooth transport service doesn't
start.)

@item 
run `gnunet-core -c peer1.conf -i' to get the first peer's ID. If
you already know your peer ID (you saved it from the first command),
this can be skipped.

@item 
run `gnunet-transport -c peer2.conf -p=PEER1_ID -s' to start sending
data for benchmarking to the other peer.
@end itemize

This scenario will try to connect the second peer to the first one and
then start sending data for benchmarking.

If you have two different machines and your configuration files are good
you can use the same scenario presented on the beginning of this
section.

Another way to test the plugin functionality is to create your own
application which will use the GNUnet framework with the Bluetooth
transport service.

@node The implementation of the Bluetooth transport plugin,WLAN plugin,How can I test it?,TRANSPORT
@anchor{developers/apis/transport id7}@anchor{344}@anchor{developers/apis/transport the-implementation-of-the-bluetooth-transport-plugin}@anchor{345}
@subsubsection The implementation of the Bluetooth transport plugin


This page describes the implementation of the Bluetooth transport
plugin.

First I want to remind you that the Bluetooth transport plugin uses
virtually the same code as the WLAN plugin and only the helper binary is
different. Also the scope of the helper binary from the Bluetooth
transport plugin is the same as the one used for the WLAN transport
plugin: it accesses the interface and then it forwards traffic in both
directions between the Bluetooth interface and stdin/stdout of the
process involved.

The Bluetooth plugin transport could be used both on GNU/Linux and
Windows platforms.


@itemize -

@item 
Linux functionality

@item 
Pending Features
@end itemize

@menu
* Linux functionality:: 
* THE INITIALIZATION:: 
* THE LOOP:: 
* Details about the broadcast implementation:: 
* Pending features:: 

@end menu

@node Linux functionality,THE INITIALIZATION,,The implementation of the Bluetooth transport plugin
@anchor{developers/apis/transport id8}@anchor{346}@anchor{developers/apis/transport linux-functionality}@anchor{347}
@subsubsection Linux functionality


In order to implement the plugin functionality on GNU/Linux I used the
BlueZ stack. For the communication with the other devices I used the
RFCOMM protocol. Also I used the HCI protocol to gain some control over
the device. The helper binary takes a single argument (the name of the
Bluetooth interface) and is separated in two stages:
@anchor{developers/apis/transport the-initialization}@anchor{348}
@cartouche
@quotation Todo 
'THE INITIALIZATION' should be in bigger letters or stand out, not
starting a new section?
@end quotation
@end cartouche

@node THE INITIALIZATION,THE LOOP,Linux functionality,The implementation of the Bluetooth transport plugin
@anchor{developers/apis/transport id10}@anchor{349}
@subsubsection THE INITIALIZATION



@itemize -

@item 
first, it checks if we have root privileges (`Remember that we need
to have root privileges in order to be able to bring the interface up
if it is down or to change its state.').

@item 
second, it verifies if the interface with the given name exists.

`If the interface with that name exists and it is a Bluetooth
interface:'

@item 
it creates a RFCOMM socket which will be used for listening and call
the `open_device' method

On the `open_device' method:


@itemize -

@item 
creates a HCI socket used to send control events to the device

@item 
searches for the device ID using the interface name

@item 
saves the device MAC address

@item 
checks if the interface is down and tries to bring it UP

@item 
checks if the interface is in discoverable mode and tries to make
it discoverable

@item 
closes the HCI socket and binds the RFCOMM one

@item 
switches the RFCOMM socket in listening mode

@item 
registers the SDP service (the service will be used by the other
devices to get the port on which this device is listening on)
@end itemize

@item 
drops the root privileges

`If the interface is not a Bluetooth interface the helper exits with
a suitable error'
@end itemize

@node THE LOOP,Details about the broadcast implementation,THE INITIALIZATION,The implementation of the Bluetooth transport plugin
@anchor{developers/apis/transport id11}@anchor{34a}@anchor{developers/apis/transport the-loop}@anchor{34b}
@subsubsection THE LOOP


The helper binary uses a list where it saves all the connected neighbour
devices (`neighbours.devices') and two buffers (`write_pout' and
`write_std'). The first message which is send is a control message with
the device's MAC address in order to announce the peer presence to the
neighbours. Here are a short description of what happens in the main
loop:


@itemize -

@item 
Every time when it receives something from the STDIN it processes the
data and saves the message in the first buffer (`write_pout'). When
it has something in the buffer, it gets the destination address from
the buffer, searches the destination address in the list (if there is
no connection with that device, it creates a new one and saves it to
the list) and sends the message.

@item 
Every time when it receives something on the listening socket it
accepts the connection and saves the socket on a list with the
reading sockets.

@item 
Every time when it receives something from a reading socket it parses
the message, verifies the CRC and saves it in the `write_std' buffer
in order to be sent later to the STDOUT.
@end itemize

So in the main loop we use the select function to wait until one of the
file descriptor saved in one of the two file descriptors sets used is
ready to use. The first set (`rfds') represents the reading set and it
could contain the list with the reading sockets, the STDIN file
descriptor or the listening socket. The second set (`wfds') is the
writing set and it could contain the sending socket or the STDOUT file
descriptor. After the select function returns, we check which file
descriptor is ready to use and we do what is supposed to do on that kind
of event. `For example:' if it is the listening socket then we accept a
new connection and save the socket in the reading list; if it is the
STDOUT file descriptor, then we write to STDOUT the message from the
`write_std' buffer.

To find out on which port a device is listening on we connect to the
local SDP server and search the registered service for that device.

`You should be aware of the fact that if the device fails to connect to
another one when trying to send a message it will attempt one more time.
If it fails again, then it skips the message.' `Also you should know
that the transport Bluetooth plugin has support for'`broadcast
messages'`.'

@node Details about the broadcast implementation,Pending features,THE LOOP,The implementation of the Bluetooth transport plugin
@anchor{developers/apis/transport details-about-the-broadcast-implementation}@anchor{34c}@anchor{developers/apis/transport id12}@anchor{34d}
@subsubsection Details about the broadcast implementation


First I want to point out that the broadcast functionality for the
CONTROL messages is not implemented in a conventional way. Since the
inquiry scan time is too big and it will take some time to send a
message to all the discoverable devices I decided to tackle the problem
in a different way. Here is how I did it:


@itemize -

@item 
If it is the first time when I have to broadcast a message I make an
inquiry scan and save all the devices' addresses to a vector.

@item 
After the inquiry scan ends I take the first address from the list
and I try to connect to it. If it fails, I try to connect to the next
one. If it succeeds, I save the socket to a list and send the message
to the device.

@item 
When I have to broadcast another message, first I search on the list
for a new device which I'm not connected to. If there is no new
device on the list I go to the beginning of the list and send the
message to the old devices. After 5 cycles I make a new inquiry scan
to check out if there are new discoverable devices and save them to
the list. If there are no new discoverable devices I reset the
cycling counter and go again through the old list and send messages
to the devices saved in it.
@end itemize

`Therefore':


@itemize -

@item 
every time when I have a broadcast message I look up on the list for
a new device and send the message to it

@item 
if I reached the end of the list for 5 times and I'm connected to all
the devices from the list I make a new inquiry scan. `The number of
the list's cycles after an inquiry scan could be increased by
redefining the MAX_LOOPS variable'

@item 
when there are no new devices I send messages to the old ones.
@end itemize

Doing so, the broadcast control messages will reach the devices but with
delay.

`NOTICE:' When I have to send a message to a certain device first I
check on the broadcast list to see if we are connected to that device.
If not we try to connect to it and in case of success we save the
address and the socket on the list. If we are already connected to that
device we simply use the socket.

@node Pending features,,Details about the broadcast implementation,The implementation of the Bluetooth transport plugin
@anchor{developers/apis/transport id13}@anchor{34e}@anchor{developers/apis/transport pending-features}@anchor{34f}
@subsubsection Pending features



@itemize -

@item 
Implement a testcase for the helper : `The testcase consists of a
program which emulates the plugin and uses the helper. It will
simulate connections, disconnections and data transfers.'
@end itemize

If you have a new idea about a feature of the plugin or suggestions
about how I could improve the implementation you are welcome to comment
or to contact me.

@node WLAN plugin,,The implementation of the Bluetooth transport plugin,TRANSPORT
@anchor{developers/apis/transport id14}@anchor{350}@anchor{developers/apis/transport wlan-plugin}@anchor{351}
@subsubsection WLAN plugin


This section documents how the wlan transport plugin works. Parts which
are not implemented yet or could be better implemented are described at
the end.

@anchor{developers/apis/transport transport2014}@anchor{338}@w{(Transport2014)} 
@indicateurl{https://bib.gnunet.org/date.html#paper_5fshort2014}

@node REST API,C Tutorial,Service APIs,Developer Handbook
@anchor{developers/rest-api/index doc}@anchor{352}@anchor{developers/rest-api/index rest-api}@anchor{353}
@section REST API


@menu
* Configuration: Configuration<2>. 
* Conventions:: 
* Identity:: 
* Namestore:: 
* GNS: GNS<3>. 
* Peerinfo:: 
* Config:: 

@end menu

@node Configuration<2>,Conventions,,REST API
@anchor{developers/rest-api/configuration doc}@anchor{354}@anchor{developers/rest-api/configuration configuration}@anchor{355}
@subsection Configuration


In order to start the REST service, execute:

@example
$ gnunet-arm -i rest
@end example

The REST service will listen by default on port 7776.
The service is run by each user so you may have to modify the port accordingly:

@example
$ gnunet-config -s rest -o HTTP_PORT -V 7788
@end example

Note that you may need to authenticate agains the API using HTTP basic authentication.
The REST service autogenerates a password upon first launch.
You can get your user-specific authentication secret by executing:

@example
$ SECRET=$(gnunet-config -f -s rest -o BASIC_AUTH_SECRET_FILE)
@end example

To access the REST API, you can use any HTTP client such as a browser or cURL:

@example
$ curl localhost:7776/identity -u<$USER>:<$SECRET>
@end example

You may disable the authentication if you want to by executing:

@example
$ gnunet-config -s rest -o BASIC_AUTH_ENABLED -V NO
@end example

However, disabling authentication is not recommended.

@node Conventions,Identity,Configuration<2>,REST API
@anchor{developers/rest-api/conventions doc}@anchor{356}@anchor{developers/rest-api/conventions conventions}@anchor{357}
@subsection Conventions


The GNUnet Web API is based in REST principles. Data resources are accessed via HTTP requests to an API endpoint. Where possible, the Web API uses the following HTTP methods for each action:


@multitable {xxxxxxxxxxxx} {xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx} 
@headitem

`Method'

@tab

`Action'

@item

GET

@tab

Retrieve object(s)

@item

POST

@tab

Create new object(s)

@item

PUT

@tab

Edit existing object(s)

@item

DELETE

@tab

Delete existing object(s)

@item

OPTIONS

@tab

Get allowed headers

@end multitable


@node Identity,Namestore,Conventions,REST API
@anchor{developers/rest-api/identity doc}@anchor{358}@anchor{developers/rest-api/identity identity}@anchor{359}
@subsection Identity


@menu
* Definition:: 
* Requests:: 

@end menu

@node Definition,Requests,,Identity
@anchor{developers/rest-api/identity definition}@anchor{35a}
@subsubsection Definition


Variables in single quotes @code{'...'} can or must be changed according to your specific case.

@code{public_key} is the public key of an identity.

@code{name} is the name of an identity.

@code{newname} is the new name of an identity for the rename request.

@code{subsystem} is a subsystem, e.g. namestore.

@menu
* Identity: Identity<2>. 
* Error Response:: 
* Response Code:: 

@end menu

@node Identity<2>,Error Response,,Definition
@anchor{developers/rest-api/identity id1}@anchor{35b}
@subsubsection Identity


An identity consists of a @code{public key} and a @code{name}. An identity can be assigned to a @code{subsystem}. Each subsystem can only have one default identity.

@node Error Response,Response Code,Identity<2>,Definition
@anchor{developers/rest-api/identity error-response}@anchor{35c}
@subsubsection Error Response


An error response is sent in the JSON format: @code{@{"error":"*error_description*"@}}

Following numbers are added for references inside the documentation only.

Error descriptions are:

@example
Nr. Error Description           - Explanation
1)  Unknown Error               - Error is not specified
2)  No identity found           - Identity was not found with given name, public key or no identity was found at all
3)  Missing identity public key - Identity public key length is zero
4)  Missing identity name       - Identity name length is zero
5)  Missing subsystem name      - Subsystem name length is zero
6)  No data                     - No JSON data given
7)  Data invalid                - Wrong JSON data given
8)  Rename failed               - Rename request failed due to wrong name, etc.
9)  Setting subsystem failed    - Setting the subsystem for an identity failed (usually this error does not occur)
@end example

Error @code{1)} is always possible and is not listed in following requests.

ATTENTION: Any error message from the Identity API (not REST API) can occur and can be returned in the error response. These responses are not listed here.

@node Response Code,,Error Response,Definition
@anchor{developers/rest-api/identity response-code}@anchor{35d}
@subsubsection Response Code


A response of a message has a HTTP response code. Usually, this code is 200 OK for a successful response. The code changes in some cases:

@example
a) 200 OK         - Normal response (but may contain an error message)
b) 201 Created    - Success after POST request
c) 204 No Content - Success PUT or DELETE request
d) 404 Not Found  - Identity is not found with identifier
e) 409 Conflict   - PUT or POST request not possible due to existing duplicate
@end example

@code{d) 404 Not Found} is always used when the error message is either @code{2)}, @code{3)} or @code{4)}.

@node Requests,,Definition,Identity
@anchor{developers/rest-api/identity requests}@anchor{35e}
@subsubsection Requests


@menu
* GET Requests:: 
* POST Request:: 
* PUT Request:: 
* DELETE Request:: 
* OPTIONS Request:: 

@end menu

@node GET Requests,POST Request,,Requests
@anchor{developers/rest-api/identity get-requests}@anchor{35f}
@subsubsection GET Requests



@multitable {xxxxxxxxxxxxxxxxxxxxxx} {xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx} 
@item

`Title'

@tab

Returns all identities with name and public key

@item

`URL'

@tab

@code{/identity/all}

@item

`Method'

@tab

`GET'

@item

`URL Params'

@tab

none

@item

`Data Params'

@tab

none

@item

`Success Response'

@tab

[@{"pubkey":"`public_key'", "name":"`name'"@},...]

@item

`Error Response'

@tab

@{"error":"`error_desc'"@} @w{^2}

@item

`Attention'

@tab

The response in this request is an array!

@end multitable



@display
@w{ }
@end display




@multitable {xxxxxxxxxxxxxxxxxxxxxx} {xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx} 
@item

`Title'

@tab

Returns only a specific identity

@item

`URL'

@tab

@code{/identity/pubkey/'public_key'} or @code{/identity/name/'name'}

@item

`Method'

@tab

`GET'

@item

`URL Params'

@tab

none

@item

`Data Params'

@tab

none

@item

`Success Response'

@tab

@{"pubkey":"`public_key'", "name":"`name'"@}

@item

`Error Response'

@tab

@{"error":"`error_desc'"@} @w{^2; 3 or 4}

@end multitable



@display
@w{ }
@end display




@multitable {xxxxxxxxxxxxxxxxxxxxxx} {xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx} 
@item

`Title'

@tab

Returns default identity for specific subsystem

@item

`URL'

@tab

@code{/identity/subsystem/'subsystem'}

@item

`Method'

@tab

`GET'

@item

`URL Params'

@tab

none

@item

`Data Params'

@tab

none

@item

`Success Response'

@tab

@{"pubkey":"`public_key'", "name":"`name'"@}

@item

`Error Response'

@tab

@{"error":"`error_desc'"@} @w{^2; 5}

@end multitable


@node POST Request,PUT Request,GET Requests,Requests
@anchor{developers/rest-api/identity post-request}@anchor{360}
@subsubsection POST Request



@multitable {xxxxxxxxxxxxxxxxxxxxxx} {xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx} 
@item

`Title'

@tab

Creates an identity

@item

`URL'

@tab

@code{/identity}

@item

`Method'

@tab

`POST'

@item

`URL Params'

@tab

none

@item

`Data Params'

@tab

@{"name":'`name''@}

@item

`Success Response'

@tab

Response Code: :literal:` b) 201 Created`

@item

`Error Response'

@tab


@display
@{"error":"`error_desc'"@} @w{^6; 7}@w{ }
`or'@w{ }
Response Code: @code{e) 409 Conflict} if name in use@w{ }
@end display



@end multitable


@node PUT Request,DELETE Request,POST Request,Requests
@anchor{developers/rest-api/identity put-request}@anchor{361}
@subsubsection PUT Request



@multitable {xxxxxxxxxxxxxxxxxxxxxx} {xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx} 
@item

`Title'

@tab

Changes name of identity

@item

`URL'

@tab

@code{/identity/pubkey/'public_key'} or @code{/identity/name/'name'}

@item

`Method'

@tab

`PUT'

@item

`URL Params'

@tab

none

@item

`Data Params'

@tab

@{"newname":'`newname''@}

@item

`Success Response'

@tab

Response Code: @code{c) 204 No Content}

@item

`Error Response'

@tab


@display
@{"error":"`error_desc'"@} @w{^2; 3 or 4; 6; 7; 8}@w{ }
`or'@w{ }
Response Code: @code{e) 409 Conflict} if newname in use@w{ }
@end display



@end multitable



@display
@w{ }
@end display




@multitable {xxxxxxxxxxxxxxxxxxxxxx} {xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx} 
@item

`Title'

@tab

Sets identity as default for a subsystem

@item

`URL'

@tab

@code{/identity/subsystem/'name'}

@item

`Method'

@tab

`PUT'

@item

`URL Params'

@tab

none

@item

`Data Params'

@tab

@{"subsystem":'`subsystem''@}

@item

`Success Response'

@tab

Response Code: @code{c) 204 No Content}

@item

`Error Response'

@tab

@{"error":"`error_desc'"@} @w{^2; 4; 6; 7; 9}

@end multitable


@node DELETE Request,OPTIONS Request,PUT Request,Requests
@anchor{developers/rest-api/identity delete-request}@anchor{362}
@subsubsection DELETE Request



@multitable {xxxxxxxxxxxxxxxxxxxxxx} {xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx} 
@item

`Title'

@tab

Deletes specific identity

@item

`URL'

@tab

@code{/identity/pubkey/'public_key'} or @code{/identity/name/'name'}

@item

`Method'

@tab

`DELETE'

@item

`URL Params'

@tab

none

@item

`Data Params'

@tab

none

@item

`Success Response'

@tab

Response Code: @code{c) 204 No Content}

@item

`Error Response'

@tab

@{"error":"`error_desc'"@} @w{^2; 3 or 4}

@end multitable


@node OPTIONS Request,,DELETE Request,Requests
@anchor{developers/rest-api/identity options-request}@anchor{363}
@subsubsection OPTIONS Request



@multitable {xxxxxxxxxxxxxxxxxxxxxx} {xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx} 
@item

`Title'

@tab

Gets request options

@item

`URL'

@tab

@code{/identity}

@item

`Method'

@tab

`OPTIONS'

@item

`URL Params'

@tab

none

@item

`Data Params'

@tab

none

@item

`Success Response'

@tab

@item

`Error Response'

@tab

none

@end multitable


@node Namestore,GNS<3>,Identity,REST API
@anchor{developers/rest-api/namestore doc}@anchor{364}@anchor{developers/rest-api/namestore namestore}@anchor{365}@anchor{developers/rest-api/namestore namestore-rest-api}@anchor{c9}
@subsection Namestore


The NAMESTORE REST API.

@menu
* Data model:: 
* Requests: Requests<2>. 

@end menu

@node Data model,Requests<2>,,Namestore
@anchor{developers/rest-api/namestore data-model}@anchor{366}
@subsubsection Data model


The following data model definitions are expected to be used in a JSON representation
in the respective requests.
Responses are provided in the JSON format accordingly.
@anchor{developers/rest-api/namestore recordset}@anchor{367}
@example
interface RecordSet @{

  // The name of the record set
  record_name: string;

  // The record set data array.
  data: RecordData[];

@}
@end example
@anchor{developers/rest-api/namestore recorddata}@anchor{368}
@example
interface RecordData @{

  // The string representation of the record data value, e.g. "1.2.3.4" for an A record
  value: string;

  // The string representation of the record type, e.g. "A" for an IPv4 address
  record_type: string;

  // The relative expiration time, in microseconds. Set if is_relative_expiration: true
  relative_expiration: integer;

  // The absolute expiration time, in microseconds. Set if is_relative_expiration: false
  absolute_expiration: integer;

  // Whether or not this is a private record
  is_private: boolean;

  // Whether or not the expiration time is relative (else absolute)
  is_relative_expiration: boolean;

  // Whether or not this is a supplemental record
  is_supplemental: boolean;

  // Whether or not this is a shadow record
  is_shadow: boolean

  // Whether or not this is a maintenance record
  is_maintenance: boolean
@}
@end example
@anchor{developers/rest-api/namestore namestoreerror}@anchor{369}
@example
interface GnunetError @{

  // The error description
  error: string;

  // The error code
  error_code: integer

@}
@end example

`NOTE:' All endpoints can return with an HTTP status (5xx).
In this case the response body contains a @ref{369,,NamestoreError}.

@node Requests<2>,,Data model,Namestore
@anchor{developers/rest-api/namestore requests}@anchor{36a}
@subsubsection Requests


@anchor{developers/rest-api/namestore get--namestore-$ZNAME}@anchor{36b}
@deffn {HTTP Get} GET /namestore/$ZNAME

This endpoint returns all namestore entries for one zone identified by @code{$ZNAME}.

`Request'

@*Query Parameters: 

@itemize *

@item 
@code{record_type} -- `Optional'. The string representation of a DNS or GNS record type. If given, only record sets including record with this type will be returned.

@item 
@code{omit_private} -- `Optional'. If set, private records are omitted from the results.

@item 
@code{include_maintenance} -- `Optional'. If set, maintenance records are included in the results.
@end itemize


`Response'


@table @asis

@item 200 Ok@footnote{http://www.w3.org/Protocols/rfc2616/rfc2616-sec10.html#sec10.2.1}:

The body is a @ref{367,,RecordSet} array.

@item 404 Not found@footnote{http://www.w3.org/Protocols/rfc2616/rfc2616-sec10.html#sec10.4.5}:

The zone was not found.
@end table
@end deffn

@anchor{developers/rest-api/namestore get--namestore-$ZNAME-$LABEL}@anchor{36c}
@deffn {HTTP Get} GET /namestore/$ZNAME/$LABEL

This endpoint returns the record set under label @code{$LABEL} of the zone
identified by @code{$ZNAME}.

`Request'

@*Query Parameters: 

@itemize *

@item 
@code{record_type} -- `Optional'. The string representation of a DNS or GNS record type. If given, only record sets including record with this type will be returned.

@item 
@code{omit_private} -- `Optional'. If set, private records are omitted from the results.

@item 
@code{include_maintenance} -- `Optional'. If set, maintenance records are included in the results.
@end itemize


`Response'


@table @asis

@item 200 Ok@footnote{http://www.w3.org/Protocols/rfc2616/rfc2616-sec10.html#sec10.2.1}:

The body is a @ref{367,,RecordSet}.

@item 404 Not found@footnote{http://www.w3.org/Protocols/rfc2616/rfc2616-sec10.html#sec10.4.5}:

The zone or label was not found.
@end table
@end deffn

@anchor{developers/rest-api/namestore post--namestore-$ZNAME}@anchor{36d}
@deffn {HTTP Post} POST /namestore/$ZNAME

Create or append a @ref{367,,RecordSet} to a the zone identified by @code{$ZNAME}.


@table @asis

@item `Request'

The request body is a single @ref{367,,RecordSet}.
@end table

`Response'


@table @asis

@item 204 No Content@footnote{http://www.w3.org/Protocols/rfc2616/rfc2616-sec10.html#sec10.2.5}:

The zone was successfully added.

@item 404 Not found@footnote{http://www.w3.org/Protocols/rfc2616/rfc2616-sec10.html#sec10.4.5}:

The zone was not found.
@end table
@end deffn

@anchor{developers/rest-api/namestore put--namestore-$ZNAME}@anchor{36e}
@deffn {HTTP Put} PUT /namestore/$ZNAME

Create or replace a @ref{367,,RecordSet} in the zone identified by @code{$ZNAME}.


@table @asis

@item `Request'

The request body is a single @ref{367,,RecordSet}.
@end table

`Response'


@table @asis

@item 204 No Content@footnote{http://www.w3.org/Protocols/rfc2616/rfc2616-sec10.html#sec10.2.5}:

The zone was successfully updated.

@item 404 Not found@footnote{http://www.w3.org/Protocols/rfc2616/rfc2616-sec10.html#sec10.4.5}:

The zone was not found.
@end table
@end deffn

@anchor{developers/rest-api/namestore delete--namestore-$ZNAME-$LABEL}@anchor{36f}
@deffn {HTTP Delete} DELETE /namestore/$ZNAME/$LABEL

Delete all records under name @code{$LABEL} in the zone identified by @code{$ZNAME}.

`Response'


@table @asis

@item 204 No Content@footnote{http://www.w3.org/Protocols/rfc2616/rfc2616-sec10.html#sec10.2.5}:

The records were successfully deleted.

@item 404 Not found@footnote{http://www.w3.org/Protocols/rfc2616/rfc2616-sec10.html#sec10.4.5}:

The zone or label was not found.
@end table
@end deffn

@anchor{developers/rest-api/namestore post--namestore-import-$ZNAME}@anchor{370}
@deffn {HTTP Post} POST /namestore/import/$ZNAME

Bulk import of record sets into the zone identified by @code{$ZNAME}.
This API adds the provided array of @ref{367,,RecordSet} to the zone.
This operation does `NOT' replace existing records under the label(s).
If you want a clean import, reset your database before using this API.
This API is making sure that the records are added within one database
transaction, calling @code{GNUNET_NAMESTORE_transaction_begin},
@code{GNUNET_NAMESTORE_records_store2} (successively, if necessary) and finally
@code{GNUNET_NAMESTORE_transaction_commit}.


@table @asis

@item `Request'

The request body is a @ref{367,,RecordSet} array.
@end table

`Response'


@table @asis

@item 204 No Content@footnote{http://www.w3.org/Protocols/rfc2616/rfc2616-sec10.html#sec10.2.5}:

The record sets were successfully added.

@item 404 Not found@footnote{http://www.w3.org/Protocols/rfc2616/rfc2616-sec10.html#sec10.4.5}:

The zone $ZNAME was not found.
@end table
@end deffn

@node GNS<3>,Peerinfo,Namestore,REST API
@anchor{developers/rest-api/gns doc}@anchor{371}@anchor{developers/rest-api/gns gns}@anchor{372}
@subsection GNS


@menu
* Error Response: Error Response<2>. 
* GET Request:: 
* OPTIONS Request: OPTIONS Request<2>. 

@end menu

@node Error Response<2>,GET Request,,GNS<3>
@anchor{developers/rest-api/gns error-response}@anchor{373}
@subsubsection Error Response


An error response is sent in the JSON format: @code{@{"error":"*error_description*"@}}

Following numbers are added for references inside the documentation only.

Error descriptions are:

@example
Nr. Error Description           - Explanation
1)  Unknown Error               - Error is not specified
2)  Record not found            - GNS record was not found, this is combined with the HTTP Error Code 404 Not Found
@end example

Error @code{1)} is always possible and is not listed in following requests.

@menu
* Requests: Requests<3>. 

@end menu

@node Requests<3>,,,Error Response<2>
@anchor{developers/rest-api/gns requests}@anchor{374}
@subsubsection Requests


@node GET Request,OPTIONS Request<2>,Error Response<2>,GNS<3>
@anchor{developers/rest-api/gns get-request}@anchor{375}
@subsubsection GET Request


For this request @code{record_type} is optional. If missing or called with an invalid record_type, it will automatically be set to ANY.


@multitable {xxxxxxxxxxxxxxxxxxxxxx} {xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx} 
@item

`Title'

@tab

Return all records for given name

@item

`URL'

@tab

@code{/gns/'lookup'?record_type='type'}

@item

`Method'

@tab

`GET'

@item

`Data Params'

@tab

none

@item

`Success Response'

@tab

[@{"value": "`value'", "record_type": "`type'", "expiration_time": "`time'", "flag": `flag'@},...]

@item

`Error Response'

@tab

@{"error":"`error_desc'"@} @w{^2}

@end multitable


@node OPTIONS Request<2>,,GET Request,GNS<3>
@anchor{developers/rest-api/gns options-request}@anchor{376}
@subsubsection OPTIONS Request



@multitable {xxxxxxxxxxxxxxxxxxxxxx} {xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx} 
@item

`Title'

@tab

Get request options

@item

`URL'

@tab

@code{/gns}

@item

`Method'

@tab

`OPTIONS'

@item

`URL Params'

@tab

none

@item

`Data Params'

@tab

none

@item

`Success Response'

@tab

@item

`Error Response'

@tab

none

@end multitable


@node Peerinfo,Config,GNS<3>,REST API
@anchor{developers/rest-api/peerinfo doc}@anchor{377}@anchor{developers/rest-api/peerinfo peerinfo}@anchor{378}
@subsection Peerinfo


@menu
* Definition: Definition<2>. 
* Requests: Requests<4>. 

@end menu

@node Definition<2>,Requests<4>,,Peerinfo
@anchor{developers/rest-api/peerinfo definition}@anchor{379}
@subsubsection Definition


Variables in single quotes @code{'...'} can or must be changed according to your specific case.

@code{friend} is to enable the optional friend information. It is either @code{yes} or can be left away.

@menu
* Peer:: 
* Peerinfo Response:: 
* Error Response: Error Response<3>. 

@end menu

@node Peer,Peerinfo Response,,Definition<2>
@anchor{developers/rest-api/peerinfo peer}@anchor{37a}
@subsubsection Peer


A peer consists of an @code{identifier} and one or more @code{addresses} with @code{expiration dates}.

@node Peerinfo Response,Error Response<3>,Peer,Definition<2>
@anchor{developers/rest-api/peerinfo peerinfo-response}@anchor{37b}
@subsubsection Peerinfo Response


The response of the peerinfo API is a JSON Array:

@example
[
   @{
     "peer":'identifier',
     "array": [
                @{
                  "address":'peer_address',
                  "expires":'address_expiration'
                @},
                ...
              ]
   @},
   ...
 ]
@end example

@code{ìdentifier} is a 52-character, alphanumeric identifier of the peer.

@code{peer_address} is one URI as string.

@code{address_expiration} is the date, when the address expires, e.g. "Wed Aug 1 10:00:00 2018".

@node Error Response<3>,,Peerinfo Response,Definition<2>
@anchor{developers/rest-api/peerinfo error-response}@anchor{37c}
@subsubsection Error Response


An error response is sent in the JSON format: @code{@{"error":"*error_description*"@}}

Following numbers are added for references inside the documentation only.

Error descriptions are:

@example
Nr. Error Description           - Explanation
1)  Unknown Error               - Error is not specified
2)  No peers found              - Peers were not found, this is combined with the HTTP Error Code 404 Not Found
@end example

Error @code{1)} is always possible and is not listed in following requests.

ATTENTION: Any error message from the Peerinfo API (not REST API) can occur and can be returned in the error response. These responses are not listed here.

@node Requests<4>,,Definition<2>,Peerinfo
@anchor{developers/rest-api/peerinfo requests}@anchor{37d}
@subsubsection Requests


@menu
* GET Request: GET Request<2>. 
* OPTIONS Request: OPTIONS Request<3>. 

@end menu

@node GET Request<2>,OPTIONS Request<3>,,Requests<4>
@anchor{developers/rest-api/peerinfo get-request}@anchor{37e}
@subsubsection GET Request



@multitable {xxxxxxxxxxxxxxxxxxxxxx} {xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx} 
@item

`Title'

@tab

Returns all peers and resolves their addresses

@item

`URL'

@tab

@code{/peerinfo}

@item

`Method'

@tab

`GET'

@item

`URL Params'

@tab

@code{?friend='friend'} optional

@item

`Data Params'

@tab

none

@item

`Success Response'

@tab

Peerinfo Response `or' Response Code: @code{500 Internal Server Error}

@item

`Error Response'

@tab

@{"error":"`error_desc'"@} @w{^2}

@end multitable


@node OPTIONS Request<3>,,GET Request<2>,Requests<4>
@anchor{developers/rest-api/peerinfo options-request}@anchor{37f}
@subsubsection OPTIONS Request



@multitable {xxxxxxxxxxxxxxxxxxxxxx} {xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx} 
@item

`Title'

@tab

Gets request options

@item

`URL'

@tab

@code{/peerinfo}

@item

`Method'

@tab

`OPTIONS'

@item

`URL Params'

@tab

none

@item

`Data Params'

@tab

none

@item

`Success Response'

@tab

@item

`Error Response'

@tab

none

@end multitable


@node Config,,Peerinfo,REST API
@anchor{developers/rest-api/config doc}@anchor{380}@anchor{developers/rest-api/config config}@anchor{381}
@subsection Config


@menu
* Definition: Definition<3>. 
* Requests: Requests<5>. 

@end menu

@node Definition<3>,Requests<5>,,Config
@anchor{developers/rest-api/config definition}@anchor{382}
@subsubsection Definition


Variables in single quotes @code{'...'} can or must be changed according to your specific case.

@code{config} refers to the configuration file.

@code{'section'} is a section of settings in the configuration file.

@code{'option'} is an setting in the configuration file with a modifiable @code{'value'}.

@menu
* Configuration: Configuration<3>. 
* Error Response: Error Response<4>. 
* Response Code: Response Code<2>. 

@end menu

@node Configuration<3>,Error Response<4>,,Definition<3>
@anchor{developers/rest-api/config configuration}@anchor{383}
@subsubsection Configuration


The configuration file is divided in @code{sections}, each consisting of various @code{options} with their corresponding @code{values}.

@node Error Response<4>,Response Code<2>,Configuration<3>,Definition<3>
@anchor{developers/rest-api/config error-response}@anchor{384}
@subsubsection Error Response


An error response is sent in the JSON format: @code{@{"error":"*error_description*"@}}

Following numbers are added for references inside the documentation only.

Error descriptions are:

@example
Nr. Error Description                           - Explanation
1)  Unknown Error                               - Error is not specified
2)  Unable to parse JSON Object from "*URI*"    - Corrupt JSON data given
@end example

Error @code{1)} is always possible and is not listed in following requests.

ATTENTION: Any error message from the Configuration API (not REST API) can occur and can be returned in the error response. These responses are not listed here.

@node Response Code<2>,,Error Response<4>,Definition<3>
@anchor{developers/rest-api/config response-code}@anchor{385}
@subsubsection Response Code


A response of a message has a HTTP response code. Usually, this code is 200 OK for a successful response. The code changes in some cases:

@example
a) 200 OK           - Normal response (but may contain an error message)
b) 201 Created      - Success after POST request
c) 400 Bad Request  - Invalid request
@end example

@node Requests<5>,,Definition<3>,Config
@anchor{developers/rest-api/config requests}@anchor{386}
@subsubsection Requests


@menu
* GET Requests: GET Requests<2>. 
* POST Request: POST Request<2>. 
* OPTIONS Request: OPTIONS Request<4>. 

@end menu

@node GET Requests<2>,POST Request<2>,,Requests<5>
@anchor{developers/rest-api/config get-requests}@anchor{387}
@subsubsection GET Requests



@multitable {xxxxxxxxxxxxxxxxxxxxxx} {xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx} 
@item

`Title'

@tab

Returns the config or the specified section of the config

@item

`URL'

@tab

@code{/config}

@item

`Method'

@tab

`GET'

@item

`URL Params'

@tab

none

@item

`Data Params'

@tab

none

@item

`Success Response'

@tab

@{"section":@{"option":"`value'",...@},...@}

@item

`Error Response'

@tab

@{"error":"`error_desc'"@}

@end multitable



@display
@w{ }
@end display




@multitable {xxxxxxxxxxxxxxxxxxxxxx} {xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx} 
@item

`Title'

@tab

Returns only a specific section

@item

`URL'

@tab

@code{/config/'section'}

@item

`Method'

@tab

`GET'

@item

`URL Params'

@tab

none

@item

`Data Params'

@tab

none

@item

`Success Response'

@tab

@{"option":"`value'",...@}

@item

`Error Response'

@tab

@{"error":"`error_desc'"@}

@end multitable


@node POST Request<2>,OPTIONS Request<4>,GET Requests<2>,Requests<5>
@anchor{developers/rest-api/config post-request}@anchor{388}
@subsubsection POST Request



@multitable {xxxxxxxxxxxxxxxxxxxxxx} {xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx} 
@item

`Title'

@tab

Creates/modifies options in the config

@item

`URL'

@tab

@code{/config}

@item

`Method'

@tab

`POST'

@item

`URL Params'

@tab

none

@item

`Data Params'

@tab

@{"'section'": @{"'option'": "'value'",...@},...@}

@item

`Success Response'

@tab

Response Code: @code{b) 200 OK}

@item

`Error Response'

@tab


@display
@{"error":"`error_desc'"@}@w{ }
@end display



@end multitable


@node OPTIONS Request<4>,,POST Request<2>,Requests<5>
@anchor{developers/rest-api/config options-request}@anchor{389}
@subsubsection OPTIONS Request



@multitable {xxxxxxxxxxxxxxxxxxxxxx} {xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx} 
@item

`Title'

@tab

Gets request options

@item

`URL'

@tab

@code{/config}

@item

`Method'

@tab

`OPTIONS'

@item

`URL Params'

@tab

none

@item

`Data Params'

@tab

none

@item

`Success Response'

@tab

@item

`Error Response'

@tab

none

@end multitable


@node C Tutorial,Specifications,REST API,Developer Handbook
@anchor{developers/tutorial doc}@anchor{38a}@anchor{developers/tutorial c-tutorial}@anchor{38b}
@section C Tutorial


This tutorials explains how to install GNUnet on a GNU/Linux system and
gives an introduction on how GNUnet can be used to develop a
Peer-to-Peer application. Detailed installation instructions for various
operating systems and a detailed list of all dependencies can be found
on our website at @indicateurl{https://docs.gnunet.org/#Installation} and in our
Reference Documentation (GNUnet Handbook).

Please read this tutorial carefully since every single step is
important, and do not hesitate to contact the GNUnet team if you have
any questions or problems! Visit this link in your webbrowser to learn
how to contact the GNUnet team: @indicateurl{https://gnunet.org/en/contact.html}

@menu
* Introduction to GNUnet Architecture:: 
* First Steps with GNUnet:: 
* Developing Applications:: 
* GNU Free Documentation License:: 

@end menu

@node Introduction to GNUnet Architecture,First Steps with GNUnet,,C Tutorial
@anchor{developers/tutorial id1}@anchor{38c}@anchor{developers/tutorial introduction-to-gnunet-architecture}@anchor{38d}
@subsection Introduction to GNUnet Architecture


GNUnet is organized in layers and services. Each service is composed of
a main service implementation and a client library for other programs to
use the service's functionality, described by an API. Some services
provide an additional command line tool to enable the user to interact
with the service.

Very often it is other GNUnet services that will use these APIs to build
the higher layers of GNUnet on top of the lower ones. Each layer expands
or extends the functionality of the service below (for instance, to
build a mesh on top of a DHT).

The main service implementation runs as a standalone process in the
Operating System and the client code runs as part of the client program,
so crashes of a client do not affect the service process or other
clients. The service and the clients communicate via a message protocol
to be defined and implemented by the programmer.

@node First Steps with GNUnet,Developing Applications,Introduction to GNUnet Architecture,C Tutorial
@anchor{developers/tutorial first-steps-with-gnunet}@anchor{38e}@anchor{developers/tutorial id2}@anchor{38f}
@subsection First Steps with GNUnet


@menu
* Configure your peer:: 
* Start a peer:: 
* Monitor a peer:: 
* Starting Two Peers by Hand:: 
* Starting Peers Using the Testbed Service:: 

@end menu

@node Configure your peer,Start a peer,,First Steps with GNUnet
@anchor{developers/tutorial configure-your-peer}@anchor{390}@anchor{developers/tutorial id3}@anchor{391}
@subsubsection Configure your peer


First of all we need to configure your peer. Each peer is started with a
configuration containing settings for GNUnet itself and its services.
This configuration is based on the default configuration shipped with
GNUnet and can be modified. The default configuration is located in the
@code{$PREFIX/share/gnunet/config.d} directory. When starting a peer, you
can specify a customized configuration using the the @code{-c} command line
switch when starting the ARM service and all other services. When using
a modified configuration the default values are loaded and only values
specified in the configuration file will replace the default values.

Since we want to start additional peers later, we need some
modifications from the default configuration. We need to create a
separate service home and a file containing our modifications for this
peer:

@example
$ mkdir ~/gnunet1/
$ touch peer1.conf
@end example

Now add the following lines to @code{peer1.conf} to use this directory. For
simplified usage we want to prevent the peer to connect to the GNUnet
network since this could lead to confusing output. This modifications
will replace the default settings:

@example
[PATHS]
# Use this directory to store GNUnet data
GNUNET_HOME = ~/gnunet1/
[hostlist]
# prevent bootstrapping
SERVERS =
@end example

@node Start a peer,Monitor a peer,Configure your peer,First Steps with GNUnet
@anchor{developers/tutorial id4}@anchor{392}@anchor{developers/tutorial start-a-peer}@anchor{393}
@subsubsection Start a peer


Each GNUnet instance (called peer) has an identity (peer ID) based on a
cryptographic public private key pair. The peer ID is the printable hash
of the public key.

GNUnet services are controlled by a master service, the so called
Automatic Restart Manager (ARM). ARM starts, stops and even restarts
services automatically or on demand when a client connects. You interact
with the ARM service using the @code{gnunet-arm} tool. GNUnet can then be
started with @code{gnunet-arm -s} and stopped with @code{gnunet-arm -e}. An
additional service not automatically started can be started using
@code{gnunet-arm -i <service name>} and stopped using
@code{gnunet-arm -k <servicename>}.

Once you have started your peer, you can use many other GNUnet commands
to interact with it. For example, you can run:

@example
$ gnunet-core -i
@end example

to obtain the public key of your peer.

You should see an output containing the peer ID similar to:

@example
I am peer `0PA02UVRKQTS2C .. JL5Q78F6H0B1ACPV1CJI59MEQUMQCC5G'.
@end example

@node Monitor a peer,Starting Two Peers by Hand,Start a peer,First Steps with GNUnet
@anchor{developers/tutorial id5}@anchor{394}@anchor{developers/tutorial monitor-a-peer}@anchor{395}
@subsubsection Monitor a peer


In this section, we will monitor the behaviour of our peer's DHT service
with respect to a specific key. First we will start GNUnet and then
start the DHT service and use the DHT monitor tool to monitor the PUT
and GET commands we issue ussing the @code{gnunet-dht-put} and
@code{gnunet-dht-get} commands. Using the "monitor" line given below, you
can observe the behavior of your own peer's DHT with respect to the
specified KEY:

@example
# start gnunet with all default services:
$ gnunet-arm -c ~/peer1.conf -s
# start DHT service:
$ gnunet-arm -c ~/peer1.conf -i dht
$ gnunet-dht-monitor -c ~/peer1.conf -k KEY
@end example

Now open a separate terminal and:

@example
# put VALUE under KEY in the DHT:
$ gnunet-dht-put -c ~/peer1.conf -k KEY -d VALUE
# get key KEY from the DHT:
$ gnunet-dht-get -c ~/peer1.conf -k KEY
# print statistics about current GNUnet state:
$ gnunet-statistics -c ~/peer1.conf
# print statistics about DHT service:
$ gnunet-statistics -c ~/peer1.conf -s dht
@end example

@node Starting Two Peers by Hand,Starting Peers Using the Testbed Service,Monitor a peer,First Steps with GNUnet
@anchor{developers/tutorial id6}@anchor{396}@anchor{developers/tutorial starting-two-peers-by-hand}@anchor{397}
@subsubsection Starting Two Peers by Hand


This section describes how to start two peers on the same machine by
hand. The process is rather painful, but the description is somewhat
instructive. In practice, you might prefer the automated method (see
@ref{398,,Starting Peers Using the Testbed Service}).

@menu
* Setup a second peer:: 
* Start the second peer and connect the peers:: 
* How to connect manually:: 

@end menu

@node Setup a second peer,Start the second peer and connect the peers,,Starting Two Peers by Hand
@anchor{developers/tutorial id7}@anchor{399}@anchor{developers/tutorial setup-a-second-peer}@anchor{39a}
@subsubsection Setup a second peer


We will now start a second peer on your machine. For the second peer,
you will need to manually create a modified configuration file to avoid
conflicts with ports and directories. A peers configuration file is by
default located in @code{~/.config/gnunet.conf}. This file is typically
very short or even empty as only the differences to the defaults need to
be specified. The defaults are located in many files in the
@code{$PREFIX/share/gnunet/config.d} directory.

To configure the second peer, use the files
@code{$PREFIX/share/gnunet/config.d} as a template for your main
configuration file:

@example
$ cat $PREFIX/share/gnunet/config.d/*.conf > peer2.conf
@end example

Now you have to edit @code{peer2.conf} and change:


@itemize -

@item 
@code{GNUNET\_TEST\_HOME} under @code{PATHS}

@item 
Every (uncommented) value for "@code{PORT}" (add 10000) in any
section (the option may be commented out if @code{PORT} is prefixed by
"#", in this case, UNIX domain sockets are used and the PORT
option does not need to be touched)

@item 
Every value for "@code{UNIXPATH}" in any section (e.g. by adding a
"-p2" suffix)
@end itemize

to a fresh, unique value. Make sure that the PORT numbers stay below
65536. From now on, whenever you interact with the second peer, you need
to specify @code{-c peer2.conf} as an additional command line argument.

Now, generate the 2nd peer's private key:

@example
$ gnunet-core -i -c peer2.conf
@end example

This may take a while, generate entropy using your keyboard or mouse as
needed. Also, make sure the output is different from the gnunet-core
output for the first peer (otherwise you made an error in the
configuration).

@node Start the second peer and connect the peers,How to connect manually,Setup a second peer,Starting Two Peers by Hand
@anchor{developers/tutorial id8}@anchor{39b}@anchor{developers/tutorial start-the-second-peer-and-connect-the-peers}@anchor{39c}
@subsubsection Start the second peer and connect the peers


Then, you can start a second peer using:

@example
$ gnunet-arm -c peer2.conf -s
$ gnunet-arm -c peer2.conf -i dht
$ ~/gnunet/src/dht/gnunet-dht-put -c peer2.conf -k KEY -d VALUE
$ ~/gnunet/src/dht/gnunet-dht-get -c peer2.conf -k KEY
@end example

If you want the two peers to connect, you have multiple options:


@itemize -

@item 
UDP neighbour discovery (automatic)

@item 
Setup a bootstrap server

@item 
Connect manually
@end itemize

To setup peer 1 as bootstrapping server change the configuration of the
first one to be a hostlist server by adding the following lines to
@code{peer1.conf} to enable bootstrapping server:

@example
[hostlist]
OPTIONS = -p
@end example

Then change @code{peer2.conf} and replace the "@code{SERVERS}" line in the
"@code{[hostlist]}" section with "@code{http://localhost:8080/}".
Restart both peers using:

@example
# stop first peer
$ gnunet-arm -c peer1.conf -e
# start first peer
$ gnunet-arm -c peer1.conf -s
# start second peer
$ gnunet-arm -c peer2.conf -s
@end example

Note that if you start your peers without changing these settings, they
will use the "global" hostlist servers of the GNUnet P2P network and
likely connect to those peers. At that point, debugging might become
tricky as you're going to be connected to many more peers and would
likely observe traffic and behaviors that are not explicitly controlled
by you.

@node How to connect manually,,Start the second peer and connect the peers,Starting Two Peers by Hand
@anchor{developers/tutorial how-to-connect-manually}@anchor{39d}@anchor{developers/tutorial id9}@anchor{39e}
@subsubsection How to connect manually


If you want to use the @code{peerinfo} tool to connect your peers, you
should:


@itemize -

@item 
Set @code{IMMEDIATE_START = NO} in section @code{hostlist} (to not connect
to the global GNUnet)

@item 
Start both peers running @code{gnunet-arm -c peer1.conf -s} and
@code{gnunet-arm -c peer2.conf -s}

@item 
Get @code{HELLO} message of the first peer running
@code{gnunet-hello -c peer1.conf --export-hello}

@item 
Give the output to the second peer by running
@code{echo "gnunet://hello/..." | gnunet-hello -c peer2.conf --import-hello}
@end itemize

Check that they are connected using @code{gnunet-core -s -c peer1.conf}, which
should give you the other peer's peer identity:

@example
$ gnunet-core -s -c peer1.conf
Peer `9TVUCS8P5A7ILLBGO6 [...shortened...] 1KNBJ4NGCHP3JPVULDG'
@end example

@node Starting Peers Using the Testbed Service,,Starting Two Peers by Hand,First Steps with GNUnet
@anchor{developers/tutorial id10}@anchor{39f}@anchor{developers/tutorial starting-peers-using-the-testbed-service}@anchor{3a0}
@subsubsection Starting Peers Using the Testbed Service


GNUnet's testbed service is used for testing scenarios where a number of
peers are to be started. The testbed can manage peers on a single host
or on multiple hosts in a distributed fashion. On a single affordable
computer, it should be possible to run around tens of peers without
drastically increasing the load on the system.

The testbed service can be access through its API
@code{include/gnunet\_testbed\_service.h}. The API provides many routines
for managing a group of peers. It also provides a helper function
@code{GNUNET\_TESTBED\_test\_run()} to quickly setup a minimalistic testing
environment on a single host.

This function takes a configuration file which will be used as a
template configuration for the peers. The testbed takes care of
modifying relevant options in the peers' configuration such as
@code{SERVICEHOME}, @code{PORT}, @code{UNIXPATH} to unique values so that peers
run without running into conflicts. It also checks and assigns the ports
in configurations only if they are free.

Additionally, the testbed service also reads its options from the same
configuration file. Various available options and details about them can
be found in the testbed default configuration file
@code{src/testbed/testbed.conf}.

With the testbed API, a sample test case can be structured as follows:

@example
#include <unistd.h>
#include <gnunet/platform.h>
#include <gnunet/gnunet_util_lib.h>
#include <gnunet/gnunet_testbed_service.h>
#include <gnunet/gnunet_dht_service.h>

#define NUM_PEERS 20

static struct GNUNET_TESTBED_Operation *dht_op;

static struct GNUNET_DHT_Handle *dht_handle;


struct MyContext
@{
  int ht_len;
@} ctxt;


static int result;


static void
shutdown_task (void *cls)
@{
  if (NULL != dht_op)
  @{
    GNUNET_TESTBED_operation_done (dht_op);
    dht_op = NULL;
    dht_handle = NULL;
  @}
  result = GNUNET_OK;
@}


static void
service_connect_comp (void *cls,
                      struct GNUNET_TESTBED_Operation *op,
                      void *ca_result,
                      const char *emsg)
@{
  GNUNET_assert (op == dht_op);
  dht_handle = ca_result;
  // Do work here...
  GNUNET_SCHEDULER_shutdown ();
@}


static void *
dht_ca (void *cls, const struct GNUNET_CONFIGURATION_Handle *cfg)
@{
  struct MyContext *ctxt = cls;

  dht_handle = GNUNET_DHT_connect (cfg, ctxt->ht_len);
  return dht_handle;
@}


static void
dht_da (void *cls, void *op_result)
@{
  struct MyContext *ctxt = cls;

  GNUNET_DHT_disconnect ((struct GNUNET_DHT_Handle *) op_result);
  dht_handle = NULL;
@}


static void
test_master (void *cls,
             struct GNUNET_TESTBED_RunHandle *h,
             unsigned int num_peers,
             struct GNUNET_TESTBED_Peer **peers,
             unsigned int links_succeeded,
             unsigned int links_failed)
@{
  ctxt.ht_len = 10;
  dht_op = GNUNET_TESTBED_service_connect
      (NULL, peers[0], "dht",
       &service_connect_comp, NULL,
       &dht_ca, &dht_da, &ctxt);
  GNUNET_SCHEDULER_add_shutdown (&shutdown_task, NULL);
@}


int
main (int argc, char **argv)
@{
  int ret;

  result = GNUNET_SYSERR;
  ret = GNUNET_TESTBED_test_run
      ("awesome-test", "template.conf",
       NUM_PEERS, 0LL,
       NULL, NULL, &test_master, NULL);
  if ( (GNUNET_OK != ret) || (GNUNET_OK != result) )
    return 1;
  return 0;
@}
@end example

The source code for the above listing can be found at
@indicateurl{https://git.gnunet.org/gnunet.git/tree/doc/documentation/testbed_test.c}
or in the @code{doc/} folder of your repository check-out. After installing
GNUnet, the above source code can be compiled as:

@example
$ export CPPFLAGS="-I/path/to/gnunet/headers"
$ export LDFLAGS="-L/path/to/gnunet/libraries"
$ gcc $CPPFLAGS $LDFLAGS -o testbed-test testbed_test.c \
 -lgnunettestbed -lgnunetdht -lgnunetutil
# Generate (empty) configuration
$ touch template.conf
# run it (press CTRL-C to stop)
$ ./testbed-test
@end example

The @code{CPPFLAGS} and @code{LDFLAGS} are necessary if GNUnet is installed
into a different directory other than @code{/usr/local}.

All of testbed API's peer management functions treat management actions
as operations and return operation handles. It is expected that the
operations begin immediately, but they may get delayed (to balance out
load on the system). The program using the API then has to take care of
marking the operation as "done" so that its associated resources can be
freed immediately and other waiting operations can be executed.
Operations will be canceled if they are marked as "done" before their
completion.

An operation is treated as completed when it succeeds or fails.
Completion of an operation is either conveyed as events through
controller event callback or through respective operation completion
callbacks. In functions which support completion notification through
both controller event callback and operation completion callback, first
the controller event callback will be called. If the operation is not
marked as done in that callback or if the callback is given as NULL when
creating the operation, the operation completion callback will be
called. The API documentation shows which event are to be expected in
the controller event notifications. It also documents any exceptional
behaviour.

Once the peers are started, test cases often need to connect some of the
peers' services. Normally, opening a connect to a peer's service
requires the peer's configuration. While using testbed, the testbed
automatically generates per-peer configuration. Accessing those
configurations directly through file system is discouraged as their
locations are dynamically created and will be different among various
runs of testbed. To make access to these configurations easy, testbed
API provides the function @code{GNUNET\_TESTBED\_service\_connect()}. This
function fetches the configuration of a given peer and calls the Connect
Adapter. In the example code, it is the @code{dht\_ca}. A connect adapter
is expected to open the connection to the needed service by using the
provided configuration and return the created service connection handle.
Successful connection to the needed service is signaled through
@code{service\_connect\_comp\_cb}.

A dual to connect adapter is the Disconnect Adapter. This callback is
called after the connect adapter has been called when the operation from
@code{GNUNET\_TESTBED\_service\_connect()} is marked as "done". It has to
disconnect from the service with the provided service handle
(@code{op\_result}).

Exercise: Find out how many peers you can run on your system.

Exercise: Find out how to create a 2D torus topology by changing the
options in the configuration file. See section "The GNUnet Reference
Documentation" in The GNUnet Reference Documentation, then use the DHT
API to store and retrieve values in the network.

@node Developing Applications,GNU Free Documentation License,First Steps with GNUnet,C Tutorial
@anchor{developers/tutorial developing-applications}@anchor{3a1}@anchor{developers/tutorial id11}@anchor{3a2}
@subsection Developing Applications


@menu
* gnunet-ext:: 
* Adapting the Template:: 
* Writing a Client Application:: 
* Writing a Service:: 
* Interacting directly with other Peers using the CORE Service:: 
* Storing peer-specific data using the PEERSTORE service:: 
* Using the DHT:: 
* Debugging with gnunet-arm:: 

@end menu

@node gnunet-ext,Adapting the Template,,Developing Applications
@anchor{developers/tutorial gnunet-002dext}@anchor{3a3}@anchor{developers/tutorial gnunet-ext}@anchor{3a4}
@subsubsection gnunet-ext


To develop a new peer-to-peer application or to extend GNUnet we provide
a template build system for writing GNUnet extensions in C. It can be
obtained as follows:

@example
$ git clone https://git.gnunet.org/gnunet-ext.git
$ cd gnunet-ext/
$ ./bootstrap
$ ./configure --prefix=$PREFIX --with-gnunet=$PREFIX
$ make
$ make install
$ make check
@end example

The GNUnet ext template includes examples and a working buildsystem for
a new GNUnet service. A common GNUnet service consists of the following
parts which will be discussed in detail in the remainder of this
document. The functionality of a GNUnet service is implemented in:


@itemize -

@item 
the GNUnet service (gnunet-ext/src/ext/gnunet-service-ext.c)

@item 
the client API (gnunet-ext/src/ext/ext_api.c)

@item 
the client application using the service API
(gnunet-ext/src/ext/gnunet-ext.c)
@end itemize

The interfaces for these entities are defined in:


@itemize -

@item 
client API interface (gnunet-ext/src/ext/ext.h)

@item 
the service interface
(gnunet-ext/src/include/gnunet_service_SERVICE.h)

@item 
the P2P protocol (gnunet-ext/src/include/gnunet_protocols_ext.h)
@end itemize

In addition the ext systems provides:


@itemize -

@item 
a test testing the API (gnunet-ext/src/ext/test_ext_api.c)

@item 
a configuration template for the service
(gnunet-ext/src/ext/ext.conf.in)
@end itemize

@node Adapting the Template,Writing a Client Application,gnunet-ext,Developing Applications
@anchor{developers/tutorial adapting-the-template}@anchor{3a5}@anchor{developers/tutorial id12}@anchor{3a6}
@subsubsection Adapting the Template


The first step for writing any extension with a new service is to ensure
that the @code{ext.conf.in} file contains entries for the @code{UNIXPATH},
@code{PORT} and @code{BINARY} for the service in a section named after the
service.

If you want to adapt the template rename the @code{ext.conf.in} to match
your services name, you have to modify the @code{AC\_OUTPUT} section in
@code{configure.ac} in the @code{gnunet-ext} root.

@node Writing a Client Application,Writing a Service,Adapting the Template,Developing Applications
@anchor{developers/tutorial id13}@anchor{3a7}@anchor{developers/tutorial writing-a-client-application}@anchor{3a8}
@subsubsection Writing a Client Application


When writing any client application (for example, a command-line tool),
the basic structure is to start with the @code{GNUNET\_PROGRAM\_run}
function. This function will parse command-line options, setup the
scheduler and then invoke the @code{run} function (with the remaining
non-option arguments) and a handle to the parsed configuration (and the
configuration file name that was used, which is typically not needed):

@example
#include <gnunet/platform.h>
#include <gnunet/gnunet_util_lib.h>

static int ret;

static void
run (void *cls,
     char *const *args,
     const char *cfgfile,
     const struct GNUNET_CONFIGURATION_Handle *cfg)
@{
  // main code here
  ret = 0;
@}

int
main (int argc, char *const *argv)
@{
  struct GNUNET_GETOPT_CommandLineOption options[] = @{
    GNUNET_GETOPT_OPTION_END
  @};
  return (GNUNET_OK ==
          GNUNET_PROGRAM_run (argc,
                              argv,
                              "binary-name",
                              gettext_noop ("binary description text"),
                              options, &run, NULL)) ? ret : 1;
@}
@end example

@menu
* Handling command-line options:: 
* Writing a Client Library:: 
* Writing a user interface:: 

@end menu

@node Handling command-line options,Writing a Client Library,,Writing a Client Application
@anchor{developers/tutorial handling-command-002dline-options}@anchor{3a9}@anchor{developers/tutorial handling-command-line-options}@anchor{3aa}
@subsubsection Handling command-line options


Options can then be added easily by adding global variables and
expanding the @code{options} array. For example, the following would add a
string-option and a binary flag (defaulting to @code{NULL} and
@code{GNUNET\_NO} respectively):

@example
static char *string_option;
static int a_flag;

// ...
  struct GNUNET_GETOPT_CommandLineOption options[] = @{
    GNUNET_GETOPT_option_string ('s', "name", "SOMESTRING",
     gettext_noop ("text describing the string_option NAME"),
     &string_option@},
    GNUNET_GETOPT_option_flag ('f', "flag",
     gettext_noop ("text describing the flag option"),
     &a_flag),
    GNUNET_GETOPT_OPTION_END
  @};
  string_option = NULL;
  a_flag = GNUNET_SYSERR;
// ...
@end example

Issues such as displaying some helpful text describing options using the
@code{--help} argument and error handling are taken care of when using this
approach. Other @code{GNUNET\_GETOPT\_}-functions can be used to obtain
integer value options, increment counters, etc. You can even write
custom option parsers for special circumstances not covered by the
available handlers. To check if an argument was specified by the user
you initialize the variable with a specific value (e.g. NULL for a
string and GNUNET_SYSERR for a integer) and check after parsing
happened if the values were modified.

Inside the @code{run} method, the program would perform the
application-specific logic, which typically involves initializing and
using some client library to interact with the service. The client
library is supposed to implement the IPC whereas the service provides
more persistent P2P functions.

Exercise: Add a few command-line options and print them inside of
@code{run}. What happens if the user gives invalid arguments?

@node Writing a Client Library,Writing a user interface,Handling command-line options,Writing a Client Application
@anchor{developers/tutorial id14}@anchor{3ab}@anchor{developers/tutorial writing-a-client-library}@anchor{3ac}
@subsubsection Writing a Client Library


The first and most important step in writing a client library is to
decide on an API for the library. Typical API calls include connecting
to the service, performing application-specific requests and cleaning
up. Many examples for such service APIs can be found in the
@code{gnunet/src/include/gnunet\_*\_service.h} files.

Then, a client-service protocol needs to be designed. This typically
involves defining various message formats in a header that will be
included by both the service and the client library (but is otherwise
not shared and hence located within the service's directory and not
installed by @code{make install}). Each message must start with a
@code{struct GNUNET\_MessageHeader} and must be shorter than 64k. By
convention, all fields in IPC (and P2P) messages must be in big-endian
format (and thus should be read using @code{ntohl} and similar functions
and written using @code{htonl} and similar functions). Unique message types
must be defined for each message struct in the @code{gnunet\_protocols.h}
header (or an extension-specific include file).

@menu
* Connecting to the Service:: 
* Sending messages:: 
* Receiving Replies from the Service:: 

@end menu

@node Connecting to the Service,Sending messages,,Writing a Client Library
@anchor{developers/tutorial connecting-to-the-service}@anchor{3ad}@anchor{developers/tutorial id15}@anchor{3ae}
@subsubsection Connecting to the Service


Before a client library can implement the application-specific protocol
with the service, a connection must be created:

@example
struct GNUNET_MQ_MessageHandlers handlers[] = @{
    // ...
  GNUNET_MQ_handler_end ()
@};
struct GNUNET_MQ_Handle *mq;

mq = GNUNET_CLIENT_connect (cfg,
                            "service-name",
                            handlers,
                            &error_cb,
                            NULL);
@end example

As a result a @code{GNUNET\_MQ\_Handle} is returned which can to used
henceforth to transmit messages to the service. The complete MQ API can
be found in @code{gnunet\_mq\_lib.h}. The @code{handlers} array in the example
above is incomplete. Here is where you will define which messages you
expect to receive from the service, and which functions handle them. The
@code{error\_cb} is a function that is to be called whenever there are
errors communicating with the service.

@node Sending messages,Receiving Replies from the Service,Connecting to the Service,Writing a Client Library
@anchor{developers/tutorial id16}@anchor{3af}@anchor{developers/tutorial sending-messages}@anchor{3b0}
@subsubsection Sending messages


In GNUnet, messages are always sent beginning with a
@code{struct GNUNET\_MessageHeader} in big endian format. This header
defines the size and the type of the message, the payload follows after
this header.

@example
struct GNUNET_MessageHeader
@{
  uint16_t size GNUNET_PACKED;
  uint16_t type GNUNET_PACKED;
@};
@end example

Existing message types are defined in @code{gnunet\_protocols.h}. A common
way to create a message is with an envelope:

@example
struct GNUNET_MQ_Envelope *env;
struct GNUNET_MessageHeader *msg;

env = GNUNET_MQ_msg_extra (msg, payload_size, GNUNET_MY_MESSAGE_TYPE);
GNUNET_memcpy (&msg[1],
               &payload,
               payload_size);
// Send message via message queue 'mq'
GNUNET_mq_send (mq, env);
@end example

Exercise: Define a message struct that includes a 32-bit unsigned
integer in addition to the standard GNUnet MessageHeader. Add a C struct
and define a fresh protocol number for your message. Protocol numbers in
gnunet-ext are defined in
@code{gnunet-ext/src/include/gnunet_protocols_ext.h}

Exercise: Find out how you can determine the number of messages in a
message queue.

Exercise: Find out how you can determine when a message you have queued
was actually transmitted.

Exercise: Define a helper function to transmit a 32-bit unsigned integer
(as payload) to a service using some given client handle.

@node Receiving Replies from the Service,,Sending messages,Writing a Client Library
@anchor{developers/tutorial id17}@anchor{3b1}@anchor{developers/tutorial receiving-replies-from-the-service}@anchor{3b2}
@subsubsection Receiving Replies from the Service


Clients can receive messages from the service using the handlers
specified in the @code{handlers} array we specified when connecting to the
service. Entries in the the array are usually created using one of two
macros, depending on whether the message is fixed size or variable size.
Variable size messages are managed using two callbacks, one to check
that the message is well-formed, the other to actually process the
message. Fixed size messages are fully checked by the MQ-logic, and thus
only need to provide the handler to process the message. Note that the
prefixes @code{check\_} and @code{handle\_} are mandatory.

@example
static void
handle_fix (void *cls, const struct MyMessage *msg)
@{
  // process 'msg'
@}

static int
check_var (void *cls, const struct MyVarMessage *msg)
@{
  // check 'msg' is well-formed
  return GNUNET_OK;
@}

static void
handle_var (void *cls, const struct MyVarMessage *msg)
@{
  // process 'msg'
@}

struct GNUNET_MQ_MessageHandler handlers[] = @{
  GNUNET_MQ_hd_fixed_size (fix,
                          GNUNET_MESSAGE_TYPE_MY_FIX,
                          struct MyMessage,
                          NULL),
  GNUNET_MQ_hd_fixed_size (var,
                          GNUNET_MESSAGE_TYPE_MY_VAR,
                          struct MyVarMessage,
                          NULL),

  GNUNET_MQ_handler_end ()
@};
@end example

Exercise: Expand your helper function to receive a response message (for
example, containing just the @code{struct GNUnet MessageHeader} without any
payload). Upon receiving the service's response, you should call a
callback provided to your helper function's API.

Exercise: Figure out where you can pass values to the closures
(@code{cls}).

@node Writing a user interface,,Writing a Client Library,Writing a Client Application
@anchor{developers/tutorial id18}@anchor{3b3}@anchor{developers/tutorial writing-a-user-interface}@anchor{3b4}
@subsubsection Writing a user interface


Given a client library, all it takes to access a service now is to
combine calls to the client library with parsing command-line options.

Exercise: Call your client API from your @code{run()} method in your client
application to send a request to the service. For example, send a 32-bit
integer value based on a number given at the command-line to the
service.

@node Writing a Service,Interacting directly with other Peers using the CORE Service,Writing a Client Application,Developing Applications
@anchor{developers/tutorial id19}@anchor{3b5}@anchor{developers/tutorial writing-a-service}@anchor{3b6}
@subsubsection Writing a Service


Before you can test the client you've written so far, you'll need to
also implement the corresponding service.

@menu
* Code Placement:: 
* Starting a Service:: 

@end menu

@node Code Placement,Starting a Service,,Writing a Service
@anchor{developers/tutorial code-placement}@anchor{3b7}@anchor{developers/tutorial id20}@anchor{3b8}
@subsubsection Code Placement


New services are placed in their own subdirectory under @code{gnunet/src}.
This subdirectory should contain the API implementation file
@code{SERVICE\_api.c}, the description of the client-service protocol
@code{SERVICE.h} and P2P protocol @code{SERVICE\_protocol.h}, the
implementation of the service itself @code{gnunet-service-SERVICE.h} and
several files for tests, including test code and configuration files.

@node Starting a Service,,Code Placement,Writing a Service
@anchor{developers/tutorial id21}@anchor{3b9}@anchor{developers/tutorial starting-a-service}@anchor{3ba}
@subsubsection Starting a Service


The key API definition for creating a service is the
@code{GNUNET\_SERVICE\_MAIN} macro:

@example
GNUNET_SERVICE_MAIN
("service-name",
 GNUNET_SERVICE_OPTION_NONE,
 &run,
 &client_connect_cb,
 &client_disconnect_cb,
 NULL,
 GNUNET_MQ_hd_fixed_size (...),
 GNUNET_MQ_hd_var_size (...),
 GNUNET_MQ_handler_end ());
@end example

In addition to the service name and flags, the macro takes three
functions, typically called @code{run}, @code{client\_connect\_cb} and
@code{client\_disconnect\_cb} as well as an array of message handlers that
will be called for incoming messages from clients.

A minimal version of the three central service functions would look like
this:

@example
static void
run (void *cls,
     const struct GNUNET_CONFIGURATION_Handle *c,
     struct GNUNET_SERVICE_Handle *service)
@{
@}

static void *
client_connect_cb (void *cls,
                   struct GNUNET_SERVICE_Client *c,
                   struct GNUNET_MQ_Handle *mq)
@{
  return c;
@}

static void
client_disconnect_cb (void *cls,
                      struct GNUNET_SERVICE_Client *c,
                      void *internal_cls)
@{
  GNUNET_assert (c == internal_cls);
@}
@end example

Exercise: Write a stub service that processes no messages at all in your
code. Create a default configuration for it, integrate it with the build
system and start the service from @code{gnunet-service-arm} using
@code{gnunet-arm -i NAME}.

Exercise: Figure out how to set the closure (@code{cls}) for handlers of a
service.

Exercise: Figure out how to send messages from the service back to the
client.

Each handler function in the service `must' eventually (possibly in
some asynchronous continuation) call
@code{GNUNET\_SERVICE\_client\_continue()}. Only after this call additional
messages from the same client may be processed. This way, the service
can throttle processing messages from the same client.

Exercise: Change the service to "handle" the message from your client
(for now, by printing a message). What happens if you forget to call
@code{GNUNET\_SERVICE\_client\_continue()}?

@node Interacting directly with other Peers using the CORE Service,Storing peer-specific data using the PEERSTORE service,Writing a Service,Developing Applications
@anchor{developers/tutorial id22}@anchor{3bb}@anchor{developers/tutorial interacting-directly-with-other-peers-using-the-core-service}@anchor{3bc}
@subsubsection Interacting directly with other Peers using the CORE Service


FIXME: This section still needs to be updated to the latest API!

One of the most important services in GNUnet is the @code{CORE} service
managing connections between peers and handling encryption between
peers.

One of the first things any service that extends the P2P protocol
typically does is connect to the @code{CORE} service using:

@example
#include <gnunet/gnunet_core_service.h>

struct GNUNET_CORE_Handle *
GNUNET_CORE_connect (const struct GNUNET_CONFIGURATION_Handle *cfg,
                     void *cls,
                     GNUNET_CORE_StartupCallback init,
                     GNUNET_CORE_ConnectEventHandler connects,
                     GNUNET_CORE_DisconnectEventHandler disconnects,
                     const struct GNUNET_MQ_MessageHandler *handlers);
@end example

@menu
* New P2P connections:: 
* Receiving P2P Messages:: 
* Sending P2P Messages:: 
* End of P2P connections:: 

@end menu

@node New P2P connections,Receiving P2P Messages,,Interacting directly with other Peers using the CORE Service
@anchor{developers/tutorial id23}@anchor{3bd}@anchor{developers/tutorial new-p2p-connections}@anchor{3be}
@subsubsection New P2P connections


Before any traffic with a different peer can be exchanged, the peer must
be known to the service. This is notified by the @code{CORE} @code{connects}
callback, which communicates the identity of the new peer to the
service:

@example
void *
connects (void *cls,
          const struct GNUNET_PeerIdentity *peer,
          struct GNUNET_MQ_Handle *mq)
@{
  return mq;
@}
@end example

Note that whatever you return from @code{connects} is given as the @code{cls}
argument to the message handlers for messages from the respective peer.

Exercise: Create a service that connects to the @code{CORE}. Then start
(and connect) two peers and print a message once your connect callback
is invoked.

@node Receiving P2P Messages,Sending P2P Messages,New P2P connections,Interacting directly with other Peers using the CORE Service
@anchor{developers/tutorial id24}@anchor{3bf}@anchor{developers/tutorial receiving-p2p-messages}@anchor{3c0}
@subsubsection Receiving P2P Messages


To receive messages from @code{CORE}, you pass the desired @code{handlers} to
the @code{GNUNET\_CORE\_connect()} function, just as we showed for
services.

It is your responsibility to process messages fast enough or to
implement flow control. If an application does not process CORE messages
fast enough, CORE will randomly drop messages to not keep a very long
queue in memory.

Exercise: Start one peer with a new service that has a message handler
and start a second peer that only has your "old" service without message
handlers. Which "connect" handlers are invoked when the two peers are
connected? Why?

@node Sending P2P Messages,End of P2P connections,Receiving P2P Messages,Interacting directly with other Peers using the CORE Service
@anchor{developers/tutorial id25}@anchor{3c1}@anchor{developers/tutorial sending-p2p-messages}@anchor{3c2}
@subsubsection Sending P2P Messages


You can transmit messages to other peers using the @code{mq} you were given
during the @code{connect} callback. Note that the @code{mq} automatically is
released upon @code{disconnect} and that you must not use it afterwards.

It is your responsibility to not over-fill the message queue, GNUnet
will send the messages roughly in the order given as soon as possible.

Exercise: Write a service that upon connect sends messages as fast as
possible to the other peer (the other peer should run a service that
"processes" those messages). How fast is the transmission? Count using
the STATISTICS service on both ends. Are messages lost? How can you
transmit messages faster? What happens if you stop the peer that is
receiving your messages?

@node End of P2P connections,,Sending P2P Messages,Interacting directly with other Peers using the CORE Service
@anchor{developers/tutorial end-of-p2p-connections}@anchor{3c3}@anchor{developers/tutorial id26}@anchor{3c4}
@subsubsection End of P2P connections


If a message handler returns @code{GNUNET\_SYSERR}, the remote peer shuts
down or there is an unrecoverable network disconnection, CORE notifies
the service that the peer disconnected. After this notification no more
messages will be received from the peer and the service is no longer
allowed to send messages to the peer. The disconnect callback looks like
the following:

@example
void
disconnects (void *cls,
             const struct GNUNET_PeerIdentity * peer)
@{
    /* Remove peer's identity from known peers */
    /* Make sure no messages are sent to peer from now on */
@}
@end example

Exercise: Fix your service to handle peer disconnects.

@node Storing peer-specific data using the PEERSTORE service,Using the DHT,Interacting directly with other Peers using the CORE Service,Developing Applications
@anchor{developers/tutorial storing-peer-002dspecific-data-using-the-peerstore-service}@anchor{3c5}@anchor{developers/tutorial storing-peer-specific-data-using-the-peerstore-service}@anchor{3c6}
@subsubsection Storing peer-specific data using the PEERSTORE service


GNUnet's PEERSTORE service offers a persistorage for arbitrary
peer-specific data. Other GNUnet services can use the PEERSTORE to
store, retrieve and monitor data records. Each data record stored with
PEERSTORE contains the following fields:


@itemize -

@item 
subsystem: Name of the subsystem responsible for the record.

@item 
peerid: Identity of the peer this record is related to.

@item 
key: a key string identifying the record.

@item 
value: binary record value.

@item 
expiry: record expiry date.
@end itemize

The first step is to start a connection to the PEERSTORE service:

@example
#include "gnunet_peerstore_service.h"

peerstore_handle = GNUNET_PEERSTORE_connect (cfg);
@end example

The service handle @code{peerstore_handle} will be needed for all
subsequent PEERSTORE operations.

@menu
* Storing records:: 
* Retrieving records:: 
* Monitoring records:: 
* Disconnecting from PEERSTORE:: 

@end menu

@node Storing records,Retrieving records,,Storing peer-specific data using the PEERSTORE service
@anchor{developers/tutorial id27}@anchor{3c7}@anchor{developers/tutorial storing-records}@anchor{3c8}
@subsubsection Storing records


To store a new record, use the following function:

@example
struct GNUNET_PEERSTORE_StoreContext *
GNUNET_PEERSTORE_store (struct GNUNET_PEERSTORE_Handle *h,
                        const char *sub_system,
                        const struct GNUNET_PeerIdentity *peer,
                        const char *key,
                        const void *value,
                        size_t size,
                        struct GNUNET_TIME_Absolute expiry,
                        enum GNUNET_PEERSTORE_StoreOption options,
                        GNUNET_PEERSTORE_Continuation cont,
                        void *cont_cls);
@end example

The @code{options} parameter can either be
@code{GNUNET_PEERSTORE_STOREOPTION_MULTIPLE} which means that multiple
values can be stored under the same key combination (subsystem, peerid,
key), or @code{GNUNET_PEERSTORE_STOREOPTION_REPLACE} which means that
PEERSTORE will replace any existing values under the given key
combination (subsystem, peerid, key) with the new given value.

The continuation function @code{cont} will be called after the store
request is successfully sent to the PEERSTORE service. This does not
guarantee that the record is successfully stored, only that it was
received by the service.

The @code{GNUNET_PEERSTORE_store} function returns a handle to the store
operation. This handle can be used to cancel the store operation only
before the continuation function is called:

@example
void
GNUNET_PEERSTORE_store_cancel (struct GNUNET_PEERSTORE_StoreContext
                               *sc);
@end example

@node Retrieving records,Monitoring records,Storing records,Storing peer-specific data using the PEERSTORE service
@anchor{developers/tutorial id28}@anchor{3c9}@anchor{developers/tutorial retrieving-records}@anchor{3ca}
@subsubsection Retrieving records


To retrieve stored records, use the following function:

@example
struct GNUNET_PEERSTORE_IterateContext *
GNUNET_PEERSTORE_iterate (struct GNUNET_PEERSTORE_Handle *h,
                          const char *sub_system,
                          const struct GNUNET_PeerIdentity *peer,
                          const char *key,
                          GNUNET_PEERSTORE_Processor callback,
                          void *callback_cls);
@end example

The values of @code{peer} and @code{key} can be @code{NULL}. This allows the
iteration over values stored under any of the following key
combinations:


@itemize -

@item 
(subsystem)

@item 
(subsystem, peerid)

@item 
(subsystem, key)

@item 
(subsystem, peerid, key)
@end itemize

The @code{callback} function will be called once with each retrieved record
and once more with a @code{NULL} record to signal the end of results.

The @code{GNUNET_PEERSTORE_iterate} function returns a handle to the
iterate operation. This handle can be used to cancel the iterate
operation only before the callback function is called with a @code{NULL}
record.

@node Monitoring records,Disconnecting from PEERSTORE,Retrieving records,Storing peer-specific data using the PEERSTORE service
@anchor{developers/tutorial id29}@anchor{3cb}@anchor{developers/tutorial monitoring-records}@anchor{3cc}
@subsubsection Monitoring records


PEERSTORE offers the functionality of monitoring for new records stored
under a specific key combination (subsystem, peerid, key). To start the
monitoring, use the following function:

@example
struct GNUNET_PEERSTORE_WatchContext *
GNUNET_PEERSTORE_watch (struct GNUNET_PEERSTORE_Handle *h,
                        const char *sub_system,
                        const struct GNUNET_PeerIdentity *peer,
                        const char *key,
                        GNUNET_PEERSTORE_Processor callback,
                        void *callback_cls);
@end example

Whenever a new record is stored under the given key combination, the
@code{callback} function will be called with this new record. This will
continue until the connection to the PEERSTORE service is broken or the
watch operation is canceled:

@example
void
GNUNET_PEERSTORE_watch_cancel (struct GNUNET_PEERSTORE_WatchContext
                               *wc);
@end example

@node Disconnecting from PEERSTORE,,Monitoring records,Storing peer-specific data using the PEERSTORE service
@anchor{developers/tutorial disconnecting-from-peerstore}@anchor{3cd}@anchor{developers/tutorial id30}@anchor{3ce}
@subsubsection Disconnecting from PEERSTORE


When the connection to the PEERSTORE service is no longer needed,
disconnect using the following function:

@example
void
GNUNET_PEERSTORE_disconnect (struct GNUNET_PEERSTORE_Handle *h,
                             int sync_first);
@end example

If the @code{sync_first} flag is set to @code{GNUNET_YES}, the API will delay
the disconnection until all store requests are received by the PEERSTORE
service. Otherwise, it will disconnect immediately.

@node Using the DHT,Debugging with gnunet-arm,Storing peer-specific data using the PEERSTORE service,Developing Applications
@anchor{developers/tutorial id31}@anchor{3cf}@anchor{developers/tutorial using-the-dht}@anchor{3d0}
@subsubsection Using the DHT


The DHT allows to store data so other peers in the P2P network can
access it and retrieve data stored by any peers in the network. This
section will explain how to use the DHT. Of course, the first thing to
do is to connect to the DHT service:

@example
dht_handle = GNUNET_DHT_connect (cfg, parallel_requests);
@end example

The second parameter indicates how many requests in parallel to expect.
It is not a hard limit, but a good approximation will make the DHT more
efficient.

@menu
* Storing data in the DHT:: 
* Obtaining data from the DHT:: 
* Implementing a block plugin:: 
* Monitoring the DHT: Monitoring the DHT<2>. 

@end menu

@node Storing data in the DHT,Obtaining data from the DHT,,Using the DHT
@anchor{developers/tutorial id32}@anchor{3d1}@anchor{developers/tutorial storing-data-in-the-dht}@anchor{3d2}
@subsubsection Storing data in the DHT


Since the DHT is a dynamic environment (peers join and leave frequently)
the data that we put in the DHT does not stay there indefinitely. It is
important to "refresh" the data periodically by simply storing it again,
in order to make sure other peers can access it.

The put API call offers a callback to signal that the PUT request has
been sent. This does not guarantee that the data is accessible to others
peers, or even that is has been stored, only that the service has
requested to a neighboring peer the retransmission of the PUT request
towards its final destination. Currently there is no feedback about
whether or not the data has been successfully stored or where it has
been stored. In order to improve the availablilty of the data and to
compensate for possible errors, peers leaving and other unfavorable
events, just make several PUT requests!

@example
message_sent_cont (void *cls,
                   const struct GNUNET_SCHEDULER_TaskContext *tc)
@{
  // Request has left local node
@}

struct GNUNET_DHT_PutHandle *
GNUNET_DHT_put (struct GNUNET_DHT_Handle *handle,
                const struct GNUNET_HashCode *key,
                uint32_t desired_replication_level,
                enum GNUNET_DHT_RouteOption options,
                enum GNUNET_BLOCK_Type type,
                size_t size,
                const void *data,
                struct GNUNET_TIME_Absolute exp,
                struct GNUNET_TIME_Relative timeout,
                GNUNET_DHT_PutContinuation cont, void *cont_cls)
@end example

Exercise: Store a value in the DHT periodically to make sure it is
available over time. You might consider using the function
@code{GNUNET\_SCHEDULER\_add\_delayed} and call @code{GNUNET\_DHT\_put} from
inside a helper function.

@node Obtaining data from the DHT,Implementing a block plugin,Storing data in the DHT,Using the DHT
@anchor{developers/tutorial id33}@anchor{3d3}@anchor{developers/tutorial obtaining-data-from-the-dht}@anchor{3d4}
@subsubsection Obtaining data from the DHT


As we saw in the previous example, the DHT works in an asynchronous
mode. Each request to the DHT is executed "in the background" and the
API calls return immediately. In order to receive results from the DHT,
the API provides a callback. Once started, the request runs in the
service, the service will try to get as many results as possible
(filtering out duplicates) until the timeout expires or we explicitly
stop the request. It is possible to give a "forever" timeout with
@code{GNUNET\_TIME\_UNIT\_FOREVER\_REL}.

If we give a route option @code{GNUNET\_DHT\_RO\_RECORD\_ROUTE} the
callback will get a list of all the peers the data has travelled, both
on the PUT path and on the GET path.

@example
static void
get_result_iterator (void *cls, struct GNUNET_TIME_Absolute expiration,
                     const struct GNUNET_HashCode *key,
                     const struct GNUNET_PeerIdentity *get_path,
                     unsigned int get_path_length,
                     const struct GNUNET_PeerIdentity *put_path,
                     unsigned int put_path_length,
                     enum GNUNET_BLOCK_Type type, size_t size,
                     const void *data)
@{
  // Optionally:
  GNUNET_DHT_get_stop (get_handle);
@}

get_handle =
      GNUNET_DHT_get_start (dht_handle,
                            block_type,
                            &key,
                            replication,
                            GNUNET_DHT_RO_NONE,
                            NULL,
                            0,
                            &get_result_iterator,
                            cls)
@end example

Exercise: Store a value in the DHT and after a while retrieve it. Show
the IDs of all the peers the requests have gone through. In order to
convert a peer ID to a string, use the function @code{GNUNET\_i2s}. Pay
attention to the route option parameters in both calls!

@node Implementing a block plugin,Monitoring the DHT<2>,Obtaining data from the DHT,Using the DHT
@anchor{developers/tutorial id34}@anchor{3d5}@anchor{developers/tutorial implementing-a-block-plugin}@anchor{3d6}
@subsubsection Implementing a block plugin


In order to store data in the DHT, it is necessary to provide a block
plugin. The DHT uses the block plugin to ensure that only well-formed
requests and replies are transmitted over the network.

The block plugin should be put in a file @code{plugin\_block\_SERVICE.c} in
the service's respective directory. The mandatory functions that need to
be implemented for a block plugin are described in the following
sections.

@menu
* Validating requests and replies:: 
* Deriving a key from a reply:: 
* Initialization of the plugin:: 
* Shutdown of the plugin:: 
* Integration of the plugin with the build system:: 

@end menu

@node Validating requests and replies,Deriving a key from a reply,,Implementing a block plugin
@anchor{developers/tutorial id35}@anchor{3d7}@anchor{developers/tutorial validating-requests-and-replies}@anchor{3d8}
@subsubsection Validating requests and replies


The evaluate function should validate a reply or a request. It returns a
@code{GNUNET\_BLOCK\_EvaluationResult}, which is an enumeration. All
possible answers are in @code{gnunet\_block\_lib.h}. The function will be
called with a @code{reply\_block} argument of @code{NULL} for requests. Note
that depending on how @code{evaluate} is called, only some of the possible
return values are valid. The specific meaning of the @code{xquery} argument
is application-specific. Applications that do not use an extended query
should check that the @code{xquery\_size} is zero. The block group is
typically used to filter duplicate replies.

@example
static enum GNUNET_BLOCK_EvaluationResult
block_plugin_SERVICE_evaluate (void *cls,
                              enum GNUNET_BLOCK_Type type,
                              struct GNUNET_BlockGroup *bg,
                              const GNUNET_HashCode *query,
                              const void *xquery,
                              size_t xquery_size,
                              const void *reply_block,
                              size_t reply_block_size)
@{
  // Verify type, block and bg
@}
@end example

Note that it is mandatory to detect duplicate replies in this function
and return the respective status code. Duplicate detection is typically
done using the Bloom filter block group provided by
@code{libgnunetblockgroup.so}. Failure to do so may cause replies to circle
in the network.

@node Deriving a key from a reply,Initialization of the plugin,Validating requests and replies,Implementing a block plugin
@anchor{developers/tutorial deriving-a-key-from-a-reply}@anchor{3d9}@anchor{developers/tutorial id36}@anchor{3da}
@subsubsection Deriving a key from a reply


The DHT can operate more efficiently if it is possible to derive a key
from the value of the corresponding block. The @code{get\_key} function is
used to obtain the key of a block --- for example, by means of hashing.
If deriving the key is not possible, the function should simply return
@code{GNUNET\_SYSERR} (the DHT will still work just fine with such blocks).

@example
static int
block_plugin_SERVICE_get_key (void *cls, enum GNUNET_BLOCK_Type type,
                             const void *block, size_t block_size,
                             struct GNUNET_HashCode *key)
@{
  // Store the key in the key argument, return GNUNET_OK on success.
@}
@end example

@node Initialization of the plugin,Shutdown of the plugin,Deriving a key from a reply,Implementing a block plugin
@anchor{developers/tutorial id37}@anchor{3db}@anchor{developers/tutorial initialization-of-the-plugin}@anchor{3dc}
@subsubsection Initialization of the plugin


The plugin is realized as a shared C library. The library must export an
initialization function which should initialize the plugin. The
initialization function specifies what block types the plugin cares
about and returns a struct with the functions that are to be used for
validation and obtaining keys (the ones just defined above).

@example
void *
libgnunet_plugin_block_SERVICE_init (void *cls)
@{
  static enum GNUNET_BLOCK_Type types[] =
  @{
    GNUNET_BLOCK_TYPE_SERVICE_BLOCKYPE,
    GNUNET_BLOCK_TYPE_ANY
  @};
  struct GNUNET_BLOCK_PluginFunctions *api;

  api = GNUNET_new (struct GNUNET_BLOCK_PluginFunctions);
  api->evaluate = &block_plugin_SERICE_evaluate;
  api->get_key = &block_plugin_SERVICE_get_key;
  api->types = types;
  return api;
@}
@end example

@node Shutdown of the plugin,Integration of the plugin with the build system,Initialization of the plugin,Implementing a block plugin
@anchor{developers/tutorial id38}@anchor{3dd}@anchor{developers/tutorial shutdown-of-the-plugin}@anchor{3de}
@subsubsection Shutdown of the plugin


Following GNUnet's general plugin API concept, the plugin must export a
second function for cleaning up. It usually does very little.

@example
void *
libgnunet_plugin_block_SERVICE_done (void *cls)
@{
  struct GNUNET_TRANSPORT_PluginFunctions *api = cls;

  GNUNET_free (api);
  return NULL;
@}
@end example

@node Integration of the plugin with the build system,,Shutdown of the plugin,Implementing a block plugin
@anchor{developers/tutorial id39}@anchor{3df}@anchor{developers/tutorial integration-of-the-plugin-with-the-build-system}@anchor{3e0}
@subsubsection Integration of the plugin with the build system


In order to compile the plugin, the @code{Makefile.am} file for the service
SERVICE should contain a rule similar to this:

@example
plugindir = $(libdir)/gnunet

plugin_LTLIBRARIES = \
        libgnunet_plugin_block_ext.la
libgnunet_plugin_block_ext_la_SOURCES = \
        plugin_block_ext.c
libgnunet_plugin_block_ext_la_LIBADD = \
        $(prefix)/lib/libgnunethello.la \
        $(prefix)/lib/libgnunetblock.la \
        $(prefix)/lib/libgnunetutil.la
libgnunet_plugin_block_ext_la_LDFLAGS = \
        $(GN_PLUGIN_LDFLAGS)
libgnunet_plugin_block_ext_la_DEPENDENCIES = \
        $(prefix)/lib/libgnunetblock.la
@end example

Exercise: Write a block plugin that accepts all queries and all replies
but prints information about queries and replies when the respective
validation hooks are called.

@node Monitoring the DHT<2>,,Implementing a block plugin,Using the DHT
@anchor{developers/tutorial id40}@anchor{3e1}@anchor{developers/tutorial monitoring-the-dht}@anchor{3e2}
@subsubsection Monitoring the DHT


It is possible to monitor the functioning of the local DHT service. When
monitoring the DHT, the service will alert the monitoring program of any
events, both started locally or received for routing from another peer.
The are three different types of events possible: a GET request, a PUT
request or a response (a reply to a GET).

Since the different events have different associated data, the API gets
3 different callbacks (one for each message type) and optional type and
key parameters, to allow for filtering of messages. When an event
happens, the appropriate callback is called with all the information
about the event.

@example
static void
get_callback (void *cls,
              enum GNUNET_DHT_RouteOption options,
              enum GNUNET_BLOCK_Type type,
              uint32_t hop_count,
              uint32_t desired_replication_level,
              unsigned int path_length,
              const struct GNUNET_PeerIdentity *path,
              const struct GNUNET_HashCode * key)
@{
@}


static void
get_resp_callback (void *cls,
                   enum GNUNET_BLOCK_Type type,
                   const struct GNUNET_PeerIdentity *get_path,
                   unsigned int get_path_length,
                   const struct GNUNET_PeerIdentity *put_path,
                   unsigned int put_path_length,
                   struct GNUNET_TIME_Absolute exp,
                   const struct GNUNET_HashCode * key,
                   const void *data,
                   size_t size)
@{
@}


static void
put_callback (void *cls,
              enum GNUNET_DHT_RouteOption options,
              enum GNUNET_BLOCK_Type type,
              uint32_t hop_count,
              uint32_t desired_replication_level,
              unsigned int path_length,
              const struct GNUNET_PeerIdentity *path,
              struct GNUNET_TIME_Absolute exp,
              const struct GNUNET_HashCode * key,
              const void *data,
              size_t size)
@{
@}


monitor_handle = GNUNET_DHT_monitor_start (dht_handle,
                                          block_type,
                                          key,
                                          &get_callback,
                                          &get_resp_callback,
                                          &put_callback,
                                          cls);
@end example

@node Debugging with gnunet-arm,,Using the DHT,Developing Applications
@anchor{developers/tutorial debugging-with-gnunet-002darm}@anchor{3e3}@anchor{developers/tutorial debugging-with-gnunet-arm}@anchor{3e4}
@subsubsection Debugging with gnunet-arm


Even if services are managed by @code{gnunet-arm}, you can start them with
@code{gdb} or @code{valgrind}. For example, you could add the following lines
to your configuration file to start the DHT service in a @code{gdb} session
in a fresh @code{xterm}:

@example
[dht]
PREFIX=xterm -e gdb --args
@end example

Alternatively, you can stop a service that was started via ARM and run
it manually:

@example
$ gnunet-arm -k dht
$ gdb --args gnunet-service-dht -L DEBUG
$ valgrind gnunet-service-dht -L DEBUG
@end example

Assuming other services are well-written, they will automatically
re-integrate the restarted service with the peer.

GNUnet provides a powerful logging mechanism providing log levels
@code{ERROR}, @code{WARNING}, @code{INFO} and @code{DEBUG}. The current log level is
configured using the @code{$GNUNET_FORCE_LOG} environmental variable. The
@code{DEBUG} level is only available if @code{--enable-logging=verbose} was
used when running @code{configure}. More details about logging can be found
under @indicateurl{https://docs.gnunet.org/#Logging}.

You should also probably enable the creation of core files, by setting
@code{ulimit}, and echo'ing @code{1} into
@code{/proc/sys/kernel/core\_uses\_pid}. Then you can investigate the core
dumps with @code{gdb}, which is often the fastest method to find simple
errors.

Exercise: Add a memory leak to your service and obtain a trace pointing
to the leak using @code{valgrind} while running the service from
@code{gnunet-service-arm}.

@node GNU Free Documentation License,,Developing Applications,C Tutorial
@anchor{developers/tutorial gnu-free-documentation-license}@anchor{3e5}@anchor{developers/tutorial id41}@anchor{3e6}
@subsection GNU Free Documentation License


license, GNU Free Documentation License
Version 1.3, 3 November 2008

@example
Copyright © 2000, 2001, 2002, 2007, 2008 Free Software Foundation, Inc.
http://fsf.org/

Everyone is permitted to copy and distribute verbatim copies
of this license document, but changing it is not allowed.
@end example


@enumerate 

@item 
PREAMBLE

The purpose of this License is to make a manual, textbook, or other
functional and useful document free in the sense of freedom: to
assure everyone the effective freedom to copy and redistribute it,
with or without modifying it, either commercially or
noncommercially. Secondarily, this License preserves for the author
and publisher a way to get credit for their work, while not being
considered responsible for modifications made by others.

This License is a kind of "copyleft", which means that derivative
works of the document must themselves be free in the same sense. It
complements the GNU General Public License, which is a copyleft
license designed for free software.

We have designed this License in order to use it for manuals for
free software, because free software needs free documentation: a
free program should come with manuals providing the same freedoms
that the software does. But this License is not limited to software
manuals; it can be used for any textual work, regardless of subject
matter or whether it is published as a printed book. We recommend
this License principally for works whose purpose is instruction or
reference.

@item 
APPLICABILITY AND DEFINITIONS

This License applies to any manual or other work, in any medium,
that contains a notice placed by the copyright holder saying it can
be distributed under the terms of this License. Such a notice grants
a world-wide, royalty-free license, unlimited in duration, to use
that work under the conditions stated herein. The "Document", below,
refers to any such manual or work. Any member of the public is a
licensee, and is addressed as "you". You accept the license if you
copy, modify or distribute the work in a way requiring permission
under copyright law.

A "Modified Version" of the Document means any work containing the
Document or a portion of it, either copied verbatim, or with
modifications and/or translated into another language.

A "Secondary Section" is a named appendix or a front-matter section
of the Document that deals exclusively with the relationship of the
publishers or authors of the Document to the Document's overall
subject (or to related matters) and contains nothing that could fall
directly within that overall subject. (Thus, if the Document is in
part a textbook of mathematics, a Secondary Section may not explain
any mathematics.) The relationship could be a matter of historical
connection with the subject or with related matters, or of legal,
commercial, philosophical, ethical or political position regarding
them.

The "Invariant Sections" are certain Secondary Sections whose titles
are designated, as being those of Invariant Sections, in the notice
that says that the Document is released under this License. If a
section does not fit the above definition of Secondary then it is
not allowed to be designated as Invariant. The Document may contain
zero Invariant Sections. If the Document does not identify any
Invariant Sections then there are none.

The "Cover Texts" are certain short passages of text that are
listed, as Front-Cover Texts or Back-Cover Texts, in the notice that
says that the Document is released under this License. A Front-Cover
Text may be at most 5 words, and a Back-Cover Text may be at most 25
words.

A "Transparent" copy of the Document means a machine-readable copy,
represented in a format whose specification is available to the
general public, that is suitable for revising the document
straightforwardly with generic text editors or (for images composed
of pixels) generic paint programs or (for drawings) some widely
available drawing editor, and that is suitable for input to text
formatters or for automatic translation to a variety of formats
suitable for input to text formatters. A copy made in an otherwise
Transparent file format whose markup, or absence of markup, has been
arranged to thwart or discourage subsequent modification by readers
is not Transparent. An image format is not Transparent if used for
any substantial amount of text. A copy that is not "Transparent" is
called "Opaque".

Examples of suitable formats for Transparent copies include plain
ASCII without markup, Texinfo input format, LaTeX input format, SGML
or XML using a publicly available DTD, and standard-conforming
simple HTML, PostScript or PDF designed for human modification.
Examples of transparent image formats include PNG, XCF and JPG.
Opaque formats include proprietary formats that can be read and
edited only by proprietary word processors, SGML or XML for which
the DTD and/or processing tools are not generally available, and the
machine-generated HTML, PostScript or PDF produced by some word
processors for output purposes only.

The "Title Page" means, for a printed book, the title page itself,
plus such following pages as are needed to hold, legibly, the
material this License requires to appear in the title page. For
works in formats which do not have any title page as such, "Title
Page" means the text near the most prominent appearance of the
work's title, preceding the beginning of the body of the text.

The "publisher" means any person or entity that distributes copies
of the Document to the public.

A section "Entitled XYZ" means a named subunit of the Document whose
title either is precisely XYZ or contains XYZ in parentheses
following text that translates XYZ in another language. (Here XYZ
stands for a specific section name mentioned below, such as
"Acknowledgements", "Dedications", "Endorsements", or "History".) To
"Preserve the Title" of such a section when you modify the Document
means that it remains a section "Entitled XYZ" according to this
definition.

The Document may include Warranty Disclaimers next to the notice
which states that this License applies to the Document. These
Warranty Disclaimers are considered to be included by reference in
this License, but only as regards disclaiming warranties: any other
implication that these Warranty Disclaimers may have is void and has
no effect on the meaning of this License.

@item 
VERBATIM COPYING

You may copy and distribute the Document in any medium, either
commercially or noncommercially, provided that this License, the
copyright notices, and the license notice saying this License
applies to the Document are reproduced in all copies, and that you
add no other conditions whatsoever to those of this License. You may
not use technical measures to obstruct or control the reading or
further copying of the copies you make or distribute. However, you
may accept compensation in exchange for copies. If you distribute a
large enough number of copies you must also follow the conditions in
section 3.

You may also lend copies, under the same conditions stated above,
and you may publicly display copies.

@item 
COPYING IN QUANTITY

If you publish printed copies (or copies in media that commonly have
printed covers) of the Document, numbering more than 100, and the
Document's license notice requires Cover Texts, you must enclose the
copies in covers that carry, clearly and legibly, all these Cover
Texts: Front-Cover Texts on the front cover, and Back-Cover Texts on
the back cover. Both covers must also clearly and legibly identify
you as the publisher of these copies. The front cover must present
the full title with all words of the title equally prominent and
visible. You may add other material on the covers in addition.
Copying with changes limited to the covers, as long as they preserve
the title of the Document and satisfy these conditions, can be
treated as verbatim copying in other respects.

If the required texts for either cover are too voluminous to fit
legibly, you should put the first ones listed (as many as fit
reasonably) on the actual cover, and continue the rest onto adjacent
pages.

If you publish or distribute Opaque copies of the Document numbering
more than 100, you must either include a machine-readable
Transparent copy along with each Opaque copy, or state in or with
each Opaque copy a computer-network location from which the general
network-using public has access to download using public-standard
network protocols a complete Transparent copy of the Document, free
of added material. If you use the latter option, you must take
reasonably prudent steps, when you begin distribution of Opaque
copies in quantity, to ensure that this Transparent copy will remain
thus accessible at the stated location until at least one year after
the last time you distribute an Opaque copy (directly or through
your agents or retailers) of that edition to the public.

It is requested, but not required, that you contact the authors of
the Document well before redistributing any large number of copies,
to give them a chance to provide you with an updated version of the
Document.

@item 
MODIFICATIONS

You may copy and distribute a Modified Version of the Document under
the conditions of sections 2 and 3 above, provided that you release
the Modified Version under precisely this License, with the Modified
Version filling the role of the Document, thus licensing
distribution and modification of the Modified Version to whoever
possesses a copy of it. In addition, you must do these things in the
Modified Version:


@enumerate A

@item 
Use in the Title Page (and on the covers, if any) a title
distinct from that of the Document, and from those of previous
versions (which should, if there were any, be listed in the
History section of the Document). You may use the same title as a
previous version if the original publisher of that version gives
permission.

@item 
List on the Title Page, as authors, one or more persons or
entities responsible for authorship of the modifications in the
Modified Version, together with at least five of the principal
authors of the Document (all of its principal authors, if it has
fewer than five), unless they release you from this requirement.

@item 
State on the Title page the name of the publisher of the Modified
Version, as the publisher.

@item 
Preserve all the copyright notices of the Document.

@item 
Add an appropriate copyright notice for your modifications
adjacent to the other copyright notices.

@item 
Include, immediately after the copyright notices, a license
notice giving the public permission to use the Modified Version
under the terms of this License, in the form shown in the
Addendum below.

@item 
Preserve in that license notice the full lists of Invariant
Sections and required Cover Texts given in the Document's license
notice.

@item 
Include an unaltered copy of this License.

@item 
Preserve the section Entitled "History", Preserve its Title, and
add to it an item stating at least the title, year, new authors,
and publisher of the Modified Version as given on the Title Page.
If there is no section Entitled "History" in the Document, create
one stating the title, year, authors, and publisher of the
Document as given on its Title Page, then add an item describing
the Modified Version as stated in the previous sentence.

@item 
Preserve the network location, if any, given in the Document for
public access to a Transparent copy of the Document, and likewise
the network locations given in the Document for previous versions
it was based on. These may be placed in the "History" section.
You may omit a network location for a work that was published at
least four years before the Document itself, or if the original
publisher of the version it refers to gives permission.

@item 
For any section Entitled "Acknowledgements" or "Dedications",
Preserve the Title of the section, and preserve in the section
all the substance and tone of each of the contributor
acknowledgements and/or dedications given therein.

@item 
Preserve all the Invariant Sections of the Document, unaltered in
their text and in their titles. Section numbers or the equivalent
are not considered part of the section titles.

@item 
Delete any section Entitled "Endorsements". Such a section may
not be included in the Modified Version.

@item 
Do not retitle any existing section to be Entitled "Endorsements"
or to conflict in title with any Invariant Section.

@item 
Preserve any Warranty Disclaimers.
@end enumerate

If the Modified Version includes new front-matter sections or
appendices that qualify as Secondary Sections and contain no
material copied from the Document, you may at your option designate
some or all of these sections as invariant. To do this, add their
titles to the list of Invariant Sections in the Modified Version's
license notice. These titles must be distinct from any other section
titles.

You may add a section Entitled "Endorsements", provided it contains
nothing but endorsements of your Modified Version by various
parties---for example, statements of peer review or that the text
has been approved by an organization as the authoritative definition
of a standard.

You may add a passage of up to five words as a Front-Cover Text, and
a passage of up to 25 words as a Back-Cover Text, to the end of the
list of Cover Texts in the Modified Version. Only one passage of
Front-Cover Text and one of Back-Cover Text may be added by (or
through arrangements made by) any one entity. If the Document
already includes a cover text for the same cover, previously added
by you or by arrangement made by the same entity you are acting on
behalf of, you may not add another; but you may replace the old one,
on explicit permission from the previous publisher that added the
old one.

The author(s) and publisher(s) of the Document do not by this
License give permission to use their names for publicity for or to
assert or imply endorsement of any Modified Version.

@item 
COMBINING DOCUMENTS

You may combine the Document with other documents released under
this License, under the terms defined in section 4 above for
modified versions, provided that you include in the combination all
of the Invariant Sections of all of the original documents,
unmodified, and list them all as Invariant Sections of your combined
work in its license notice, and that you preserve all their Warranty
Disclaimers.

The combined work need only contain one copy of this License, and
multiple identical Invariant Sections may be replaced with a single
copy. If there are multiple Invariant Sections with the same name
but different contents, make the title of each such section unique
by adding at the end of it, in parentheses, the name of the original
author or publisher of that section if known, or else a unique
number. Make the same adjustment to the section titles in the list
of Invariant Sections in the license notice of the combined work.

In the combination, you must combine any sections Entitled "History"
in the various original documents, forming one section Entitled
"History"; likewise combine any sections Entitled
"Acknowledgements", and any sections Entitled "Dedications". You
must delete all sections Entitled "Endorsements."

@item 
COLLECTIONS OF DOCUMENTS

You may make a collection consisting of the Document and other
documents released under this License, and replace the individual
copies of this License in the various documents with a single copy
that is included in the collection, provided that you follow the
rules of this License for verbatim copying of each of the documents
in all other respects.

You may extract a single document from such a collection, and
distribute it individually under this License, provided you insert a
copy of this License into the extracted document, and follow this
License in all other respects regarding verbatim copying of that
document.

@item 
AGGREGATION WITH INDEPENDENT WORKS

A compilation of the Document or its derivatives with other separate
and independent documents or works, in or on a volume of a storage
or distribution medium, is called an "aggregate" if the copyright
resulting from the compilation is not used to limit the legal rights
of the compilation's users beyond what the individual works permit.
When the Document is included in an aggregate, this License does not
apply to the other works in the aggregate which are not themselves
derivative works of the Document.

If the Cover Text requirement of section 3 is applicable to these
copies of the Document, then if the Document is less than one half
of the entire aggregate, the Document's Cover Texts may be placed on
covers that bracket the Document within the aggregate, or the
electronic equivalent of covers if the Document is in electronic
form. Otherwise they must appear on printed covers that bracket the
whole aggregate.

@item 
TRANSLATION

Translation is considered a kind of modification, so you may
distribute translations of the Document under the terms of section
4. Replacing Invariant Sections with translations requires special
permission from their copyright holders, but you may include
translations of some or all Invariant Sections in addition to the
original versions of these Invariant Sections. You may include a
translation of this License, and all the license notices in the
Document, and any Warranty Disclaimers, provided that you also
include the original English version of this License and the
original versions of those notices and disclaimers. In case of a
disagreement between the translation and the original version of
this License or a notice or disclaimer, the original version will
prevail.

If a section in the Document is Entitled "Acknowledgements",
"Dedications", or "History", the requirement (section 4) to Preserve
its Title (section 1) will typically require changing the actual
title.

@item 
TERMINATION

You may not copy, modify, sublicense, or distribute the Document
except as expressly provided under this License. Any attempt
otherwise to copy, modify, sublicense, or distribute it is void, and
will automatically terminate your rights under this License.

However, if you cease all violation of this License, then your
license from a particular copyright holder is reinstated (a)
provisionally, unless and until the copyright holder explicitly and
finally terminates your license, and (b) permanently, if the
copyright holder fails to notify you of the violation by some
reasonable means prior to 60 days after the cessation.

Moreover, your license from a particular copyright holder is
reinstated permanently if the copyright holder notifies you of the
violation by some reasonable means, this is the first time you have
received notice of violation of this License (for any work) from
that copyright holder, and you cure the violation prior to 30 days
after your receipt of the notice.

Termination of your rights under this section does not terminate the
licenses of parties who have received copies or rights from you
under this License. If your rights have been terminated and not
permanently reinstated, receipt of a copy of some or all of the same
material does not give you any rights to use it.

@item 
FUTURE REVISIONS OF THIS LICENSE

The Free Software Foundation may publish new, revised versions of
the GNU Free Documentation License from time to time. Such new
versions will be similar in spirit to the present version, but may
differ in detail to address new problems or concerns. See
@indicateurl{http://www.gnu.org/copyleft/}.

Each version of the License is given a distinguishing version
number. If the Document specifies that a particular numbered version
of this License "or any later version" applies to it, you have the
option of following the terms and conditions either of that
specified version or of any later version that has been published
(not as a draft) by the Free Software Foundation. If the Document
does not specify a version number of this License, you may choose
any version ever published (not as a draft) by the Free Software
Foundation. If the Document specifies that a proxy can decide which
future versions of this License can be used, that proxy's public
statement of acceptance of a version permanently authorizes you to
choose that version for the Document.

@item 
RELICENSING

"Massive Multiauthor Collaboration Site" (or "MMC Site") means any
World Wide Web server that publishes copyrightable works and also
provides prominent facilities for anybody to edit those works. A
public wiki that anybody can edit is an example of such a server. A
"Massive Multiauthor Collaboration" (or "MMC") contained in the site
means any set of copyrightable works thus published on the MMC site.

"CC-BY-SA" means the Creative Commons Attribution-Share Alike 3.0
license published by Creative Commons Corporation, a not-for-profit
corporation with a principal place of business in San Francisco,
California, as well as future copyleft versions of that license
published by that same organization.

"Incorporate" means to publish or republish a Document, in whole or
in part, as part of another Document.

An MMC is "eligible for relicensing" if it is licensed under this
License, and if all works that were first published under this
License somewhere other than this MMC, and subsequently incorporated
in whole or in part into the MMC, (1) had no cover texts or
invariant sections, and (2) were thus incorporated prior to November
1, 2008.

The operator of an MMC Site may republish an MMC contained in the
site under CC-BY-SA on the same site at any time before August 1,
2009, provided the MMC is eligible for relicensing.
@end enumerate

`ADDENDUM: How to use this License for your documents'

To use this License in a document you have written, include a copy of
the License in the document and put the following copyright and license
notices just after the title page:

@example
Copyright (C)  year  your name.
Permission is granted to copy, distribute and/or modify this document
under the terms of the GNU Free Documentation License, Version 1.3
or any later version published by the Free Software Foundation;
with no Invariant Sections, no Front-Cover Texts, and no Back-Cover
Texts.  A copy of the license is included in the section entitled `@w{`}GNU
Free Documentation License'@w{'}.
@end example

If you have Invariant Sections, Front-Cover Texts and Back-Cover Texts,
replace the "with...Texts." line with this:

@example
with the Invariant Sections being list their titles, with
the Front-Cover Texts being list, and with the Back-Cover Texts
being list.
@end example

If you have Invariant Sections without Cover Texts, or some other
combination of the three, merge those two alternatives to suit the
situation.

If your document contains nontrivial examples of program code, we
recommend releasing these examples in parallel under your choice of free
software license, such as the GNU General Public License, to permit
their use in free software.

@node Specifications,Doxygen,C Tutorial,Developer Handbook
@anchor{developers/livingstandards doc}@anchor{3e7}@anchor{developers/livingstandards specifications}@anchor{3e8}
@section Specifications


On this page you can find links to our RFC-style technical protocol
specifications. We have dubbed our specifications "Living Standards Documents (LSDs)".
Some LSDs have at some point become RFCs, in which case you can find the link
to the RFC below.
RFCs, once published, are not changed.
LSDs, however, "live" and always represent the current status of the implemented
protocol. The RFC in such cases only represents a snapshot in this case which
may or may not represent the current status of the corresponding LSD.


@itemize -

@item 
LSD 0000: Reserved

@item 
LSD 0001@footnote{https://lsd.gnunet.org/lsd0001} [RFC 9498@footnote{https://www.rfc-editor.org/rfc/rfc9498.html}]: `The GNU Name System'

@item 
LSD 0002@footnote{https://lsd.gnunet.org/lsd0002}: `re:claimID'

@item 
LSD 0003@footnote{https://lsd.gnunet.org/lsd0003}: `Byzantine Fault Tolerant Set Reconciliation'

@item 
LSD 0004@footnote{https://lsd.gnunet.org/lsd0004}: `The R5N Distributed Hash Table'

@item 
LSD 0005@footnote{https://lsd.gnunet.org/lsd0005}: `The GNU Name System DID Method'

@item 
LSD 0006@footnote{https://lsd.gnunet.org/lsd0006}: `The 'taler' URI scheme for GNU Taler Wallet interactions'

@item 
LSD 0007@footnote{https://lsd.gnunet.org/lsd0007}: `The GNUnet communicators'
@end itemize

@node Doxygen,,Specifications,Developer Handbook
@anchor{developers/doxygen doc}@anchor{3e9}@anchor{developers/doxygen doxygen}@anchor{3ea}
@section Doxygen


The code documentation will soon be ported here.
Until then you can find it in the old place: @indicateurl{https://docs.gnunet.org/doxygen/}

@c .. doxygenindex::

@node GUIs,FAQs,Developer Handbook,Top
@anchor{guis/index doc}@anchor{3eb}@anchor{guis/index gnu-affero-general-public-license}@anchor{3ec}@anchor{guis/index guis}@anchor{3ed}
@chapter GUIs


@menu
* The graphical configuration interface:: 
* GTK File-sharing User Interface:: 

@end menu

@node The graphical configuration interface,GTK File-sharing User Interface,,GUIs
@anchor{guis/gtk doc}@anchor{3ee}@anchor{guis/gtk id1}@anchor{3ef}@anchor{guis/gtk the-graphical-configuration-interface}@anchor{3f0}
@section The graphical configuration interface


@c Inserted from Installation Handbook in original "order".

@c original FIXME said to move to the user's handbook...
@c WGL thinks somewhere else...

If you also would like to use @code{gnunet-gtk} and @code{gnunet-setup}
(highly recommended for beginners), do:

@menu
* Configuring your peer:: 
* Configuration of the HOSTLIST proxy settings: Configuration of the HOSTLIST proxy settings<2>. 
* Configuration of the HTTP and HTTPS transport plugins: Configuration of the HTTP and HTTPS transport plugins<2>. 

@end menu

@node Configuring your peer,Configuration of the HOSTLIST proxy settings<2>,,The graphical configuration interface
@anchor{guis/gtk configuring-your-peer}@anchor{3f1}@anchor{guis/gtk id2}@anchor{3f2}
@subsection Configuring your peer


This chapter will describe the various configuration options in GNUnet.

The easiest way to configure your peer is to use the @code{gnunet-setup}
tool. @code{gnunet-setup} is part of the @code{gnunet-gtk} package. You might
have to install it separately.

Many of the specific sections from this chapter actually are linked from
within @code{gnunet-setup} to help you while using the setup tool.

While you can also configure your peer by editing the configuration file
by hand, this is not recommended for anyone except for developers as it
requires a more in-depth understanding of the configuration files and
internal dependencies of GNUnet.

@node Configuration of the HOSTLIST proxy settings<2>,Configuration of the HTTP and HTTPS transport plugins<2>,Configuring your peer,The graphical configuration interface
@anchor{guis/gtk configuration-of-the-hostlist-proxy-settings}@anchor{3f3}@anchor{guis/gtk id3}@anchor{3f4}
@subsection Configuration of the HOSTLIST proxy settings


The hostlist client can be configured to use a proxy to connect to the
hostlist server. This functionality can be configured in the
configuration file directly or using the @code{gnunet-setup} tool.

The hostlist client supports the following proxy types at the moment:


@itemize -

@item 
HTTP and HTTP 1.0 only proxy

@item 
SOCKS 4/4a/5/5 with hostname
@end itemize

In addition authentication at the proxy with username and password can
be configured.

To configure proxy support for the hostlist client in the
@code{gnunet-setup} tool, select the "hostlist" tab and select the
appropriate proxy type. The hostname or IP address (including port if
required) has to be entered in the "Proxy hostname" textbox. If
required, enter username and password in the "Proxy username" and
"Proxy password" boxes. Be aware that this information will be stored
in the configuration in plain text (TODO: Add explanation and generalize
the part in Chapter 3.6 about the encrypted home).

@node Configuration of the HTTP and HTTPS transport plugins<2>,,Configuration of the HOSTLIST proxy settings<2>,The graphical configuration interface
@anchor{guis/gtk configuration-of-the-http-and-https-transport-plugins}@anchor{3f5}@anchor{guis/gtk id4}@anchor{3f6}
@subsection Configuration of the HTTP and HTTPS transport plugins


The client parts of the http and https transport plugins can be
configured to use a proxy to connect to the hostlist server. This
functionality can be configured in the configuration file directly or
using the gnunet-setup tool.

Both the HTTP and HTTPS clients support the following proxy types at the
moment:


@itemize -

@item 
HTTP 1.1 proxy

@item 
SOCKS 4/4a/5/5 with hostname
@end itemize

In addition authentication at the proxy with username and password can
be configured.

To configure proxy support for the clients in the gnunet-setup tool,
select the "transport" tab and activate the respective plugin. Now you
can select the appropriate proxy type. The hostname or IP address
(including port if required) has to be entered in the "Proxy hostname"
textbox. If required, enter username and password in the "Proxy
username" and "Proxy password" boxes. Be aware that these information
will be stored in the configuration in plain text.

@node GTK File-sharing User Interface,,The graphical configuration interface,GUIs
@anchor{guis/fs-gtk doc}@anchor{3f7}@anchor{guis/fs-gtk gtk-file-sharing-user-interface}@anchor{3f8}@anchor{guis/fs-gtk gtk-user-interface}@anchor{3f9}
@section GTK File-sharing User Interface


This chapter describes first steps for file-sharing with GNUnet. To
start, you should launch @code{gnunet-fs-gtk}.

As we want to be sure that the network contains the data that we are
looking for for testing, we need to begin by publishing a file.

@menu
* Publishing: Publishing<2>. 
* Searching: Searching<2>. 
* Downloading: Downloading<2>. 

@end menu

@node Publishing<2>,Searching<2>,,GTK File-sharing User Interface
@anchor{guis/fs-gtk gtk-002dpublishing}@anchor{3fa}@anchor{guis/fs-gtk publishing}@anchor{3fb}
@subsection Publishing


To publish a file, select "File Sharing" in the menu bar just below
the "Statistics" icon, and then select "Publish" from the menu.

Afterwards, the following publishing dialog will appear:

@image{gnunet-figures/gnunet-gtk-0-10-fs-publish,,,,png}

In this dialog, select the "Add File" button. This will open a file
selection dialog:

@image{gnunet-figures/gnunet-gtk-0-10-fs-publish-select,,,,png}

Now, you should select a file from your computer to be published on
GNUnet. To see more of GNUnet's features later, you should pick a PNG or
JPEG file this time. You can leave all of the other options in the
dialog unchanged. Confirm your selection by pressing the "OK" button
in the bottom right corner. Now, you will briefly see a "Messages..."
dialog pop up, but most likely it will be too short for you to really
read anything. That dialog is showing you progress information as GNUnet
takes a first look at the selected file(s). For a normal image, this is
virtually instant, but if you later import a larger directory you might
be interested in the progress dialog and potential errors that might be
encountered during processing. After the progress dialog automatically
disappears, your file should now appear in the publishing dialog:

@image{gnunet-figures/gnunet-gtk-0-10-fs-publish-with-file,,,,png}

Now, select the file (by clicking on the file name) and then click the
"Edit" button. This will open the editing dialog:

@image{gnunet-figures/gnunet-gtk-0-10-fs-publish-editing,,,,png}

In this dialog, you can see many details about your file. In the top
left area, you can see meta data extracted about the file, such as the
original filename, the mimetype and the size of the image. In the top
right, you should see a preview for the image (if GNU libextractor was
installed correctly with the respective plugins). Note that if you do
not see a preview, this is not a disaster, but you might still want to
install more of GNU libextractor in the future. In the bottom left, the
dialog contains a list of keywords. These are the keywords under which
the file will be made available. The initial list will be based on the
extracted meta data. Additional publishing options are in the right
bottom corner. We will now add an additional keyword to the list of
keywords. This is done by entering the keyword above the keyword list
between the label "Keyword" and the "Add keyword" button. Enter
"test" and select "Add keyword". Note that the keyword will appear
at the bottom of the existing keyword list, so you might have to scroll
down to see it. Afterwards, push the "OK" button at the bottom right
of the dialog.

You should now be back at the "Publish content on GNUnet" dialog.
Select "Execute" in the bottom right to close the dialog and publish
your file on GNUnet! Afterwards, you should see the main dialog with a
new area showing the list of published files (or ongoing publishing
operations with progress indicators).

@node Searching<2>,Downloading<2>,Publishing<2>,GTK File-sharing User Interface
@anchor{guis/fs-gtk gtk-002dsearching}@anchor{3fc}@anchor{guis/fs-gtk searching}@anchor{3fd}
@subsection Searching


Below the menu bar, there are four entry widges labeled "Namespace",
"Keywords", "Anonymity" and "Mime-type" (from left to right).
These widgets are used to control searching for files in GNUnet. Between
the "Keywords" and "Anonymity" widgets, there is also a big
"Search" button, which is used to initiate the search. We will ignore
the "Namespace", "Anonymity" and "Mime-type" options in this
tutorial, please leave them empty. Instead, simply enter "test" under
"Keywords" and press "Search". Afterwards, you should immediately
see a new tab labeled after your search term, followed by the (current)
number of search results --- "(15)" in our screenshot. Note that your
results may vary depending on what other users may have shared and how
your peer is connected.

You can now select one of the search results. Once you do this,
additional information about the result should be displayed on the
right. If available, a preview image should appear on the top right.
Meta data describing the file will be listed at the bottom right.

Once a file is selected, at the bottom of the search result list a
little area for downloading appears.

@node Downloading<2>,,Searching<2>,GTK File-sharing User Interface
@anchor{guis/fs-gtk downloading}@anchor{3fe}@anchor{guis/fs-gtk gtk-002ddownloading}@anchor{3ff}
@subsection Downloading


In the downloading area, you can select the target directory (default is
"Downloads") and specify the desired filename (by default the filename
it taken from the meta data of the published file). Additionally, you
can specify if the download should be anonymous and (for directories) if
the download should be recursive. In most cases, you can simply start
the download with the "Download!" button.

Once you selected download, the progress of the download will be
displayed with the search result. You may need to resize the result list
or scroll to the right. The "Status" column shows the current status
of the download, and "Progress" how much has been completed. When you
close the search tab (by clicking on the "X" button next to the
"test" label), ongoing and completed downloads are not aborted but
moved to a special "*" tab.

You can remove completed downloads from the "*" tab by clicking the
cleanup button next to the "*". You can also abort downloads by right
clicking on the respective download and selecting "Abort download"
from the menu.

That's it, you now know the basics for file-sharing with GNUnet!

@node FAQs,Index,GUIs,Top
@anchor{faq doc}@anchor{400}@anchor{faq faqs}@anchor{401}
@chapter FAQs


@menu
* General:: 
* Features: Features<2>. 
* GNU Name System:: 
* Error messages:: 
* File-sharing: File-sharing<2>. 
* Contributing: Contributing<2>. 

@end menu

@node General,Features<2>,,FAQs
@anchor{faq general}@anchor{402}
@section General


General questions about the project.


@table @asis

@item What do I do if my question is not answered here?

A: There are many other sources of information. You can read additional documentation or ask the question on the @email{help-gnunet@@gnu.org} mailing list.

@item When are you going to release the next version?

A: The general answer is, when it is ready. A better answer may be: earlier if you contribute (test, debug, code, document). Every release will be anounced on the @email{info-gnunet@@gnu.org} mailing list and on planet GNU. You can subscribe to the mailing list or the RSS feed of this site to automatically receive a notification.

@item Is the code free?

A: GNUnet is free software, available under the GNU Affero Public License (AGPL).

@item Are there any known bugs?

A: We track the list of currently known bugs in the Mantis system. Some bugs are occasionally reported directly to developers or the developer mailing list. This is discouraged since developers often do not have the time to feed these bugs back into the Mantis database. Please report bugs directly to the bug tracking system. If you believe a bug is sensitive, you can set its view status to private (this should be the exception).

@item Is there a graphical user interface?

A: gnunet-gtk is a separate download. The package contains various GTK+ based graphical interfaces, including a graphical tool for configuration.

@item Why does gnunet-service-nse create a high CPU load?

A: The gnunet-service-nse process will initially compute a so-called "proof-of-work" which is used to convince the network that your peer is real (or, rather, make it expensive for an adversary to mount a Sybil attack on the network size estimator). The calculation is expected to take a few days, depending on how fast your CPU is. If the CPU load is creating a problem for you, you can set the value "WORKDELAY" in the "nse" section of your configuration file to a higher value. The default is "5 ms".

@item How does GNUnet compare to Tor?

A: Tor focuses on anonymous communication and censorship-resistance for TCP connections and, with the Tor Browser Bundle, for the Web in particular. GNUnet does not really have one focus; our theme is secure decentralized networking, but that is too broad to be called a focus.

@item How does GNUnet compare to I2P?

A: Both GNUnet and I2P want to build a better, more secure, more decentralized Internet. However, on the technical side, there are almost no overlaps.
I2P is written in Java, and has (asymmetric) tunnels using onion (or garlic) routing as the basis for various (anonymized) applications. I2P is largely used via a Web frontend.

@item Is GNUnet ready for use on production systems?

A: GNUnet is still undergoing major development. It is largely not yet ready for usage beyond developers. Your mileage will vary depending on the functionality you use, but you will always likely run into issues with our current low-level transport system. We are currently in the process of rewriting it (Project "Transport Next Generation [TNG]")

@item Is GNUnet build using distributed ledger technologies?

A: No. GNUnet is a new network protocol stack for building secure, distributed, and privacy-preserving applications. While a ledger could be built using GNUnet, we currently have no plans in doing so.
@end table

@node Features<2>,GNU Name System,General,FAQs
@anchor{faq features}@anchor{403}
@section Features



@table @asis

@item What can I do with GNUnet?

A: GNUnet is a peer-to-peer framework, by which we mostly mean that it can do more than just one thing. Naturally, the implementation and documentation of some of the features that exist are more advanced than others. For users, GNUnet offers anonymous and non-anonymous file-sharing, a fully decentralized and censorship-resistant replacement for DNS and a mechanism for IPv4-IPv6 protocol translation and tunneling (NAT-PT with DNS-ALG). See also: Applications.

@item Is it possible to surf the WWW anonymously with GNUnet?

A: It is not possible use GNUnet for anonymous browsing at this point. We recommend that you use Tor for anonymous surfing.

@item Is it possible to access GNUnet via a browser as an anonymous WWW?

A: There is currently no proxy (like fproxy in Freenet) for GNUnet that would make it accessible via a browser. It is possible to build such a proxy and all one needs to know is the protocol used between the browser and the proxy and the GNUnet code for file-sharing.

@item Is there a graphical user interface?

A: There are actually a few graphical user interfaces for different functions. gnunet-setup is to configure GNUnet, and gnunet-fs-gtk is for file-sharing. There are a few other gnunet-XXX-gtk GUIs of lesser importance. Note that in order to obtain the GUI, you need to install the gnunet-gtk package, which is a separate download. gnunet-gtk is a meta GUI that integrates most of the other GUIs in one window. One exception is gnunet-setup, which must still be run separately at this time (as setup requires the peer to be stopped).

@item Which operating systems does GNUnet run on?

A: GNUnet is being developed and tested primarily under Debian GNU/Linux. Furthermore, we regularly build and test GNUnet on Fedora, Ubuntu, Arch, FreeBSD and macOS. We have reports of working versions on many other GNU/Linux distributions; in the past we had reports of working versions on NetBSD, OpenBSD and Solaris. However, not all of those reports are recent, so if you cannot get GNUnet to work on those systems please let us know.
@end table

@node GNU Name System,Error messages,Features<2>,FAQs
@anchor{faq gnu-name-system}@anchor{404}
@section GNU Name System



@table @asis

@item Who runs the GNS root zone?

A: Short answer: you. The long answer is the GNUnet will ship with a default configuration of top-level domains. The governance of this default configuration is not yet established. In any case, the user will be able to modify this configuration at will. We expect normal users to have no need to edit their own GNS zone(s) unless they host services themselves.

@item Where is the per-user GNS database kept?

A: The short answer is that the database is kept at the user's GNUnet peer. Now, a user may run multiple GNUnet peers, in which case the database could be kept at each peer (however, we don't have code for convenient replication). Similarly, multiple GNUnet peers can share one instance of the database --- the "gnunet-service-namestore" can be accessed from remote (via TCP). The actual data can be stored in a Postgres database, for which various replication options are again applicable. Ultimately, there are many options for how users can store (and secure) their GNS database.

@item What is the expected average size of a GNS namestore database?

A: Pretty small. Based on our user study where we looked at browser histories and the number of domains visited, we expect that GNS databases will only grow to a few tens of thousands of entries, small enough to fit even on mobile devices.

@item Is GNS resistant to the attacks on DNS used by the US?

A: We believe so, as there is no entity that any government could force to change the mapping for a name except for each individual user (and then the changes would only apply to the names that this user is the authority for). So if everyone used GNS, the only practical attack of a government would be to force the operator of a server to change the GNS records for his server to point elsewhere. However, if the owner of the private key for a zone is unavailable for enforcement, the respective zone cannot be changed and any other zone delegating to this zone will achieve proper resolution.

@item How does GNS compare to other name systems?

A: A scientific paper on this topic has been published@footnote{https://grothoff.org/christian/ns2018.pdf} and below is a table from the publication. For detailed descriptions please refer to the paper.

@item What is the difference between GNS and CoDoNS?

A: CoDoNS decentralizes the DNS database (using a DHT) but preserves the authority structure of DNS. With CoDoNS, IANA/ICANN are still in charge, and there are still registrars that determine who owns a name.
With GNS, we decentralize the database and also decentralize the responsibility for naming: each user runs their own personal root zone and is thus in complete control of the names they use. GNS also has many additional features (to keep names short and enable migration) which don't even make sense in the context of CoDoNS.

@item What is the difference between GNS and SocialDNS?

A: Like GNS, SocialDNS allows each user to create DNS mappings. However, with SocialDNS the mappings are shared through the social network and subjected to ranking. As the social relationships evolve, names can thus change in surprising ways. With GNS, names are primarily shared via delegation, and thus mappings will only change if the user responsible for the name (the authority) manually changes the record.

@item What is the difference between GNS and ODDNS?

A: ODDNS is primarily designed to bypass the DNS root zone and the TLD registries (such as those for ".com" and ".org"). Instead of using those, each user is expected to maintain a database of (second-level) domains (like "gnu.org") and the IP addresses of the respective name servers. Resolution will fail if the target name servers change IPs.

@item What is the difference between GNS and Handshake?

A: Handshake is a blockchain-based method for root zone governance. Hence, it does not address the name resolution process itself but delegates resolution into DNS after the initial TLD resolution. Not taking sustainablility considerations into account, Handshake could be used as an additional supporting GNS root zone governance model, but we currently do not have such plans in mind.

@item What is the difference between GNS and TrickleDNS?

A: TrickleDNS pushes ("critical") DNS records between DNS resolvers of participating domains to provide "better availability, lower query resolution times, and faster update propagation". Thus TrickleDNS is focused on defeating attacks on the availability (and performance) of record propagation in DNS, for example via DDoS attacks on DNS root servers. TrickleDNS is thus concerned with how to ensure distribution of authoritative records, and authority remains derived from the DNS hierarchy.

@item Does GNS require real-world introduction (secure PKEY exchange) in the style of the PGP web of trust?

A: For security, it is well known that an initial trust path between the two parties must exist. However, for applications where this is not required, weaker mechanisms can be used. For example, we have implemented a first-come-first-served (FCFS) authority which allows arbitrary users to register arbitrary names. The key of this authority is included with every GNUnet installation. Thus, any name registered with FCFS is in fact global and requires no further introduction. However, the security of these names depends entirely on the trustworthiness of the FCFS authority. The authority can be queried under the ".pin" TLD.

@item How can a legitimate domain owner tell other people to not use his name in GNS?

A: Names have no owners in GNS, so there cannot be a "legitimate" domain owner. Any user can claim any name (as his preferred name or "pseudonym") in his NICK record. Similarly, all other users can choose to ignore this preference and use a name of their choice (or even assign no name) for this user.

@item Did you consider the privacy implications of making your personal GNS zone visible?

A: Each record in GNS has a flag "private". Records are shared with other users (via DHT or zone transfers) only if this flag is not set. Thus, users have full control over what information about their zones is made public.

@item Are "Legacy Host" (LEHO) records not going to be obsolete with IPv6?

A: The question presumes that (a) virtual hosting is only necessary because of IPv4 address scarcity, and (b) that LEHOs are only useful in the context of virtual hosting. However, LEHOs are also useful to help with X.509 certificate validation (as they specify for which legacy hostname the certificate should be valid). Also, even with IPv6 fully deployed and "infinite" IP addresses being available, we're not sure that virtual hosting would disappear. Finally, we don't want to have to wait for IPv6 to become commonplace, GNS should work with today's networks.

@item Why does GNS not use a trust metric or consensus to determine globally unique names?

A: Trust metrics have the fundamental problem that they have thresholds. As trust relationships evolve, mappings would change their meaning as they cross each others thresholds. We decided that the resulting unpredictability of the resolution process was not acceptable. Furthermore, trust and consensus might be easy to manipulate by adversaries.

@item How do you handle compromised zone keys in GNS?

A: The owner of a private key can create a revocation message. This one can then be flooded throughout the overlay network, creating a copy at all peers. Before using a public key, peers check if that key has been revoked. All names that involve delegation via a revoked zone will then fail to resolve. Peers always automatically check for the existence of a revocation message when resolving names.

@item Could the signing algorithm of GNS be upgraded in the future?

A: Yes. In our efforts to standardize GNS, we have already modified the protocol to support alternative delegation records.
Naturally, deployed GNS implementations would have to be updated to support the new signature scheme. The new scheme can then be run in parallel with the existing system by using a new record type to indicate the use of a different cipher system.

@item How can a GNS zone maintain several name servers, e.g. for load balancing?

A: We don't expect this to be necessary, as GNS records are stored (and replicated) in the R5N DHT. Thus the authority will typically not be contacted whenever clients perform a lookup. Even if the authority goes (temporarily) off-line, the DHT will cache the records for some time. However, should having multiple servers for a zone be considered truly necessary, the owner of the zone can simply run multiple peers (and share the zone's key and database among them).

@item Why do you believe it is worth giving up unique names for censorship resistance?

A: The GNU Name system offers an alternative to DNS that is censorship resistant. As with any security mechanism, this comes at a cost (names are not globally unique). To draw a parallel, HTTPS connections use more bandwidth and have higher latency than HTTP connections. Depending on your application, HTTPS may not be worth the cost. However, for users that are experiencing censorship (or are concerned about it), giving up globally unique names may very well be worth the cost. After all, what is a "globally" unique name worth, if it does not resolve?

@item Why do you say that DNS is 'centralized' and 'distributed'?

A: We say that DNS is 'centralized' because it has a central component / central point of failure --- the root zone and its management by IANA/ICANN. This centralization creates vulnerabilities. For example, the US government was able to reassign the management of the country-TLDs of Afganistan and Iraq during the wars at the beginning of the 21st century.

@item How does GNS protect against layer-3 censorship?

A: GNS does not directly help with layer-3 censorship, but it does help indirectly in two ways:
Many websites today use virtual hosting, so blocking a particular IP address causes much more collateral damage than blocking a DNS name. It thus raises the cost of censorship.
Existing layer-3 circumvention solutions (such as Tor) would benefit from a censorship resistant naming system. Accessing Tor's ".onion" namespace currently requires users to use unmemorable cryptographic identifiers. With nicer names, Tor and tor2web-like services would be even easier to use.

@item Does GNS work with search engines?

A: GNS creates no significant problems for search engines, as they can use GNS to perform name resolution as well as any normal user. Naturally, while we typically expect normal users to install custom software for name resolution, this is unlikely to work for search engines today. However, the DNS2GNS gateway allows search engines to use DNS to resolve GNS names, so they can still index GNS resources. However, as using DNS2GNS gateways breaks the cryptographic chain of trust, legacy search engines will obviously not obtain censorship-resistant names.

@item How does GNS compare to the Unmanaged Internet Architecture (UIA)?

A: UIA and GNS both share the same basic naming model, which actually originated with Rivest's SDSI. However, UIA is not concerned about integration with legacy applications and instead focuses on universal connectivity between a user's many machines. In contrast, GNS was designed to interoperate with DNS as much as possible, and to also work as much as possible with the existing Web infrastructure. UIA is not at all concerned about legacy systems (clean slate).

@item Doesn't GNS increase the trusted-computing base compared to DNS(SEC)?

A: First of all, in GNS you can explicitly see the trust chain, so you know if a name you are resolving belongs to a friend, or a friend-of-a-friend, and can thus decide how much you trust the result. Naturally, the trusted-computing base (TCB) can become arbitrarily large this way --- however, given the name length restriction, for an individual name it is always less than about 128 entities.

@item How does GNS handle SRV/TLSA records where service and protocol are part of the domain name?

A: When GNS splits a domain name into labels for resolution, it detects the "_Service._Proto" syntax, converts "Service" to the corresponding port number and "Proto" to the corresponding protocol number. The rest of the name is resolved as usual. Then, when the result is presented, GNS looks for the GNS-specific "BOX" record type. A BOX record is a record that contains another record (such as SRV or TLSA records) and adds a service and protocol number (and the original boxed record type) to it.
@end table

@node Error messages,File-sharing<2>,GNU Name System,FAQs
@anchor{faq error-messages}@anchor{405}
@section Error messages



@table @asis

@item I receive many "WARNING Calculated flow delay for X at Y for Z". Should I worry?

A: Right now, this is expected and a known cause for high latency in GNUnet. We have started a major rewrite to address this and other problems, but until the Transport Next Generation (TNG) is ready, these warnings are expected.

@item Error opening @code{/dev/net/tun: No such file or directory}?

A: If you get this error message, the solution is simple. Issue the following commands (as root) to create the required device file
@end table

@example
# mkdir /dev/net
# mknod /dev/net/tun c 10 200
@end example


@table @asis

@item @code{iptables: No chain/target/match by that name.} (when running @code{gnunet-service-dns})?

A: For GNUnet DNS, your iptables needs to have "owner" match support. This is accomplished by having the correct kernel options. Check if your kernel has @code{CONFIG_NETFILTER_XT_MATCH_OWNER} set to either @code{y} or @code{m} (and the module is loaded).

@item @code{Timeout was reached} when running PT on Fedora (and possibly others)?

A: If you get an error stating that the VPN timeout was reached, check if your firewall is enabled and blocking the connections.

@item I'm getting an @code{error while loading shared libraries: libgnunetXXX.so.X}

A: This error usually occurs when your linker fails to locate one of GNUnet's libraries. This can have two causes. First, it is theoretically possible that the library is not installed on your system;
however, if you compiled GNUnet the normal way and/or used a binary package, that is highly unlikely.
The more common cause is that you installed GNUnet to a directory that your linker does not search.
There are several ways to fix this that are described below. If you are @code{root} and you installed to a system folder (such as @code{/usr/local}), you want to add the libraries to the system-wide search path.
This is done by adding a line @code{/usr/local/lib/} to @code{/etc/ld.so.conf} and running "ldconfig".
If you installed GNUnet to @code{/opt} or any other similar path, you obviously have to change @code{/usr/local} accordingly.
If you do not have @code{root} rights or if you installed GNUnet to say @code{/home/$USER/}, then you can explicitly tell your linker to search a particular directory for libraries using the @code{LD_LIBRARY_PATH} environment variable.
For example, if you configured GNUnet using a prefix of @code{$HOME/gnunet/} you want to run:
@end table

@example
$ export LD_LIBRARY_PATH=$HOME/gnunet/lib:$LD_LIBRARY_PATH
$ export PATH=$HOME/gnunet/bin:$PATH
@end example

to ensure GNUnet's binaries and libraries are found. In order to avoid having to do so each time, you can add the above lines (without the "$") to your .bashrc or .profile file. You will have to logout and login again to have this new profile be applied to all shells (including your desktop environment).


@table @asis

@item What error messages can be ignored?

A: Error messages flagged as @code{DEBUG} should be disabled in binaries built for end-users and can always be ignored.
Error messages flagged as @code{INFO} always refer to harmless events that require no action.
For example, GNUnet may use an @code{INFO} message to indicate that it is currently performing an expensive operation that will take some time.
GNUnet will also use @code{INFO} messages to display information about important configuration values.
@end table

@node File-sharing<2>,Contributing<2>,Error messages,FAQs
@anchor{faq file-sharing}@anchor{406}
@section File-sharing



@table @asis

@item How does GNUnet compare to other file-sharing applications?

A: As opposed to Napster, Gnutella, Kazaa, FastTrack, eDonkey and most other P2P networks, GNUnet was designed with security in mind as the highest priority. We intend on producing a network with comprehensive security features. Many other P2P networks are open to a wide variety of attacks, and users have little privacy. GNUnet is also Free Software and thus the source code is available, so you do not have to worry about being spied upon by the software. The following table summarises the main differences between GNUnet and other systems. The information is accurate to the best of our knowledge. The comparison is difficult since there are sometimes differences between various implementations of (almost) the same protocol. In general, we pick a free implementation as the reference implementation since it is possible to inspect the free code. Also, all of these systems are changing over time and thus the data below may not be up-to-date. If you find any flaws, please let us know. Finally, the table is not saying terribly much (it is hard to compare these systems this briefly), so if you want the real differences, read the research papers (and probably the code).


@multitable {xxxxxxxxx} {xxxxxxxxx} {xxxxxxxxx} {xxxxxxxxx} {xxxxxxxxx} {xxxxxxxxx} {xxxxxxxxx} {xxxxxxxxx} {xxxxxxxxx} 
@item

Net-
work

@tab

GNUnet
FS

@tab

One
Swarm

@tab

Nap-
ster

@tab

Direct
Co-
nnect

@tab

Fast
Track

@tab

e-
Donkey

@tab

Gnu-
tella

@tab

Free-
net

@item

Distri
buted
Quer-
ies

@tab

yes

@tab

yes

@tab

no

@tab

hubs

@tab

s
uper-
peers

@tab

DHT
(e
Mule)

@tab

yes

@tab

yes

@item

Multi-
source
Down-
load

@tab

yes

@tab

yes

@tab

no

@tab

no

@tab

yes

@tab

yes

@tab

yes

@tab

no

@item

Econ-
omics

@tab

yes

@tab

yes

@tab

no

@tab

no

@tab

no

@tab

yes

@tab

no

@tab

no

@item

Anon-
ymity

@tab

yes

@tab

maybe

@tab

no

@tab

no

@tab

no

@tab

no

@tab

no

@tab

yes

@item

Lan-
guage

@tab

C

@tab

Java

@tab

often
C

@tab

C++

@tab

C

@tab

C++

@tab

often
C

@tab

Java

@item

Trans-
port
Proto-
col

@tab

UDP,
TCP,
SMTP,
HTTP

@tab

TCP

@tab

TCP

@tab

TCP?

@tab

UDP,
TCP

@tab

UDP,
TCP

@tab

TCP

@tab

TCP

@item

Query
Format
(UI)

@tab

key
words
/ CHK

@tab

fil
ename
/
SHA?

@tab

key
words

@tab

file
name,
THEX

@tab

file
name,
SHA

@tab

file
name,
MD4?

@tab

file
name,
SHA

@tab

s
ecret
key,
CHK

@item

Rout-
ing

@tab

dy-
namic
(in-
direct
di-
rect)

@tab

static
(in-
direct
di-
rect)

@tab

always

direct

@tab

always

direct

@tab

always

direct

@tab

always

direct

@tab

always

direct

@tab

always
in-
direct

@item

Licen-
se

@tab

GPL

@tab

GPL

@tab

GPL
(knap
ster)

@tab

GPL
(Val
knut)

@tab

GPL
(giFT)

@tab

GPL
(e-
Mule)

@tab

GPL
(gtk
-gnut
ella)

@tab

GPL

@end multitable


Another important point of reference are the various anonymous peer-to-peer networks. Here, there are differences in terms of application domain and how specifically anonymity is achieved. Anonymous routing is a hard research topic, so for a superficial comparison like this one we focus on the latency. Another important factor is the programming language. Type-safe languages may offer certain security benefits; however, this may come at the cost of significant increases in resource consumption which in turn may reduce anonymity.

@item Are there any known attacks (on GNUnet's file-sharing application)?

A: Generally, there is the possibility of a known plaintext attack on keywords, but since the user has control over the keywords that are associated with the content he inserts, the user can take advantage of the same techniques used to generate reasonable passwords to defend against such an attack. In any event, we are not trying to hide content; thus, unless the user is trying to insert information into the network that can only be shared with a small group of people, there is no real reason to try to obfuscate the content by choosing a difficult keyword anyway.

@item What do you mean by anonymity?

A: Anonymity is the lack of distinction of an individual from a (large) group. A central goal for anonymous file-sharing in GNUnet is to make all users (peers) form a group and to make communications in that group anonymous, that is, nobody (but the initiator) should be able to tell which of the peers in the group originated the message. In other words, it should be difficult to impossible for an adversary to distinguish between the originating peer and all other peers.

@item What does my system do when participating in GNUnet file sharing?

A: In GNUnet you set up a node (a peer). It is identified by an ID (hash of its public key) and has a number of addresses it is reachable by (may have no addresses, for instance when it's behind a NAT). You specify bandwidth limits (how much traffic GNUnet is allowed to consume) and datastore quote (how large your on-disk block storage is) . Your node will then proceed to connect to other nodes, becoming part of the network.
@end table

@node Contributing<2>,,File-sharing<2>,FAQs
@anchor{faq contributing}@anchor{407}
@section Contributing



@table @asis

@item How can I help translate this webpage into other languages?

A: First, you need to register an account with our weblate system. Please send an e-mail with the desired target language to @email{translators@@gnunet.org} or ask for help on the #gnunet chat on irc.freenode.net. Typically someone with sufficient permissions will then grant you access. Naturally, any abuse will result in the loss of permissions.

@item I have some great idea for a new feature, what should I do?

A: Sadly, we have many more feature requests than we can possibly implement. The best way to actually get a new feature implemented is to do it yourself --- and to then send us a patch.
@end table

@node Index,,FAQs,Top
@unnumbered Index


@printindex ge

@anchor{installing Config-file-format}@w{                              }
@anchor{users/configuration Configuring-the-GNU-Name-System}@w{                              }
@anchor{1d}@w{                              }
@anchor{398}@w{                              }
@anchor{175}@w{                              }
@anchor{developers/tutorial Starting-Peers-Using-the-Testbed-Service}@w{                              }

@c %**end of body
@bye
