Let's talk XMPP - episode 3 - core and extensions (part 2)

goffi 05/08/2015, 07:09 jabber-xmpp-en technique talk_xmpp Libre

This article was initially published in French, and it got some interest, so we started to translate the whole series in English. If you can read French, you can follow the whole series there: http://www.goffi.org/tag/parlons_xmpp

The translation to English was done thanks to: Poulet, Éfrit, Paco, mbarbarosa, Weyfonk and Chteufleur. The whole series is under CC By-SA, and any help would be appreciated to help with the translation of next articles.
(Follow the corresponding link to read the previous episodes).
 
Some functionalities can be added to this central part, hence the X of XMPP (which stands for “eXtensible”).
 
Extensions are written using “XEP” (XMPP Extension Protocol) , following an idea taken from Python (if I am not mistaken). This is why the features supported by a server or a client are identified as XEP-0XXX. Obviously, no need to know this to use a XMPP client, but it can be useful to read an extension (you can find them at  https://xmpp.org/xmpp-protocols/xmpp-extensions/) to understand correctly the utility of a functionality. Two parts are particularly useful without the need to enter in the implementation details: the “abstract” part (summary) in the top that indicates what the XEP is doing, and the “introduction” section (the very first section) that provides further details about the cause of the extension and its use cases.
 
XEP-0045 abstract
 
An XEP can describe a feature, a process (for instance the XEP-0001 explains the life cycle of the XEPs themselves), a historical legacy (related to something created before the XMPP Standard Foundation), information (like good practices), even a joke (yes, there are also lousy jokes in the XEPs!). It can have several statuses, detailed in the XEP-0001. It is interesting to note that a lot of XEPs are “experimental” and so technically not (yet) standards, but often implemented anyway. Such XEPs can be widely modified before they get the “Draft” status and eventually become a standard.
 
Why am I talking about all this? For you to get one thing loud and clear: XMPP is not only about instant messaging!
 
Some interesting examples of extensions:
 
  • Extended Stanza Addressing (XEP-0033): to send messages to several recipients at once, or to do carbon copies, as well as blind carbon copies (as the… yes, yes you see what I mean).
  • Multi-User Chat (XEP-0045): IRC-like group chat.
  • Ad-Hoc Commands (XEP-0050): a generic system to handle any kind of commands. Related to users' permissions, it is a great tool!
  • vcard-temp (Business cards, temporary version, XEP-0054): The legacy way to manage electronic business cards, which are a kind of public profile. A new extension will eventually replace it (XEP-0292).
  • Jabber Search (XEP-0055): used to search for jid, mostly used by directories.
  • Publish/Subscribe (XEP-0060): a big piece, used to publish all kinds of information, and its recovery based on permissions, with a notification system in real time.
  • XHTML-IM (XEP-0071): to publish with an XHTML subset, which enables formatting (to write bold text or include a picture for example).
  • Gateway Interaction (XEP-0100): to manage gateways, i.e. links to external networks.
  • Personal Eventing Protocol (XEP-0163): a kind of simplified Publish/Subscribe system.
  • Jingle (XEP-0166): Peer-to-peer session establishment with a wide range of possible applications, the most well-known being video conference.
  • Serverless Messaging (XEP-0174): unsurprisingly used to communicate without servers.
  • Message Archive Management (XEP-0313): to recover messages or other objects (works with Publish/Subscribe as well) following criteria such as a date. In particular, used to get a conversation history on a server, which is useful to access it from all your clients.
 
Keep in mind that we’ve barely scratched the surface here.
 
Further details will be provided about some of these extensions in future articles.
Please note that extensions can be referred by using short names, for instance “MAM” for “Message Archive Management”. Such names are usually mentioned at the end of the related XEPs, in the “Document Information” appendices.
 
This may lead to situations where clients and servers don’t support the same extensions subsets… How to make both parties agree on which one to use? This problem is solved through another extension (though so essential and widely implemented that it could nearly be considered as a main standard): “Service Discovery”, aka XEP-030, short name: “disco”.
 
The idea is simple: each client, server or component discloses who it is, what it can do, or associated items. A component is a service that plugs itself into a server, see below.
 

who it is

An XMPP address (or jid) can be used by all kinds of elements, be it a server, a client, a gateway, or a bot for instance. The latter is a robot, i.e. a program that automates tasks, and is often used to behave like a client.
 
When a client or another element sees a jid, it may need to know what kind of element is associated to that jid, for instance to render it in a special way in the user interface. This is how clients and gateways are shown in distinct ways in your contacts list.
 
Disco’s identity provides this through a category (“client” or “server” for instance), a type (e.g. a client can be “bot”, “web”, “game”…), and a free name (eg “ejabberd”). A list of categories and their associated types can be found on  https://xmpp.org/registrar/disco-categories.html.
 

what it can do

 
XMPP’s extensibility makes it very feature-rich, therefore knowing what the software we are talking to is able to do is mandatory. Disco provides a list of such supported features, which is why you may see disabled buttons or icons (e.g. for video conference) while talking to someone. This means the client you are talking to indicates that it does not support this feature, or more accurately: it does not reveal that it supports it.
 
These features are linked to namespaces that are mentioned in the related XEPs. A list of XEPs based on namespaces is available here: https://xmpp.org/resources/schemas.
 

associated items

 
Beside its identity and available features, an XMPP entity can have associated elements. They can be servers indicating where to find chat rooms, or gateways to other networks.

 

Let's try

Let’s make this clear with an example. Salut à Toi’s command line interface “jp” can retrieve an entity’s disco using the command “jp info disco”, see below with jabber.fr:
 
 
% jp info disco jabber.fr
Features:
 
http://jabber.org/protocol/commands
http://jabber.org/protocol/disco#info
http://jabber.org/protocol/disco#items
http://jabber.org/protocol/stats
iq
jabber:iq:register
jabber:iq:time
jabber:iq:version
msgoffline
presence
presence-invisible
urn:xmpp:time
vcard-temp
--
Identities:
 
ejabberd (server/im)
--
Extensions:
 
http://jabber.org/network/serverinfo
 
--
Items:
 
[chat.jabberfr.org]
[irc.jabberfr.org]
[j2j]
[presence.jabberfr.org]
[proxy.jabberfr.org]
[users.jabberfr.org]
 
 
As we can see, jabber.fr supports legacy vCards management (vCard-temp, XEP-0054) as well as ad-hoc commands (http://jabber.org/protocol/commands, XEP-0050), and the server (server/im) used for instant messaging is called “ejabberd”. We can also notice that various services are available, one of them being char.jabberfr.org. Let’s take a closer look:
 
% jp info disco chat.jabberfr.org
Features:
 
http://jabber.org/protocol/disco
http://jabber.org/protocol/muc
http://jabber.org/protocol/muc#unique
jabber:iq:browse
jabber:iq:last
jabber:iq:register
jabber:iq:time
jabber:iq:version
urn:xmpp:ping
vcard-temp
--
Identities:
 
Public Chatrooms (conference/text)
--
Items:
 
[...]
JabberFR (13) [jabberfr@chat.jabberfr.org]
[...]
 
 
We see that we are dealing with a chat rooms service (conference/text), which use the protocol “Multi-User Chat” (http://jabber.org/protocol/muc). It is at the moment the only one which is available for groups’ discussions (we will talk about it later). “Items” elements contain rooms’ list, with the name, the occupiers’ number with the corresponding JID. I shorten the list which was very long.
 
In jabber.fr’s information, you may have noticed the “extensions” section, it is the XEP-0128 which allow to extend disco for all kind of information. In the case of ejabberd here, it is to give contact addresses of server, but they aren’t indicated for jabber.fr
 
Below the disco’s window of Gajim:
 
Gajim disco
The first time I used XMPP, with Psi in the early 2000s, I was a bit intimidated by the “service discovery” menu, which shows near directly all information we just saw. This kind of menu is often, at my opinion, showed too abruptly in XMPP clients: using disco directly (that is all except what is automated by the client) is already advanced use.
 
 
Furthermore, I will quickly mention the “software version” extension (XEP-0092) which allows, as indicated by its name, to know to which software we are talking to, and the operating system used. jp allows to show these information with “jp info version”, let’s try on jabber.fr:
 
% jp info version jabber.fr
Client name: ejabberd
Client version: 2.1.13
Operating System: unix/linux 3.2.0
 
We now know the ejabberd version that is used. Useful when we know if a feature is supported or if a bug is fixed. And this work with every entity which implement the XEP, not only servers.
 
% jp info version chat.jabberfr.org
Client name: MU-Conference
Client version: 0.9-svn (Jan 27 2014)
Operating System: Linux 3.2.0-4-amd64
 
That’s it. Knowing about the extensions allows to truly know what we can do with a client or a server. Next time I think I will explain groups’ discussions and see what changed in relation to IRC.

Let's talk XMPP - episode 2 - core and extensions

goffi 30/07/2015, 12:18 jabber-xmpp-en technique talk_xmpp Libre

This article was initially published in French, and it got some interest, so we started to translate the whole series in English. If you can read French, you can follow the whole series there: http://www.goffi.org/tag/parlons_xmpp

The translation to English was done thanks to: Poulet, Éfrit, Paco and mbarbarosa and Weyfonk. The whole series is under CC By-SA, and any help would be appreciated to help with the translation of next articles.

Now that we know what we’re talking about, let’s see what the core of the protocol looks like.
 
Originally, XMPP is defined by 3 (previously 2) RFCs: RFC 6120, RFC 6121, and RFC 6122 (there are others, but these 3 are the main ones). They explain the core of the system, such as sending messages, presence information, statuses and so on.
 
Without getting too much into details related to developers, we can quickly explain that XMPP is based on 3 element types, or “stanzas”:
  • <presence/> to primarily indicate… our presence information (sometimes, we also add other things like our avatars, our capabilities, but let’s stay focused). The presence is broadcast by the server to all the people that you have given permission to (see below).
    We can associate a state and a message to our presence. The state can be one of the following (names can change depending on the client):
    • available (default): you’re online.
    • away: you’re away for a short period
    • chat: you want to talk
    • dnd (do not disturb): you’re occupied (often called “busy” in clients)
    • xa (eXtended Away): you’re away for a long while.
    The status messages allow you to specify your availability in a clear language (for example: “I’m watching a movie, do not disturb”), even though, in practice, it is used for any kind of message (a lot of people use a quote, for example).
  • <message/> A message’s sending of type “send and forget”. There are 5 types of messages::
    • chat: the most well-known message, used for simple instant messaging;
    • error: this one is usually managed directly by client software. It is often shown by means of a notification window in your client, indicating that something wrong occurred;
    • groupchat: as “chat”, but for discussion with multiple people. In practice the difference concerns only developers and it should be transparent in the client;
    • headline: an important message, an announcement. Normally, these messages aren’t kept offline, so if you aren’t connected while the message is sent, you shouldn’t receive it. These messages don’t expect answers. A typical example is an announcement for an imminent server maintenance;
    • normal: a little known yet interesting type. It is a message which generally has a subject, but outside of an instant conversation. Yes, exactly like an email! It is more or less its XMPP equivalent.
    Conversation threads are managed as well, but we’ll talk about it another time.
  • <IQ/> (Info/Query): used for everything that works on a question/answer pattern (it is mandatory to provide an answer to an request, even if you need to reply with an error). This is used mainly by developers, as it serves as a basis for most of the features you need. Using it is like saying "I need to know or edit this information".
 
I don't want to delve into technical details, but it seems essential for a user to know different message and presence types to understand their client.
 
One should note that there is an excellent feature that is largely underused in XMPP: it natively knows how to handle different languages because of its inheritance from XML (xml:lang). In other words, you can specify a normal or status message in French, German and Slovak at the same time. This is a major asset that we intend to use in Libervia.
 
Now let's talk about another essential part: the contact list.
 
It is called "roster" in the XMPP world. You can assign 0, 1 or several groups ("family", "friends", etc.), a name (which is an alias chosen by yourself, not by your contact), and subscription information to every contact you add to your roster.
 
Subscription lets you know whether you have allowed your contact to see your presence information and whether you are allowed to see theirs. Therefore, every time someone adds you to their contact list, your client (e.g. Gajim) asks you whether you wish to grant them access to your presence information. Note that these permissions don't need to be symmetrical: a contact may have been allowed to see your contact information without enabling you to see theirs, and vice versa. It might even be possible that neither of you can see each other's presence information (but I think most clients remove the contact from the roster in that case).
 
Groups are dependent on contacts, not the other way around (it's not a list of groups which contains the contacts): this explains why having an empty group (i.e. without any contact) isn't possible. In my opinion, groups too are underused in the XMPP world, but we'll get back to this.
 
 
That's it for today. I opted to keep the part on extensions for the next post to keep this one lighter.

Let's talk XMPP - episode 1 - the basics

goffi 29/07/2015, 09:42 jabber-xmpp-en technique talk_xmpp Libre

This article was initially published in French, and it got some interest, so we started to translate the whole series in English. If you can read French, you can follow the whole series there: http://www.goffi.org/tag/parlons_xmpp


The translation to English was done thanks to: Poulet, Éfrit, Paco and mbarbarosa. The whole series is under CC By-SA, and any help would be appreciated to help with the translation of next articles.

G'day,
 
Well, as it’s a shame that XMPP is not well known or understood, I decided to start a series of articles to explain what it is.
 
These articles are for a technical audience, but not necessarily for developers, and I hope it will help you to understand the advantages of this protocol and to use your software in a better way.
 
Being a developer of the “Salut à Toi” project, I’ll probably give examples with it quite often.
 
So let’s start with basis.
 
What’s XMPP? It’s a standard message and presence protocol, and extensible (XMPP means « eXtensible Message and Presence Protocol », that is it’s documented and used as a reference (validated by standards organization: the IETF). It allows all software which use it to speak the same language, and so to be interoperable. It’s a libre protocol, that is you can get the documentation and use it for free, without any legal or technical restrictions, and you can improve or modify it (but if you divert and don’t suggest your modifications, you risk to lose the compatibility with other software).
 
XMPP is decentralized and federated, that is you can have servers anywhere, they could (if they don’t forbid it explicitly in their configuration) communicate with each other.
 
It’s a popular protocol, many software allow to use it: for servers, there are Prosody, Ejabberd, Openfire, Tigase, Mongooselm, Metronome, etc. For clients: Gajim, Poezio, Pidgin, Psi, Swift, Jappix, Movim and of course Libervia/Salut à Toi (I let you find the official websites by yourself). A complete list (servers, clients and libraries) is available here: https://xmpp.org/xmpp-software/.
 
If you cannot install your server at home, many public ones are available: in France one can cite those of APINC ((jabber.fr, im.apinc.org, etc), of the Quadrature du Net, etc. A small list (it’s not up-to-date, don’t hesitate to contribute) is available here in French,, if not you can check on https://xmpp.net/directory.php or http://www.jabberes.org/servers (yes, there are a lot!). But I strongly suggest you to install your own server or to go meet a local organization that you can contact easily: on the one hand if you install it by yourself you’ll have a better control on your own data and on the other hand if you want a specific feature you’ll be able to ask admins for an update.
 
Knowing all that, let’s try to install an account.
 
Once the server installed or a public server chosen, you can create an account. You’ll have then an address, a “JID” (Jabber ID, “Jabber” is the former name of the protocol, this name is now owned by a private company and is kept here for historical reasons).
 
This address is on the form of local_name@domain.tld/resource, or in canonical form (or what is called “bare JID”) local_name@domain.tld. For example, mine is goffi@jabber.fr. Does it sound familiar? Yes, it really looks like email addresses!
 
So what is this resource? The resource is linked with the client software that you use to connect: XMPP has been designed from the beginning to allow several clients to connect at once (10 years ago, a few messaging protocols were able to do that, and connecting again from a different place was often resulting in the disconnection of the first client) and this resource is used to identify them. In other words, you have a resource only once connected, and it will be a different one for each client that you are using: if I connect with Libervia, Gajim and Movim, I’ll have 3 different resources.
 
It used to be several strategies to name the resource, sometimes it was used to show the connection location (“home”, “work”), clients often used their names as default value (“gajim”, “psi”). Today, it’s well accepted that it’s better to have a resource that we cannot guess, as somebody can know if you are online or not (even if you don’t want this information to be public) by doing a request to this resource. The best way is to let the server choose the resource for you.
 
Finally, the resource is linked with a priority: it allows to indicate, if several resources are available, which one will get the message. But we’ll see that later.
 
In the next episode, I'll talk about extensions an features discovery
 
That’s it. Let me know if you are interested in this series, if it seems too technical for you, or if you have any comment or fix to suggest. Everything is under the license CC By-SA, so don't hesitate to re-use, share or modify !

Salut à Toi v0.5.1

goffi 18/09/2014, 22:25 GNU-Linux jabber-xmpp-en SàT projet Libre

We are happy to announce the release of Salut à Toi v0.5.1! This version focuses on security and code refactorisation to ease the add of new features and maintenance.

We remind you that SàT is a multi-purposes and multi-frontends XMPP client, mainly developed in Python. The more advanced frontends are Primitivus (console) and Libervia (web). Jp (command line) can be used for administrative tasks. Wix (desktop/WxWidgets) will be deprecated and replaced with Bellaciao (desktop/Qt). An Android frontend is also planned.

overview_libervia

Security

A new parameter has been added to define a password for the SàT profile, it is stored hashed in the database. Its raw version is used to cipher the other passwords - including the one for the XMPP account - which are encrypted. A scheme on the wiki explains how this all works: encryption.

Libervia now handles HTTPS. The administrator can choose the service(s) to enable: HTTP, HTTPS or both.

You can uses OTR for instant messaging end-to-end encryption. Primitivus console interface uses the Python library potr while Libervia is based on the Javascript implementation otr.js. Thus, your encrypted discussions on Libervia are really private because the encryption is done directly by your browser; you may encounter some slowdowns, especially when starting OTR.

http<em />unsecure</em>warning

stdui<em />profile</em>manager<em>primitivus</em>1

stdui<em />profile</em>manager<em>primitivus</em>2

Other additions

You will notice:

  • the add of chat rooms bookmarks;
  • the display of the composing states in chat rooms;
  • a better integration of the ad-hoc commands, e.g. to allow the administration of the server from Primitivus or Libervia;
  • the possibility to erase all your blog posts, change your password or delete your account from Libervia;
  • contextual menus on the roster contacts and the discussion panels;
  • a couple of new stuff concerning the static blog pages.

bookmarks<em />manager</em>primivitus

manage<em />account</em>delete_blogposts

static_blog

Refactorisation

We care to redesign the code when some conception issues are found, or if we imagine a new mechanism which would work better. It is especially important for a project like SàT which is multi-interfaces and still in development. For this version we made several refactorisations concerning:

  • XMPP service discovery;
  • messages sending and reception;
  • textual commands management;
  • contact list management;
  • hierarchical organization of the constants;
  • Libervia's source files hierachy;
  • Primitivus keyboard shortcuts.

These modifications are not of a big interest for the users, but they ease our life, and maybe those of the people who would like to give us a hand! We see them as required first steps to initiate the development of the mobile phone frontend and the add of new features.

We extended the usage of XMLUI, our internal micro-format for describing user interfaces. We use it to manage frontend's dialogs from the backend. The user actions are now better integrated and we will keep on improving it for the next versions.

stdui<em />contact</em>list_primitivus

announcement_admin

adhoc_administration

To be mentioned

The backend is now launched as a Twisted plugin and starts by default in daemon mode, just as Libervia. The initialisation sequence backend / frontends has been improved; this fixes the issues that could occur when SàT and Libervia were launched from a script within a short time. Moreover, we added a .service file for D-Bus to automatically launch the backend when a frontend needs it.

The default paths for the user files are now compliant to the XDG recommendations: configuration file in ~/.config/sat, database and the rest in ~/.local/share/sat. Any previously existing default configuration file will be retrieved and eventually updated.

If the XMPP server address and port are left empty in the connection parameters, the actual values can be retrieved from a DNS SRV record - if one is set for the "domain" part of your JID.

There's a new log system, fully customisable and managing the colors, formatting, filtering and outputs (files, memory...).

logging

logging_primitivus

Administrative aspects

We submitted the file for the creation of the association "Salut à Toi"... which has been accepted without any trouble, positively surprising us. Our working mode is indeed a bit special for a French "1901-law" association: a collegial board with no president, secretary nor treasurer but two co-administrators. We remind you that there is behind this project a desire to fully involve ourselves, and this is not compatible with the pursue of another professional activity. This means, for the developers, the necessity to find a funding source. We will start with testing / adapting our idea of a good funding model for SàT, of course meeting the ethical and moral commitments that are defined by our social contract.

The association memberships are our favorite source of income! We defined in the internal rules several amounts for the annual subscription, between 0 and 100 euros and every one is free to choose. There's no typo! For the persons who would like to support us without being able or willing to contribute financially, that's possible - because moral support is important too. So there's no reason not to join ;-)

We are unfortunately not ready yet to offer you the possibility to make it online. We actually plan to open a bank account for the association at the end of this month, then we will prepare the online form to manage the subscriptions.

Meetings

We attended this year the "Free Software Days" (aka JDLL in French) in Lyon, "Pas Sage en Seine" in Paris and the "Libre Software Meeting" (aka RMLL) in Montpellier - see the links to watch the recorded conferences (in French). We met or saw again some nice people at the stands and during the speeches Goffi made. Many thanks to the organizers of these events, and to Reflets.info which wrote an article about SàT following the last version's release, allowing a larger public to get to know about the project.

We also participated last week to the "XMPP Summit" and its hackaton in Berlin. We met there some XMPP developers, including two with whom we have regular contact, such as Edhelas from Movim. We presented together our issues with the protocol and will suggest new XEPs to standardize some practices (especially regarding Publish-Subscribe and blogging), and push their implementations.

Saturday, September the 27th between 2pm and 5:30pm, Goffi will participate to a radio program from "Ici et maintenant", recorded in Paris and about self-hosting. There will be many guests including two from the Jappix project.

We would like to organize some gatherings via the association, at least once per year as a general assembly and maybe more often. The date and the location haven't been discussed yet.

Please also notice the recent creation of a "users" mailing list for SàT, it completes the chat room and the "dev" mailing list.

And then?

We would like to migrate our own blogs to SàT. The version 0.6 will focus on blogging (in SàT, it is based on a fine access permission system, so you only write to the people you want), pictures storage and tags implementation. These are essentials features and maybe the last important works before the publication of the first public release, which has been a bit delayed and should be stamped 0.7 or 0.8.

Salut à Toi 0.4.0: a new version of the powerful XMPP/Social software

goffi 26/02/2014, 19:04 GNU-Linux jabber-xmpp-en SàT projet Libre

G'day!

We are pleased to announce « Salut à Toi » version 0.4, which comes with many important changes, also regarding the project's life.

Salut à Toi is a powerful communication tool, free and multi-frontends. It offers some features like micro-blogging, instant messaging, an easy way to manage groups permissions (elsewhere called « aspects » or « circles »), games and much more.

Constantly designed with ethical and social thoughts (politically speaking), Salut à Toi (abbr.: SàT) is also bound to its social contract.
Being based on XMPP, SàT is decentralised and compatible with the other projects that use this protocol.

Before starting with the traditional listing of the new features, a few words about the evolution of the project. We are now two persons working full-time on SàT: so did Souliane, a friend for a long time, join me. We would like to live from the project, but we also want to respect our ideals (cf. the social contract) : we are firmly opposed to commercial advertisements, selling data or any similar behaviors.

We decided to organize ourselves as a self-managed entity, probably under the terms of a cooperative and we would like to start it this summer.
There are 3 economical models that we are thinking about:
  • the first would be based on donations. This is a precarious model but it would offer us the largest independance in our decisions taking process.
  • the second, more trendy, is the use of the so-called crowdfunding platforms. It is finally very close to the donations system, but it requires more work to file the applications etc. Moreover, we would need to select with care the platforms to be used.
  • the third one runs more classically, based on services : technical support, specific customizations etc. This is the one model which would leave us the least interdependancy, and the least time to focus on the project itself.
We often discuss about the project organisation, the features to be implemented, etc. We not only agree that privacy is primordial, we also believe that it is not the sole aspect to be studied, and that the impact these kinds of tools have on our lives deserve some special attention. Technology is not neutral, so we are thinking about how to implement and design things, and we are open to discuss (in real life better then behind a screen) ; feel free to launch a discussion with us.

That being said, let's have a look at the change log!
The list is quite long (the last version was more than a year ago), here is a selection:

Micro-blogging

The micro-blogging has been much improved. Comments are now handled, still using the fine access tuning for Pubsub: comments inherit the permissions from the micro-blogs they are replying to. The unibox of Libervia was disturbing many people and it is now optional, the default behavior for typing messages being more classic.

vue_globale.png


Rich texts have been implemented. The system is quite versatile and it's easy to add new syntaxes that would be available for all frontends.

sat_capture_rich.png

When you update some content, you can edit it using the syntax of your choice, even if it has been published with another one. Thus it is possible to edit in Markdown a post that has been originally created in XHTML.

For now, XHTML, Markdown and raw text are available, more should arrive soon (for instance Dokuwiki and Dotclear's syntax).

sat_params.png


The textual (micro)blogs are now recognizing and displaying inline URLs.
In addition to the rich text edition, a WYSIWYG edition (also to be used as a preview feature) is available in Libervia.
sat_wysiwyg.png


And there's still the static blog to display your public messages. It is now completed with an Atom feed. We are getting closer to a full decentralised blog engine and we plan to
move our own blogs to SàT by the next release, so some import tools to migrate from Dotclear and Dokuwiki are to be expected.

sat_static_blog.png

Messaging


It is possible to send messages in carbon copy or blind carbon copy, according to the XEP-0033
sat_extended_addressing.png


The MUC configuration has been implemented (the menu is currently only available in Primitivus)
Chat states notifications have been implemented, letting you know if someone is writing to you or if she/he left.

sat_chat_states.png

one plugin is adding some IRC-like commands e.g. /nick or /join

Command line

thanks to Dal, jp – the command line frontend – has been upgraded to ArgParse, allowing a proper commands refactorisation and confirming jp as being an XMPP “swiss knife”.

it's easy to forward a command or pipe output to a contact, but also to create a remote control (see below) or to transfer a file. Some more commands to publish micro-bloggs, manage your roster etc. are planned.

there's a bundled script to offer you the auto-completion with Zsh
Complètement de jp dans zsh

Misc

important work on the extra-features for the discussion rooms. The Collective Radio (allowing the members of a discussion room to upload some music and listen to it at the same time) has been improved, it also handles the user leaving and return.
sat_radiocol.png

sat_radiocol_invit.pngsat_radiocol_invit_2.png


Ad-Hoc commands are available, allowing to remotely control an entity with any XMPP client

based on the Ad-Hoc commands, a universal remote control has been implemented to offer the possibility to control most of the softwares. A post on my blog is presenting this feature, there's also a short demonstration video featuring VLC

telecommande_-_libervia_-_4.png

command export
: command export : another original feature - it's possible to redirect the input/output of a process to any of your contact (including to another network via a gateway), again the easy way with the fine accesses tuning. There is a demonstration on my blog with an FTP session export to a contact using Gajim

messages notifications are available in Libervia (a contribution from Link Mauve): if someone is talking to you while the tab is hidden, a message should appear on your desktop.

sat_notifs.png

In Libervia: in addition to the drag and drop which was already available, you can click on a contact or a group to open the associated widget. Widgets can now be moved from one tab to another.

considering the importance of the groups in SàT (used a lot to manage the permissions), a roster management user interface has been added to Libervia

sat_roster_manager.png

a mini XMPP discovery service has been started, it's already running on the libervia.org demonstration server.

Under and around the hood:

Some heavy refactorisations have been done to ease the development of future frontends e.g for small screen devices.
The SQLite database is automatically upgraded.

Some work has been done on the testing system, with the installation of a buildbot ( https://buildbot.goffi.org ) which is running a test batch after each commit.
Some SàTellites projects:
  • SàT Pubsub (based on Idavoll) is a Pubsub component for Prosody to manage fine access tuning, it's used for micro-blogging
  • Urwid SàText (based on Urwid) is a widgets library for console display, used by the frontend Primitivus.
  • "Salut" is a XMPP directory, at a very early development stage

Did you know ?

Some reminders about a couple of (hidden) features:

In Libervia you can position your widgets over several columns / lines

SàT comes with an IMAP server that let you use a MUA like Firefox or KMail to read your messages

Future

Beside our projects regarding the cooperative, the following should happen from the technical perspective:
  • an important work concerning the security. Until now, we left on purpose this aspect behind, to not do things by halves and fully focus on it when the time has come. So we plan to integrate HTTPS to Libervia, end to end encryption and password encryption in the database.
  • finish the micro-blogging support and migrate our own blogs to SàT
  • (hash)tags
  • frontends/ports for telephones and tablets, Windows (and Mac?)
  • update the Bellaciao frontend (Qt-based)
  • RSS/Atom feed integration
  • event and calendar management, etc.
We plan to release a public version this summer, it would be the first named version and, as already announced, it will be named « la Commune ».

Credits:

A big thank to contributors:
  • Link Mauve (Emmanuel Gil Peyrot): notifications Twisted plugin for Libervia, style improvments
  • Dal: jp profiles management, ArgParse in jp
  • Robotux (Thomas Preud'homme): locales correction, distribute update
thanks also to packagers (Naha and Robotux for Debian, Link Mauve for Arch, and surely other that we don't know, don't hesitate to contact us).
thanks to Elefantom for using daily the demo, and to give us several feedbacks
thanks to Parinux and Nanterrux to invite me for a talk, and to the JDLL where we'll be again this year.
thanks to the Loop to host the first Salut à Toi hackathon :). Ambiance and location where great.
thanks to Luc and Manu for the interview in the "Symbiose" emission, and for the following discuss.

Come and join us on XMPP MUC room sat@chat.jabberfr.org

A universal remote for your softwares

goffi 18/02/2014, 00:41 GNU-Linux jabber-xmpp-en SàT projet Libre

G'day,
Salut à Toi (SàT) now allows to easily create a remote for most of your softwares, and to share it with permissions management.
Suppose that you are in a flatshare, and you want that your flatmates can change the music of the Amarok running in your living room: the following command is all you need for that:

jp ad-hoc remote -g flatmates -- amarok

where "flatmates" is the name of the group which can control Amarok. You may also want to remote control Okular (e.g. for a talk), or VLC on your phone.
Below is an example with VLC.
telecommande_-_jp_-_1_-_aide.png


The syntax may seem complex in apparence, it's actualy very simple:
telecommande_-_jp_-_2_-_vlc.png
  • " jp ad-hoc remote" tell that we want a remote control
  • "-pgoffi" say thaat we want to use the profile "goffi", but if it's the default profile, it's not needed
  • "-cl" the "-c" means "connect this profile if it's not already done", the "-l" means that we want to loop on commands (after launching a command, we will return to the command menu instead of finishing the ad-hoc session).
  • "-g coloc" is a filter (coloc mean "flatmates" in french): it tells that in addition of your own jid, the jids from the group "coloc" are allowed to use the remote
  • " -- vlc" the "--" is just here to terminate optional arguments, and "vlc" is the name of the software to control.

once the command entered, SàT fill look in D-Bus's Session bus to find buses which have "vlc" in their names, and add commands which don't need arguments. The output show what has been found.

Now, let's see how we can use the remote in Libervia (web interface of SàT):



telecommande_-_libervia_-_1.png

We use the remote with the ad-hoc "commands" menu


telecommande_-_libervia_-_2.png
we enter the jid which exported the remote. So far, we have to enter manualy the full jid, but in the future, a simple right click should be enough.


telecommande_-_libervia_-_3.png
vlc is indeed visible in commands


telecommande_-_libervia_-_4.png
and voilà ! Commands are available. Once again, it's a bit unfriendly at the moment, but it will be easy to replace common names like "play", "stop", "previous", etc with nice icons...


telecommande_-_gajim_-_1.png

telecommande_-_gajim_-_2.png
And it's of course working with every XMPP clients featuring ad-hoc commands, like Gajim or Psi (here Louise can use the remote control created by Goffi because she is in her group "coloc").
There are many advantages: ease of use, it's generic (the software just need to have its commands exported via D-Bus, which is pretty common on freedesktop environments like Kde, Gnome or XFCE), authorisation can be easily managed, and it works everywhere where an XMPP client featuring ad-hoc commands is available. An another proof that XMPP can be used for far more than just instant messaging, and a demonstration of the potential of jp, the command line frontend of SàT...

To conclude, a short demo video (in french).

As usual, to watch the video you can use a recent browser (the last Firefox/Iceweasel for example).
You can also use VLC (version >=1.1 only), by using the "Media/Open Netword Stream" menu and by entering this URL: http://www.goffi.org/videos/pr%c3%a9sentation_S%c3%a0T_7_t%c3%a9l%c3%a9commande_universelle.webm
Last but not least, you can use mplayer:: mplayer "http://www.goffi.org/videos/pr%c3%a9sentation_S%c3%a0T_7_t%c3%a9l%c3%a9commande_universelle.webm"

this video is licenced under Creative Common BY-SA

Export command to a contact (with video)

goffi 22/02/2013, 21:04 projet GNU-Linux technique jabber-xmpp-en Libre SàT

G'day everybody,

I have made a small plugin for fun which export the input/outputs of an Unix command to a contact. The principle is really simple: you enter a command (so far it's not implemented in the frontends, so you'll have to directly use the D-Bus API, throught e.g. D-Feet or qdbus), then the contacts allowed to communicate with it, with options if necessary, and voilà !

There are 2 mains advantages by doing this:

the first one is that you can give access to an interpreter to any of your contacts (I'm doing this with FTP in the video, I have also made tests with bc, ipython or zsh), without using heavy tools like ssh which need to create an access account, a client, an opened port, etc. Of course, it's not elaborated, it's not a terminal, but it can help out. The escape sequences (what bring colors and others things in your interpreter) are not managed yet, and it can scramble your output (I had the case with ipython). I'm thinking about catching them, and convert to color with XHTML-IM (well, a first step would be that SàT manage XHTML-IM :p ).

the second one is to help to write bots really quickly: you just have to make a script with read stdin and react to it. You can make a bot in a couple of minutes with any scripting language (sh, Python, Ruby, etc) or others. And you can directly debug it from a terminal, without the need of an XMPP server to test it. To show you how easy it is, I have made a little test in Python, here what it looks like:

#!/usr/bin/python
#-*- coding: utf-8 -*-
import sys

class QuickBot(object):

    def out(self, msg):
        sys.stdout.write((u"%s\n" % msg).encode('utf-8'))
        sys.stdout.flush()

    def start(self):
        while(True):
            _input = raw_input().decode('utf-8','ignore')
            if _input.startswith('!'):
                args = _input[1:].split()
                try:
                    getattr(self, "cmd_%s" % args[0].encode('ascii').lower())(args[1:])
                except (IndexError, AttributeError, UnicodeEncodeError):
                    pass

    def cmd_salut(self, args):
        self.out(u"à Toi !")

if __name__ == "__main__":
    bot = QuickBot()
    bot.start()

The contact just have to enter !command [arguments] to make it do something (here !salut). To add a new command, you just have to create a new method named cmd_my_command, e.g. cmd_toto will add the command !toto. Easy isn't it ?

I have made a short video to show you this, it's in french but easy to understand even without sound, where I export a FTP server.

I have also made a small web widget for Libervia (it's not pushed on the repository yet). Of course it's limited (because of javascript security restrictions), but it allow to show whatever you want beside your conversations, and to enjoy Libervia's layout management (you'll can, for example, put 4 websites on a grid).

See you soon

As usual, to watch the video you can use a recent browser (the last Firefox/Iceweasel for example).
You can also use VLC (version >=1.1 only), by using the "Media/Open Netword Stream" menu and by entering this URL: http://www.goffi.org/videos/pr%c3%a9sentation_S%c3%a0T_6_export_commande.webm
Last but not least, you can use mplayer:: mplayer "http://www.goffi.org/videos/pr%c3%a9sentation_S%c3%a0T_6_export_commande.webm"

this video is licenced under Creative Common BY-SA

Salut à Toi version 0.3.0 (with demo online !)

goffi 11/01/2013, 00:25 GNU-Linux jabber-xmpp-en SàT projet Libre

Salut à vous !

No news is good news ! The lack of news on the blog lastly is due to the work on Salut à Toi, which is comming in a new 0.3.0 release.

Really a lot of new things in this release, the 0.2 being more that one year old. If you follow the blog, you should have seen some of them like the radio collective, the pipe over XMPP, Primitivus becoming modal, the license change to full AGPL v3 or the group microblogging improvments. Other are still in early stage like the "Bellaciao" Qt frontend, or the quiz game. This year also came the new website, and misc other additions, more or less big. I will not talk again about all of this, there were already posts about most of named features. The installation instructions are at the bottom of this page.

Instead, let's talk about the new online demo. You'll find it at http://www.libervia.org, and you'll just have to click on "register" to easily create a new account. This account come with a Jabber address (Jabber Id or jid), and will let you talk to the world. Don't hesitate to join the SàT room (menu "Groups/join room", then let the default address which is the one of the room: sat@chat.jabberfr.org) to ask your questions or just talk about the project.

Web Interface

The main goal of this demo is to let you try the web interface.


Libervia v0.3.0: sending a message to Louise

You can add widget as you want by drag and drop. This way, by moving the name of a group (well, you'll need to add some contacts of course ;) ) to the center, you'll create a new panel with the microblogs of this group. If you do the same thing with a contact, you'll start a chat. Drag over the "contact" header in the same spirit, and you'll obtain a panel with all microblogs, from all groups. In the future, it should be possible to save the layout to bring it back easily.

To send a message in a discussion, you need to click on it (the title bar become red, like for Louise in the screenshot above). When you enter a text, a green banner certify you that you're talking to one person or in a room.

If you unselect the widget, you can either click on an other one, either click on the gray space under the text entry (it's the status area). If you enter a text without nothing selected, it will be your status message.

For group microbloging, enter the target group by writing your message like this "@group: message". e.g.: "@friends: Salut à Vous !". A message written like "@@: Salut à tous" will be public (readable also for people which have no account, see below). NB: the banner will be blue for a group message, and red to say "careful, this message will be public, either for people you don't even know".

You can also try the collective radio, or the french tarot game (which will hopefuly work correctly).

public microbloging

the public messages page is a the following address: http://www.libervia.org/blog/login (e.g.: http://www.libervia.org/blog/goffi). Here you'll find open microblogs (the ones written like "@@: message") from the person, a bit like identi.ca or Twitter. This page is currently really basic, it will for sure become better with time.

using an e-mail client(MUA)

Like seen previously on the blog (well, in french actually :) ), you can use an e-mail reader like Thunderbird, KMail or Mutt to connect to SàT.


Thunderbird, connected to SàT show a message send by Psi through XMPP

So far it's really basic (it's not yet possible to create directories with IMAP, the account must be logged will using the e-mail client, you can't send message on the traditional e-mail network), but this should become better and better to finally bring a full e-mail server.

This feature show the way to many experiments through XMPP gateways or project like Weboob. Thank to it, we can for example send message through KMail to IRC, SIP or SMS (using gateways like Spectrum's ones). Add Weboob to this (using an XMPP gateway that I have started), and you'll can send messages to website like DLFP.

We can imagine "smart" answer: are you never annoyed by receiving notifications for messages from a website, and to have to connect to it using a web browser, just to read and/or answwer the message ? It will be possible to automate this, so you'll can do everything from your favorite email software.

I have other ideas in head, which should slowly come.

To use this, you have to indicate your jid (which is like login@libervia.org) as your e-mail address, the IMAP and SMTP server to use are www.libervia.org (both) and the respective ports are 10143 and 10125. So far, you have to be logged to the web interface to be able to check you e-mail with your e-mail client, a issue which will of course disappear in the future. Other issue: you can't create an IMAP directory yet, so you'll have to save your messages/draft and other templates locally (in Thunderbird, in the account parameters, you'll have to put "Local Folders" for everything in "Copies & Folders").

Once the configuration is done, you can try with a jabber contact: ask her to sent you a "normal" message (the ones with a subject) with - for example - Psi or Gajim, and answer her with you e-mail client :)

Installation

The installation should be really easy (for people on GNU/Linux: multi-platform is not yet managed; please be patient ;) ), as the project is on pypi. So, a simple "pip install sat" (as root) should be enough. If you're using Arch, a package is already here (thanks to Link Mauve). On a fresh install of Debian, the following command should install "Salut à Toi" (to enter as root):

#apt-get install python-pip python-dev
#pip install sat

Once SàT is installed, you can launch it (by entering "sat"), then use the console ui (by entering "primitivus"). To use Wix, you'll have to install media as shown on the wiki. And don't forget the new website, with useful links and some screenshots: http://sat.goffi.org

Okay, I think this post is long enough (and I need to sleep to be honest), see you soon...

Goffi

New website for Salut à Toi + some news

goffi 02/08/2012, 12:38 GNU-Linux jabber-xmpp-en SàT projet Libre

G'day,



a quick post for the opening of the new Salut à Toi's presentation website: http://sat.goffi.org. The goal was to have something more accessible than a wiki page: you should have quickly a global idea of what the project is with it. It's available in French and in English (any help to add a new language is welcome).

In addition, I was to Linux Software Meeting in Geneva (RMLL in French) in July, and I have had a talk about the project: http://video.rmll.info/videos/salut... . It's in French, I'm thinking about subtitling it soon (again, any help welcome). The meeting was good, I had a common booth with Romain from the friend project Weboob, and I have also seen again people from other project like MOVIM (we were neighbours), Jappix or Newebe.

People seemed interested by "Salut à Toi". More and more people know the project, at least by name. I hope next year the running version will be usable for everybody (not only developers)...

Other news: the project is now fully AGPL v3+ (before it was a mix between GPL v3+ and AGPL v3+) except the data which are Creative Common By-SA.
The Social Contract has been translated to English, thanks to a contribution from Matthieu Rakotojaona.

cheers
Goffi

Fine access tuning for PubSub

goffi 24/06/2012, 20:09 jabber-xmpp-en projet technique SàT Libre

Warning this post is technical

As you may know, "Salut à Toi" allow to do microbloging with groups access, that mean that you can send post to only a group of your roster (e.g. "family" or "friends"). This feature has been available for a year, but was using a dirty hack (which only works with OpenFire); that was mainly a proof of concept

We need a simple permissions system which can be used beyond microblogging, something generic, which can extend PubSub.
Rather than thinking about tons of pubsub nodes with different access model each time (e.g. a family node, an other one for friends, etc), we need a way to fine tune permissions, by items.

For this reason, I started a pubsub component to validate the concept, and which will be used by SàT to manage microblogging. If the idea proove to be working, I'll talk about it on one of the XSF mailing list, to propose it as a standard.

The way it works is simple: we use exactly the same syntax as for nodes access model (http://xmpp.org/extensions/xep-0060.html#accessmodels), and we apply it to items.
The node access model is applied first, then the ones of items (that mean that an item with an open access model won't be available to somebody which is not in the right roster group if the node has a roster access).
The PubSub component parse the item stanza, apply access model, then save it (without the configuration part).
When somebody wants to get node's items, the pubsub component first check that he is granted to access node (by checking the "access model"), then send him only the items that he can access.

Imagine that Louise has a node with 3 items: 1 with an open access (A), 1 with a roster access, allowed for "family" group (B), and one with a roster access, allowed for "friends" and "coworkers" (C).
Pierre, who is a friend of Louise, will have access to items A and C. Louise's brother will access items A and B.

There a 2 main advantages: first this protocol stay standard compliant (with XEP-0060) - so there is no change to do for an XMPP client to get items -, and secondly the clients have only one node to check by entity that they want to follow.

Some words on test implementation: I use the "Idavoll" PubSub component as a basis, which was made by Ralph Meijer*, and I have slightly modified it for my tests. The main difficuly is that currently components have a very limited access to server, and we can't access entities' roster.
To work around the issue, I use an experimental protocol which allow to access a remote roster. Unfortunately, this protocol only permit to access a part of the roster, so I have patched Prosody's implementation to get the full roster.
Finally, I used some simplifications by saying that node creator, owner and publisher are the same. This is often true for microblogging, but is very limiting when we think about XEP-0060's possibilities.

Anyway it works, but it still need works (and a standardisation) before being a proper way to do.

Below is a server dialog example. We have here Pierre and Louise which are friends, Louise being in "friends" group of Pierre. The later post a microblog as follow**:
<iq from="pierre@example.net/SàT" type="set" id="H_77" to="sat-pubsub.example.net">
  <pubsub xmlns="http://jabber.org/protocol/pubsub">
    <publish node="urn:xmpp:groupblog:pierre@example.net">
      <item id="8f532cc8-be1d-11e1-b5d3-00c0ca4f1546">
        <entry xmlns="">
          <title>G'day my friends !</title>
          <id>8f532cc8-be1d-11e1-b5d3-00c0ca4f1546</id>
          <updated>2012-06-24T18:56:36+02:00</updated>
          <author><name>pierre@example.net</name></author>
        </entry>
        <x xmlns="jabber:x:data" type="submit">
          <field var="FORM_TYPE" type="hidden">
            <value>http://jabber.org/protocol/pubsub#item-config</value>
          </field>
          <field var="pubsub#access_model">
            <value>roster</value>
          </field>
          <field var="pubsub#roster_groups_allowed">
            <value>friends</value>
          </field>
        </x>
      </item>
    </publish>
  </pubsub>
</iq>

After receiving this, the server parse configuration data (the "http://jabber.org/protocol/pubsub#item-config" form***), delete this part of the stanza, save the item with the right permissions (roster access, only for "friends" group), and only send a notification to group's members, so it send one to Louise.

The notification is sent like this:

<message to="louise@example.net" from="sat-pubsub.example.net">
  <event xmlns="http://jabber.org/protocol/pubsub#event">
    <items node="urn:xmpp:groupblog:pierre@example.net">
      <item id="8f532cc8-be1d-11e1-b5d3-00c0ca4f1546">
        <entry xmlns="">
          <title>G'day my friends !</title>
          <id>8f532cc8-be1d-11e1-b5d3-00c0ca4f1546</id>
          <updated>2012-06-24T18:56:36+02:00</updated>
          <author><name>pierre@example.net</name></author>
        </entry>
      </item>
    </items>
  </event>
</message>

There is no way to know on which group Louise is in Pierre's roster (this is a classical notification). Even is an XMPP client doesn't manage the per-item access model, it will receive the notifications and will be able to manage them as usual.

I have sub-licenced the component to AGPL V3, you will find the code in my repository (http://repos.goffi.org/sat_pubsub/). I invite XMPP community's members (authors of clients and servers, or others) to use the code, or to comment the idea

Fine permissions tuning are in my opinion a mandatory feature that XMPP really needs, and they will be more and more used by the "Salut à Toi" project. It was one of the last architecture point of SàT, a new release should follow quickly.

Stay connected :)
Goffi

* Ralph Meijer is also a contributor of Twisted and the author of "Wokkel" library, which are in the heart of SàT, in addition of being one of the authors of PubSub XEP. A huge thank to him for its valuable contributions.
** Normally, microbloging use PEP, but component limitation that I mentionned above forced me to use a custom protocol, which doesn't use PEP.
*** I have used a "jabber.org" namespace instead of "goffi.org", as I should have do for an experimental feature, because I need it so Wokkel's PubSub service could parse it.