Hello everybody, time for a new progress note.

This week I've been working with Flatpak. Flatpak is a new way to package applications using containerization, which has pros and cons: for a low overhead the applications can be run on most GNU/Linux distributions, they run in the same reproducible environment, and they have limited access to outside. I prefer using software packaged in my distribution when they're available (better integration, no overhead at all), but Flatpak is a nice way to try quickly something or to have access to software not packaged in our distribution.

About SàT, the main goal is to have super simple installation so people can try Cagou (desktop/mobile interface) easily. I'm also working on packages for other frontends mainly jp (CLI) and Primitivus (TUI), so they can be tested quickly. I hope this can encourage people to try them, and help gain some visibility.

Let's talk technical now. I have already worked on Flatpak nearly 1 year ago, and could launch several frontends using hand made manifests (the files instructing Flatpak how to build your application). I didn't found the time to polish it and it has never been made public.

This time, I wanted to have something as automated as possible: there are several frontends, and dependencies may change, I don't want to have to do everything by hand each time I update the packages.

The main difficulty when building flatpak, is that you have to deal with what is available in the "runtime" (the base of the distribution you'll use), and without network (you can't make internet connection during building process). In the case of Python, that means that you have to download every dependencies (including their own dependencies, etc.) and then install them in the right order (if you have wrong order, a module installation will fail because of missing dependency). This task is more complicated that it seems.

Flatpak ecosystem propose a script to make this task easier: Flatpak PIP Generator. It retrieves dependencies of a Python package, download them, generate the hash and retrieve download URL to prepare a list of "modules" usable in the manifest. The problem is that the list is not ordered according to dependencies, and the JSON produced is not directly usable. I quickly moved to make my own script specially tailored for SàT needs.

The first issue with dependencies is that there is no good way to find them. They are normally declared in the setup.py script which is the standard way to install a Python package, but you can't import this script to analyze it (it would execute its code and the usual if __name__ == '__main__' way to avoid that is never used with setup.py).

Well you have PyPI doing the analyze for you, and if you use its JSON API you can retrieve the mandatory requirements + optional ones using the ['info']['requires_dist'] field of the metadata object. Except that this field is far from always being complete, for instance it's not set for Twisted (I have not investigated for the reasons, maybe because Twisted is using an other file dynamically read). Twisted is far from being the only package without this data, so it's unusable in my case.

I've searched a bit on Internet and made different tries, and at the end the best I've found to reliably get the dependencies used by a Python project is to install it in a virtual environment and to parse the output of pip show <package_name>.
This command output a text with a Requires: line showing the actual requirements used in this project. Thanks to that I could solve the dependencies hell problem and my script is able to order correctly and automatically the modules needed in my manifest. The only modules not managed are setup requirements (requirements needed only for building the modules), but there are only a few of them and it's easy to specify them by hand.

I'm actually doing it in two times: first I download (using pip download) the requirements of the Python package to get the list and version needed, then I install the package in a virtual environment to analyze the output of pip show (pip handle a cache, so packages are not downloaded twice).

So my manifest generation script which I was expecting to do quickly took actually most of my free time this week and it's still not finished. I'm adding other options and I have to faces troubles likes Mercurial not being managed by Flatpak (for this case I have to retrieve myself the repository and upload it manually). I'm also sparing you other troubles (like how to organize the packages, SàT being a specific architecture I've first made my own runtime before being discouraged of doing this on the Flatpak mailing list).

At the end it's a lot of time spent just to make installation easier, and I have to do everything myself. I hope this kind of stories I'm explaining in my notes helps people to understand why it takes so long to develop a software, specially when you're on your free time. I hope also this work will help to popularize the project, and, maybe, bring some helping hands :)

See you soon.

As mentioned in my note 2 weeks ago, I was in holidays and skipped last week note.
It has been a really good thing, a relief, to be out of the screen for a couple of days, I recommend!

So I'm now back to work in my paid job, and working again on SàT on my free time. We are already in the middle of June, and the release is still not done; it's time for the home straight. As you can see if you follow progress notes, I'm working regularly, and moving forward each week, but there is so much to do. Anyway things are getting better now, the new website is online with documentation, the tickets system is working even if still buggy (it's getting better with time), and I start to have feedbacks, meaning that people are testing.

During my holidays I've got a report on ActivityPub about an issue with <link> elements, followed by a bug report. With those informations, I could spot the issue and fix it (it was a problem with caching, the data used in template were not deeply copied, and the list containing links was reused and growing each time the page was accessed). Thanks to Rydroid for reporting the issue.

When you test and see something wrong, the right thing to do is to contact me, and if possible report a bug on https://bugs.goffi.org. If it's easier for you, you can contact me by other means, but the ticket creation is the better way to be sure it's not forgotten.

I've finished and deployed the pubsub experimental option I have talked about in a previous note. I've called it consistent publisher and the goal is to keep original publisher if a node owner or an admin modify an item. The main reason is to maintain permission to edit the ticket for the original publisher, and thanks to that I can now change status of a ticket without replacing publisher.

Overwriting an item is not yet ideal in pubsub: either you follow XEP-0060 and any publisher can overwrite any other publisher item (not great in case of tickets or blog comments), or you restrict overwriting to your own published items but you need to do a workaround like this. I have ideas on how to improve the situation with node options and I'll probably propose a XEP at some point after SàT release.

As you know, I'm focusing on debugging, and I've spotted a couple of annoying bugs that I'll fix in the incoming week. My target is a release before the end of June. I have started work on packaging Cagou for desktop with Flatpak, so it should be easy to install and test on most GNU/Linux distributions, I'll probably talk about that in next week note.

Hello everybody,

during this week I've put online the new official website at https://salut-a-toi.org.

This is an important step for several reasons, let me explain.

The former site (which is currently still available at https://old.salut-a-toi.org, not sure yet how long it will stay here, but it won't be maintained anymore) was made with Django + bootstrap. It was working and not so bad, but outdated (with a really old screenshot on the front page), and it was a pain to maintain it in parallel of the work on SàT
Django is great, powerful, with a big community, but using it only for our presentation website means that we have to follow its evolution, and we have to solve issues from time to time, e.g. when we update the version. Django template engine is similar to Jinja2 that we use, but it's different enough to have to check documentation to maintain it, not a big deal, but some work that we could avoid.

By moving the website to our own Libervia framework, we stay on the exact same stack as SàT itself, so we don't have to follow other frameworks/libraries, it's easier to maintain. Also it's a good use case to test the framework, and make it evolve according to real life use case.
Furthermore, it simplifies our life for some features, like the news available at https://salut-a-toi.org/news, which is an XMPP blog. With Django we were using a iframe displaying a Libervia instance, while now we can directly show the blog.

With the new website I have added a "documentation" section. So far the documentation was mostly on the wiki, but it was not well maintained and a bit messy. Now it's inside the project itself, and written using reStructuredText which is the markup used most of time with Python project. Also we use Sphinx to render it nicely. As a result, we have a better rendered documentation, which is closer to the project (we can even use source code itself and its docstrings) so it should be up-to-date, and we can use the same source to produce things like man pages or a PDF manual.
Having to render the documentation with Sphinx, was the main reason to create our task manager in Libervia Web Framework.

The documentation is far from being complete, but I'm working on it. You can already see how to use jp for blogging or sending end-to-end encrypted message.

Now the TODO list for the release is done. What is left is debugging, writing documentation and packaging… And some rest! I'm currently in holidays so there will probably not be any progress note next week.

Nik: indeed, but I would still have had to convert existing templates, and keep up-to-date with Django evolutions.
Side note: You can simply switch Django to Jinja2 with one setting.

Hi,

this week has seen a bit of work on the website (mostly porting pages from the older one), and again the time to work on the project was short.

When your work on your free time, any moment you take for yourself (a week-end outside, reading a book/newspaper, watching a movie, having a drink, doing sport, etc.), or for the "things which must be done" (cleaning house, administrative things, fixing stuffs, etc.), or simply when you are sick, is time lost for the project, and it goes quickly. I'm living in such spirit for more than 10 years, feeling remorse for everything taking time, and it's not healthy or even sane, not for your nor your loved ones. And even in your project itself, there are a lot of stuff you have do to around it (server administration, website, documentation, communication – like progress notes –, keeping up to date with technologies, etc.) which are not directly improving the software.

I'm explaining this for you, the reader, to understand why it takes so long to develop a project, and why it can get on nerves. You can see now with the progress notes, that the release is nearly ready but still I've had little time to fix issues, and I wont be able to have everything perfectly oiled for this release.

Last week I've mentioned Pubsub, so let's move to this topic.

I won't elaborate on Pubsub itself, just for short Publish/Subscribe is a way to store/get data and to be notified when there are changes. You can see my talk at last FOSDEM (mp4 version) for some use cases. It is notably used in XMPP for blogging, i.e. for the "social network" part of it, and in SàT for events, tickets, merge requests, forums, etc.

During implementation of blogging features in SàT, years ago, the Pubsub implementations in XMPP servers were uneven, and mostly incomplete (because it was not needed for chat which most of the clients were focusing on). There was two options to work around that: choose a server with a working implementation, focus on it and recommend it, that's what have done Jappix and Movim with Metronome, or create a server agnostic component. We have chosen the latter with SàT, and have created "SàT Pubsub", a server independent implementation of PEP/Pubsub, which aims to be the most complete possible.

I've had to write and propose 2 XEPs (XEP-0355 and XEP-0356) to make this possible, but I'll pass the details to avoid making this post too long. Also, this would have hardly be possible without the work of Ralph Meijer which made the base Pubsub implementation in its library Wokkel that we use in top of Twisted, and Idavoll that SàT Pubsub derivate from.

So we now have an independent Pubsub implementation where we can implement everything we need without having to wait for every server to follow some new XEP. And even today when XMPP servers implementations have improved a lot, SàT Pubsub has some unique features, can follow novelties quickly, and is a good place for experimentation. I don't think all server implementations can be on a par with that, at least in the close future.

With that background explained, I'll now explain what I've worked on this week. In standard Pubsub (XEP-0060) when you have the suitable right, you can publish an item (a blog post for instance) with an existing id, this will create a new item and delete the one which was existing previously. But if we have a node where many people can publish, like a blog comment node, that means that anybody can replace the comment of anybody. So if Louise has written something, Pierre can replace the comment, not great.

In some pubsub implementations, which is the case for SàT Pubsub, you can't overwrite an item if you are not the publisher of this item, except if you are admin or owner of the node. That's better regarding blog comments, but there is still an issue: if Louise is owner of a node, and overwrite an item of Pierre, to modify something, she become the new publisher of the item, and Pierre can't modify it anymore. This is notably a problem with tickets: if Pierre report a bug, and Louise changes its status (e.g. because she starts working on the ticket), she becomes the new publisher, and Pierre can't edit it anymore).

To work around that, I've added a new node option which can be activated in configuration, "consistent publisher", which will keep the original publisher even if the item is overwritten (updated) if the entity overwriting the item is owner or admin. I've also prepared an option in database to restrict the maximum number of items in a node, but it's not implemented yet. I can now safely change status of reported issues for SàT without loosing initial publishers.

This week the post was longer than usual, I'll try to keep it shorter next time. As usual, feedbacks are welcome.

Little progress during this week, I've had a busy week-end far from computer (and it's a relief from time to time to be freed from screen).

I'm working mainly on the new website, so I'll explain a bit why doing a new website.

When you have a project, the website is the place people go to know what it is, and SàT is a quite difficult project to explain because it's doing a lot of things, and people get lost easily with it. We are now using the term "communication ecosystem" to explain it, as it seems to fit well: a generic tool for most of your communication needs (chat, file sharing, blog, events, etc.).

We have done current website a while ago using Django, which is one of the most popular web framework on Python (and even outside Python). Django is a great piece of software and has a huge community, but working on the official website with it means that we have to learn, maintain and follow extra technologies. Note that I'm using "we" because we are several people at the association, but I'm currently the only active developer and any extra amount of work is hard to take.

On the other hand, 0.7 version of SàT brings a web framework, based on the ecosystem and so on XMPP. That has many advantage: notably there is nothing new to learn and follow, and it's a concrete use case which allow to test and improve the ecosystem.
Furthermore, the XMPP integration allows to integrate effortlessly things like the official blog, while in the former version we had to include a page of Libervia in a <iframe>.

To summarize, moving the official website on Libervia web framework make the task more easy and is an occasion to test and improve the whole ecosystem.

One of the things that has been improved in the last few months is the integration of a task system, i.e. a way to launch piece of code on certain conditions (in particular if some files are changed), something similar to what you would have with Grunt, for instance, in Javascript world.
This is used to generate SàT documentation, using Sphinx to create HTML page from reStructuredText: when Libervia is launched with --dev-mode, each time the doc is modified, the HTML pages are rebuilt, really handy to check quickly that the syntax gives what you're expecting.

The new official website will be simple, with only a small page of presentation of the software, the documentation, social contract, and few extra pages. Most of it is done, and now I'm porting some pages from Django (the template system is similar to Jinja2 that we use, but it still needs adaptations, and the CSS classes are not the same).

During this, I'm also fixing bugs when I see them or somebody reports them. For instance, I've fixed this week a bug with registration on https://www.libervia.org. If you test any part of the ecosystem (being it desktop or mobile with Cagou, web with Libervia, TUI with Primitivus or CLI with jp), please give feedback either on our chat room (web), or via the SàT/XMPP based ticket system at https://bugs.goffi.org.

I was going to explain some changes I need to do on SàT Pubsub but this post is already long enough, I guess I'll keep it for next week (wow, teasing!).

Hi everybody,

this week has been quite busy with the release of SàT 0.7.0b1. Entering the beta phase means that no more big features will be implemented (small modifications may still be done if necessary), and I'll concentrate on debugging and stability.
The UI of Cagou and Libervia are usable now, even if not perfect, but it was necessary to release to get feedback. Also Python 3 port is the first thing which will be done for next version (0.8), and some features/improvements are waiting for it.

I'm currently focusing on the new website based on Libervia web framework, and on reworking the documentation. The documentation is important to work on, as many people don't know the capabilities of SàT. Furthermore, it can be later used to generate man pages, notably for jp (CLI frontend), and will be available on the new website.

So the plan now is to have the website online a soon as possible, then finish the work started months ago to have SàT (and mostly Cagou) available on flatpak. It would be good to have Cagou available at least on F-Droid, that's another thing to check.

Last but not least I'm planning to work on SàT Pubsub, the server independent PubSub implementation, I'm missing some features before releasing it, but I'll talk about that in a future progress note.

Salut à Toi version 0.7.0 beta 1 has been released. In other words, we are starting the beta phase, the feature are frozen (beside small adjustments) and now we will concentrate on stability, debugging and usability.

As a reminder, "Salut à Toi" is a libre, decentralised, and ethical communication ecosystem. It offers numerous functionalities (instant messaging, end-to-end encryption, blogging, file sharing, events, forums, etc.) and works natively on desktop, mobile devices (Android), on the web, and as well without graphic interface (text interface and command line).

It has been a relief to move over this stage after enourmous amount of work since the last version (0.6.1), released more than 2 and half years ago!

Now, it's your turn: testing, feeding back (on https://bugs.goffi.org or on the project room, also available on the web), and doing your suggestions, it's the time!

If you know Python, I am looking for people who could help me to make the project installable on Windows, Mac OS and *BSD. I am also working on a flatpak package. It would be good to have he web frontend on Yunohost, there again help would be much appreciated (cf. work already started).

Please note that SàT is already available on Debian and probably its derivated (however, beta is not there yet), as well as on Arch Linux (you'll have to use the sat-*-hg packages to test the beta).

The installation instructions are available in the documentation, and I'm working on a new website, which would be more reading friendly, and should be on-line this week.

Here are the links for installation files, and you'll find most of them on Pypy (please note that you'll have to go through "release history" to get beta version)):

I'm concluding with a screenshot of the Android version of Cagou, and note that this blog is also made with this project:

Cagou v0.7.0b1 on Android screenshot

Hello,

this week I've had to fix the way jid are used for file transfer.

To explain you the problem, I need to explain you how communication is done in XMPP (*). To be brief, when an entity (with jid louise@example.net) want to access a component (with jid files.example.net), it sends a "stanza" which is an XML element. There are 3 kinds of stanza (presence, message, and iq).
In our case we use iq stanza, so Louise sends something like:

<iqfrom="louise@example.net"to="files.example.net"type="set">
  […]
</iq>

and the component answers with something like this:

<iqfrom="files.example.net"to="louise@example.net"type="result">
  […]
</iq>

Last week I've explained how I've used the local part in components to access files from somebody else. So far, when creating a stanza, I was using the jid we are connecting with in the from attribute (i.e. where we set the sender of the message).

This is working well with clients, or components if you don't use local part, but when somebody was accessing files from pierre@files.example.net, the jid used by the component is files.example.net, a different one, so this was not working anymore.

To fix that, I've simply had to change the code to use the jid used to contact the components (i.e. the one in the to attribute of the original request) instead.

I've also fixed a MAM (Message Archive Management, XEP-0313) bug where the same messages were requested again on next start up under certain conditions.

Finally, I've completed a generic invitation mechanism to notify somebody when an event or photo album is available. When the invitation is received, the data needed to retrieve the thing is saved in a private "list of interest", which is a pubsub node. With that I can now diplay in Libervia the available photo albums as you can see in the screen capture below.

Libervia Photo Album Vignette

We are nearly there, I still have to complete the guest page (page for people without XMPP account), so they can see events or photo albums, and I can launch the beta. It's a matter of days now.

Thanks for reading, as always feedbacks are welcome.

(*) to learn more about XMPP, you can check my series of articles "Let's talk XMPP". It's originally written in French, and only 4 articles on 10 are translated, help would be welcomed to translate the other ones.

Hello,

this week I've had little time to work on SàT : thanks to the extended easter week-end, I've had a break.

I did manage to work a bit on the file sharing service, still focusing on the photo album. As I've explained last week, file sharing can be used directly from device to device (e.g. retrieving pictures taken on a mobile phone from desktop computer), or with a server-side service.

XMPP allows to create "components" which can be seen as generic plugins to create services on the server (generic in the way that it's not tied to a specific server implementation, but can be used with anyone implementing XEP-0114). Starting with the incoming v0.7, SàT can be used as a component, and file sharing is the first one.

Until now, it was only used to store personal data, so the files could be accessed with the jid of the component. For instance, if your server is example.net your file sharing component may have the jid files.example.net. To access photo albums, we need the path of the album (e.g. /photos/week-end) which is set in the node attribute in XEP-0329, so you can request files.example.net to retrieve your files (i.e. photo album) from /photos/week-end.
Problem: if Louise is connected to files.example.net and wants to access /photos/week-end, she will get her own album. How to do if she wants to access the one of Pierre?

That's were the local part is used, i.e. the part before the @ in a jid. The files.example.net component will receive all traffic sent to any jid finishing with @files.example.net. So when Louise wants to access Pierre files, she can use pierre@files.example.net instead of just the domain name. That's what I've implemented this week. By using just pierre, the component will associate this with its own server (i.e. example.net), and will look for files from pierre@example.net (and check which ones Louise can access).

But what for people with an account outside of example.net, for instance what if file sharing service allows nestor@example.org to store files? Louise can't request nestor@files.example.net because it would get the files of nestor@example.net (note the .net insteaf of .org , it's a different user). To allow to retrieve the files of an external account, we should be able to use a full jid in the request, but it's forbidden to use @ in local part of the jid.

That's where XEP-0106 is used, it allows to escape a jid to do exactly that. I've implemented that so Louise can request nestor\40example.org@files.example.net to retrieve files from nestor@example.org. Don't worry for the complicated address, the end-user should not have to type it herself.

Last but not least, I have also fixed the margin issue with paragraphs and added missing Atom feeds links, it should hopefully be more confortable to read this blog.

To summarize, during this short week I've implemented a way to retrieve files from an other user in file sharing component, I've implemented XEP-0106, I've fixed a small CSS issue on the blog and I've added missing links for Atom feed.

Hello everybody,

this is the time for a second progress report.

After the implementation of the button for translations, I've updated the French version of Libervia, and of the new website I'm currently working on (which is not yet online). For now this is done locally using tools like Gtranslator or Poedit, I plan to install at some point a web app like Weblate or Pootle and if possible to integrate is with SàT/XMPP (at least for authentication) to make contributions easier.

Beside that I've mainly been working on photos album, I want to be able to use it with 0.7.
For a bit of background, photo album is a specialized view of file sharing. I've evaluated two XEP (XMPP Extension Protocol) for that:

  • File Repository and Sharing (XEP-0214) which is based on Pubsub and Collections Nodes (XEP-0248)
  • File Information Sharing (XEP-0329) which is a simple way to share a repository

File sharing is usable either with direct sharing of a repository from a device (e.g. photos from a mobile phone), or with a server component which host files.

I've chosed the second one (File Information Sharing) for now because the Pubsub one is based on Collections which is, in my opinion, currently not usable: permissions from collection nodes are overwritting ones of leaf nodes, and as a result a private node can be accidentaly opened. This needs to be addressed, it's not the first time I discard pubsub collections because of that, I'll try to propose changes to standard after the 0.7 release.

The other reason I've chosen "File Information Sharing" (FIS) is that the Pubsub one handles mirrors and versions, which I feeled overcomplicated at the time. With FIS, I could make an implementation quickly, and I have a working UI now, which is already quite usable (see this blog post to see how it looks like).

But when I'm using the component, I have no way to change access (everything is managed, there is just no interface to change it), so when I put a file on the file sharing component, it stays private for now, not ideal when you want to share a photo album.

So I've worked on a quick way to do it, using ad-hoc commands: one to change file/directory permissions, and one to delete files. It's nearly finished and will be the last thing before starting beta phase.

That said, Pubsub has already everything needed to manage access and subscriptions (to know when new files/photos are available), so I plan to re-evaluate XEP-0214 at a later point, and if I still find it ill-adapted, maybe propose an other option.

I've also noticed a couple of CSS issues on the blog engine (mainly some padding around paragraph would make it easier to read), and I've been noticed that link to Atom feed is missing on the blog. I was planning to fix that this week but could not find the time (I'm working on SàT on my free time). So I hope to do this in the next few days.

That's it for this week. I'm looking forward to start debugging phase, and then finally release.