Libervia progress note 2021-W31

goffi 4 months ago jabber-xmpp-en SàT Libervia project libre Libervia progress SàT progress

Hello,

last weeks have been exhausting with lot of things at the same time. I've been preparing the release of the 0.8 version, and I wanted to have a couple of thing ready for that, notably a proper way to do translation.

Preparation of 0.8

As you may know, I've implemented a docker integration into Libervia to be able to run relatively easily third party software. This is working, but when testing in the production website I had to put the finishing touches to make it work (notably I've improved HTTP proxy and HTTPS management). I have then created projects and updated a couple of translations files.

As you can now see on https://salut-a-toi.org/, there is a translate menu. Unfortunately I've closed the account creation for the moment, as I have to deal with licensing first. Indeed, nearly all Libervia ecosystem is for now in AGPL v3+, as there are only a few contributors (2 mains one, then only a small patches). The intent was and still is to be sure that the ecosystem stays in an Libre license, but this license may cause trouble in some edge cases, notably if we want to make an iOS frontend (the fruit store is notoriously causing trouble with AGPL licences).

Thus, I'll bring the subject at next general assemble of the "Salut à Toi" association, and see what we should do. One option could be to use FSF's Fiducial Licence Agreement to let the association the possibility to modify the licence as long as it stays a libre one. It would then be possible to add an exception for an iOS frontend. An other would be to avoid totally iOS. Anyway, this need some time and discussions, and if I open translations and get several contributions under AGPL v3+, it may be harder to set this up.

Weblate integrated in the official website

An other time consuming task was to continue with renaming and adapt package names (notably in Pypi). I've used a little trick to redirect legacy packages to the new ones: a new version of each legacy package is a simple setup.py depending on the new package (you can see it e.g. for sat package). I've also put in place a redirection on the Mercurial repositories, using the old repos will redirect to the new ones.

Finally, I've published the 0.8.0 beta 1. You can install it easily with pipx:

  • First install pipx as explained in its documentation
  • Then install the backend with pipx install libervia-backend. You can follow the documentation to see how to configure it and launch it. This will include the CLI and TUI frontends.
  • If you want to test graphical frontends, you'll have to install Libervia Media with hg clone https://repos.goffi.org/libervia-media (assuming that you have Mercurial already installed), then add it into your libervia.conf
  • To install the Desktop frontend, use pipx install libervia-desktop
  • To install the Web frontend, use pipx install libervia-web

Note that the Desktop frontend is still for early adopters, I need to refactor message handling and do some optimisation and stabilisation to make it pleasant to use.

Please send feedbacks either as bug reports/feature requests on the official bug tracker, or on the XMPP chat room at sat@chat.jabberfr.org. I plan to only fix major issues though, as I'm now fully working on 0.9 and I'm focusing mainly on the ActivityPub gateway. However, bug reports/feature requests will be taken into account for 0.9 if not fixed directly in 0.8.

ActivityPub Gateway

After the hard work to move 0.8 close to the finish line has been done, I've started to work on 0.9 and thus the ActivityPub gateway. The first major thing to do was a refactoring of offline storage. Indeed Libervia (or SàT at the time) was started a long time ago with an Async framework (Twisted) long before asyncio even existed. SQLite has been chosen as backend to store data, and a hand made module based on Twisted's adbapi has been created. Despite the rough edges is has been working quite well all this time, and there was even a semi automatic way to update schemas between version. But the whole thing was starting to be difficult to maintain, and the schema update instructions were all kept in the same file.

Fortunately, SQLAlchemy, the most famous SQL databases abstraction with an optional Object Relational Mapper has recently added support for asyncio.

SQLAlchemy is a very powerful and widely used tool, so it has been a quite obvious decision to use it to replace the old system. But to use it, Twisted needs to use an asyncio loop, and Libervia was using GLib loop (or reactor in Twisted terms), due to the dependency to dbus-python.

Dbus-Python is, by its own authors words, not be the best D-Bus binding to use due to unfortunate design decision, so it was the occasion to replace it, and I've moved the backend to TxDBus, a Twisted native D-Bus implementation, which can run with any Twisted reactor. For technical reason, dbus-python is still used for frontends at the moment, but I plan to completely replace it before the end of 0.9 development.

This has required some work, but it was worth it, and after that I could switch to asyncio reactor and implement SQLAlchemy. I've decided to go with the ORM and not the core only as it is opening neat possibilities. I've first made a mapping corresponding to the last version of the database used for Libervia 0.8.

Once SQLAlchemy has been implemented and working, the next obvious decision was to use Alembic, the recommended SQLAlchemy based database migration tools (by the same authors). Thanks to this, migration files are now in separated files, and are really easy to create (Alembic can autogenerate a good part of a script when a migration is needed).

Thanks to all this, I can now easily make changes in database (while in old system I was hesitating due to the work implied). SQLAlchemy also paves the way to support other databases than SQLite. Even if I'm currently sticking with SQLite only, to keep focus, I'll probably add support for PostgreSQL and MariaDB/MySQL at some point.

Once all this work on storage backend has been finalised, the pubsub cache has been implemented.

Pubsub cache is operated transparently for end-user, and stores locally pubsub items (according to internal criteria). This is useful for many reasons: performances of course, but also it allows to do data analyse/indexing, for instance to retrieve all items with some terms (e.g.: to search by categories or hashtags). Pubsub cache is also useful to store data in a component (what is of interest for the ActivityPub gateway), or to store decrypted data (which will be of interest when we'll work on the e2e encryption with pubsub).

I'll pass the implementation details, you'll find the code on the 0.9 bookmark, notably in the pubsub cache plugin, and I've written documentation for developers for some explanations.

New commands has been added to libervia-cli to manage the cache, in particular there is a purge command to delete items according to given criteria, which will save resources and disk space. With it, it's possible to delete only certain types of items (e.g. only blog posts), for all or only some profiles (for instance, only for the AP gateway). You can say a limit (e.g. delete all items which have not been updated for 6 months). Here again, documentation has been written to explain the commands.

While doing all that, I've noticed problem to cache correctly items (because of the flexibility of XMPP Pubsub, it's hard to impossible to say if we can share cache between users), thus I've written a protoXEP (i.e. a proposition for an XMPP Extension Protocol, or XEP) to fix the problem: https://xmpp.org/extensions/inbox/pubsub-caching-hints.html.

I've also submitted a pull request to fix a problem in XEP-0060 (Publish-Subscribe).

While I was a working with standards, I've updated a XEP I've authored a couple of years ago to specify order of items: XEP-0413: Order-By.

Last but not least, while doing the tests for the pubsub cache I've created some helping methods (or fixtures in pytest terms) to help doing unit test.

This concludes the first step of the XMPP-ActivityPub gateway which was, as anticipated, a big one. The following steps should be done more quickly, and work on 0.8 should not be on the way anymore (I plan to publish 0.8 in early September).

That's all for this note, see you next time.

E

edhelas 4 months ago

M

matlag 4 months ago

debacle 4 months ago