Sergey Budaev

Dec 31, 2023

Shit happens: det kommer garantert til å skje hvis dumhet gjentas

Shit happens. Det er en triviell visdom. Ofte er det en direkte konsekvens av en enkelt stupid ting. I mange tilfeller kan lite gjøres for å forhindre det skal skje. Uansett, sannsynligheten antas veldig lav. Katastrofen er uforutsigbar. Det er en ulykke, helt tilfeldig. Ikke sant?

Den bærbare datamaskinens tastatur er dekket av kaffe (eller brus). Å shit...

Det gjelder for en enkelt hendelse. Kanskje en enkelt hendelse av dumhet eller klønete... Men hvis toskeskap gjentas (f.eks. hvis det er en vane) er situasjonen en helt forskjellige. Sannsynligheten av shit som skal skje er nå

P(1|n)=1-(1-p)^n

her er P(1|n) sannsynligheten for at shit skjer minst én gang i en gruppe av n hendelser; hver hendelse har sjansen p (veldig lav!) til å skje, og n er antall hendelser.

For eksempel, hvis sjansen for en singel ulykke er så lavt som 0.01 og antallet dumme handlinger er 365 (bare en gang om dagen i løpet av et år), blir sjansen for at shit skjer i løpet av denne tiden

1-(1-0.01)^365=0.97

Det er nesten sikkert at shit skjer minst én gang i løpet av et år.

  • Drikker du kaffe/brus/smoothie/vin på den bærbare datamaskinen til vanlig? Forberede for å erstatte tastaturet. Det vil skje.

  • Vant til å sende sms mens du kjører? Har du en god forsikring?

  • Løper ofte over veien foran lastebil/buss/bil? Det er på tide å bestille krykker (eller enda kiste) på forhånd.

posted at 10:00  ·   ·  Blog  Wisdom  Q&A

Nov 05, 2023

Telefonen til et barn

Både voksne og barn blir stadig mer avhengige av smarttelefonene sine. Et morsomt begrep for slike rusavhengige er smarttelefonzombie. Men dette er ikke morsomt. Faktisk, smarttelefoner dreper. For eksempel har det vært en økning i antall dødsfall hos barn fordi barna sitter klistret til telefonene sine

Stadig flere barn nå eier smarttelefoner. Nesten alle ungdommer eier en smarttelefon i Norge, Storbritannia, USA og mange andre land. Smarttelefonavhengighet er en verdensomspennende plage (Olson et al., 2022).

Smartphone zombie sign

Imidlertid, forskning viser at smarttelefonavhengighet fører til en rekke alvorlige psykologiske, helse- og velværeproblemer, inkludert nevrologiske lidelser (e.g. Ratan et al., 2022; Achangwa et al., 2023). Mange undersøkelser viser at bruk av smarttelefoner påvirker studentenes akademiske prestasjoner negativt (e.g. Amez & Boert, 2020; Sapci etal. 2021).

Smarttelefonen din er en dedikert spionenhet, men enda mer bekymringsfull er det faktum at apper målrettet mot barn sporer, samler inn personlige data og laster dem opp til ukjente tredjeparter (e.g. Reyes et al., 2018). Men det handler ikke bare om data og reklame. Smarttelefoner kan direkte påvirke fysisk sikkerhet av barn. En russisk studie indikerte at nesten 50% av barna får nye bekjentskaper i sosiale medier og 36% av dem møter disse nye menneskene i virkeligheten etterpå (Kaspersky Lab, 2022).

Vi må løse en avveining mellom behovet for å kommunisere med barna våre, men unngå avhengighet. Så hva er løsningen? Jeg tror det er en kombinasjon av gammel stil (men ikke foreldet!) knapptelefon og et stort nettbrett.

Knappetelefoner er klassiske, men ikke udaterte!

Cool buttonphone

Fordeler med knappetelefon, i tillegg til at det neppe forårsaker avhengighet, inkludere

  • BATTERI fungerer lange eller veldig lange, ingen grunn til å tenke på lading, det er liten risiko for å sitte igjen med en død, utladet telefon i det mest uleilige øyeblikket. Batteriet dør ikke i kulden. Batteriet er avtakbart og kan enkelt skiftes ut. Det er ingen risiko for at barnet vil lade ut batteriet på grunn av intens spilling på telefonen. Det vil ikke skje i verste øyeblikk, for eksempel når han eller hun trenger hjelp fra foreldrene

  • SIKKERHET: det er ingen konstant tilkobling til Internett, viktige data, passord, personlige dokumenter, kredittkortdata lagres ikke på telefonen: det er ingen risiko for lekkasje eller hacking, selv om telefonen er mistet eller stjålet. Plasseringen kan ikke spores og lekkes. Mange hackere og sikkerhetseksperter bær ikke smarttelefoner. Men knappetelefonen tjener sin hovedfunksjon, kommunikasjon, helt perfekt.

  • PRIS telefonen er billig, ikke bry deg om det, den er lett å erstatte hvis den er ødelagt, mistet eller druknet. Men dette er spesielt viktig siden du alltid har telefonen med deg. Barn er ofte uforsiktige og kan bryte ned ting.

  • FYSISK STYRKE: En liten skjerm, sterk telefon, går ikke i stykker med det minste fall, mindre utsatt for vann. Jeg har erfaring med at en telefon ble vasket i vaskemaskin og fortsatte å virke etterpå.

  • FYSISK KNAPPER er fortsatt et av de beste brukergrensesnittene, praktisk å bruke. Du kan konfigurere ett-tasts hurtigvalg. Knapper er også lettere å bruke med hansker i kaldt vær.

  • STØRRELSE en liten telefon passer lett i lommen. Det er bare praktisk.

  • IKKE FORELDET en trykkknapptelefon kan betraktes som en "fysisk app" som ikke blir foreldet og rett og slett alltid fungerer uten å kreve konstante "oppdateringer."

Knappetelefon blir ofte sett på som noe enkelt og kjedelig, selv om det ikke er helt utdatert. Men det finnes noen få moderne, elegante, designertelefoner, for eksempel Punkt (overpriset!).

Nettbrett gir mye bedre brukeropplevelse

Men vi kan ikke frata barna våre internett, spillkommunikasjon med venner og alt annet som en smarttelefon gir! Riktig nok, men det finnes et bedre enhet enn smarttelefon: nettbrett

Tablet for the young

  • STOR SKJERM: nettbrettet har en stor skjerm som gir mye bedre brukeropplevelse for alle bruksområder: Internett, video, spill, tegning, skriving og til og med lydsamtaler. En stor skjerm kan bare ikke sammenlignes med den lille skjermstubb på typisk smarttelefon. Det er mye bedre for alle slags kreative aktiviteter.

  • LAVERE ER BEDRE: Det er ikke så lett å ta en tablett med deg hele tiden. Med andre ord er tilgjengeligheten lavere og det er noen små kostnader forbundet med bruken. Faktisk må du gå til laderen eller et bord, ta nettbrettet og først deretter bruke det. Det er ganske stor forskjell fra smarttelefonen som ofte alltid ligger i lommen. Dette gjør det mindre sannsynlig at du blir avhengig av et nettbrett.

  • STOR OG MERKBAR: Bruk av nettbrett er lettere å legge merke til. Dette gjør det også lettere for foreldrene å overvåke og kontrollere barnas nettbrettbruk.

Konklusjon

Konklusjonen er denne: i stedet for en smarttelefon, er det tilrådelig å gi en grunnleggende knappetelefon til barnet ditt å bære med seg. Men de bør også eie et nettbrett hjemme for å bruke til internett, videoer, spill, studier og alt smarttelefonen som normalt brukes til.

Anbefalt lesing

  • Gabrielsen, Bjørn (2020) Skjermslaver: hva skjermene har gjort med oss, og hva vi kan gjøre med dem. Kagge (ISBN: 9788248925231).
posted at 14:00  ·   ·  wiki  Q&A

Oct 10, 2023

XMPP server on 1-2-3

Messaging continues to be of rise. The new generation is more willing to send texts than to call. Communicating with an instant messenger has an unique advantage over the old good email: you can easily send replies over replies quickly, resulting in a dialogue. But there is a serious problem: many of the instant messengers are commercial products that work such that their "users" are in fact the exploitable resource having no control or choice.

Most corporations are fair providers of various products and services we can buy. But not these "Big Tech" that offer "free applications," including instant messengers. There is, obviously, nothing free on the Earth. Then, if you do not pay, then you are the product not the customer. The Big Tech corporations exploit the "end-users" to suck out private data, often for further resale. Nearly all of these messengers have centralised architecture and the user's account is linked to the telephone number, completely destroying privacy. The link to the telephone number is also very inconvenient because you cannot get several accounts easily, this requires obtaining several mobile subscriptions. It's just illogical, expensive and silly. Centralized architecture dictates that the communication is kept on the corporate servers so theoretically many employees can read messages by abuse.

Some of the products are advertised as end-to-end encrypted. But nearly all of them are closed source so there is no way to check how this is implemented and if and when the service owner can have access to private messages content. Moreover, we have evidence for the opposite. Many so called "end-to-end encrypted" messages are actually read by AI and human contractors. Even if communication is technically end-to-end encrypted, the company owns and fully controls the server, the client application and network traffic, so a man-in-the-middle attack by silently changing certificates is possible (e.g. in the context of lawful intercept, or unlawful abuse). Metadata (technical information information about all aspects of communication, including the addressees, their locations, IP addresses, telephone number etc.) is always accessible to the service. But metadata is often even more informative than the message content. How such metadata is used is typically unclear. The user has no authority here at all.

Nearly all of these messengering systems have closed proprietary protocol. This means that how you use the product is completely controlled by the owner company. The only way to use the product is with the official application. You cannot just choose for yourself which application program to use. This is cardinally different from the email, for example, where you can use the provider's web interface, its mobile app or any of the many available email applications such as Thunderbird or K-9 Mail. With such a third-party application you can easily consolidate several email accounts in one place and easily make use of the functionality the provider does not offer, such as end-to-end encryption. Another major problem is monopoly and lack of interoperability. The "users" (in reality, the exploited resource) are completely restricted to the owner's platform and are unable to communicate with the other (especially competing) platforms (e.g. Facebook to Snapchat) as a way to keep users within the silo. This is as if you were unable to call/send sms across different mobile operators. And this is silly. To break down monopoly, ensure fairer competition and interoperability across the services, the EU has developed the Digital Markets Act (DMA) regulation. This is a big step, but it does not solve many of the problems with centralization, privacy and regular security flaws.

Take back your freedom, privacy and security

So, why use the restricted, inconvenient, monopolistic, insecure and non-private platforms for the trivial task of sending instant messages? There are several ways to configure one's own privately controlled instant messaging system: XMPP and Matrix. XMPP is lightweight and more private, yet covers all the typical instant communication purposes: text, file share and voice. Moreover, XMPP servers are by default federated: it is easy to send messages across the different servers like in the email. There are many different applications for all operating systems and platforms the user can choose.

It is very easy to set up one's own XMPP server for a small group, company, the family or just an individual. You will need two things:

  • Server that will be the central hub for the communication network running 24x7. This can be anything, from a Rasberry PI in a cupboard to a Virtual Private Server (VPS) somewhere in a data centre or just an old PC running in your basement. A small scale VPS useful for an XMPP server can be very cheap, up to a three Euro per month. There exist even cheaper options, such as EUR 6 per year. There are also dedicated search engines to help locate cheap VPS, e.g. LowendBox and ServerHunter. A typical operating system running on the server is Linux (very secure, highly configurable, free and open source).

  • Domain name that needs to be used to connect to the XMPP server. Domain can be registered to the user (e.g. myname.no), which costs about 30 Euro yearly. But a sub-domain can be obtained for free using the https://freedns.afraid.org or similar "free DNS" services. In the later case you might have something like myownchat.mooo.com or myownchat.ptchat.net. It is possible to run the XMPP server purely on IP address even without domain name, but it is much less convenient (e.g. then federation with other servers is lost).

Given you have got a server (VPS or dedicated machine) and the domain, configuring an XMPP server can be done on 1-2-3. There exist several Linux variants (distributives) with different management commands (usually for installing software). I assume Debian Linux is used below (the same commands also work for Ubuntu and other Debian-based Linux systems).

1. Install XMPP server software

Login. When you have got a server of any kind, you need tologin to it, typically with ssh:

ssh debian@1.2.3.4

here the user name on the server is debian and the server ip is 1.2.3.4. Typically, you may need to create the ssh key and upload it to the server to authenticate (refer the server documentation, e.g. this). I assume logging-in is not a problem.

Prepare server. First of all, update the software on the new server

sudo apt update -y && sudo apt-get upgrade -y

Install some useful monitoring and security-enhancing utilities

sudo apt install -y mc htop atop nload nmon tree zip pwgen fail2ban dnsutils iptables-persistent locate unattended-upgrades

Install certbot, a system that manages the TLS certificates for secure connection

sudo apt -y install certbot

Install the ejabberd server, which is is very reliable and light on resources

sudo apt install ejabberd

Firewall. To allow incoming network access to this server by the XMPP clients and also third-party servers, the server needs to configure the firewall rules. This can be done differently in different installations. For example, some VPS may do this using a friendly web interface. The standard Linux firewall is done via iptables.

The XMPP system requires incoming acces via ports 5222, 5223, 5269, 5443, 5280, 3478. To determine the ports refer to the listen section of the XMPP configuration file below.

 sudo iptables -A INPUT -p tcp --dport 5222 -m conntrack --ctstate NEW,ESTABLISHED -j ACCEPT
 sudo iptables -A INPUT -p tcp --dport 5223 -m conntrack --ctstate NEW,ESTABLISHED -j ACCEPT
 sudo iptables -A INPUT -p tcp --dport 5269 -m conntrack --ctstate NEW,ESTABLISHED -j ACCEPT
 sudo iptables -A INPUT -p tcp --dport 5443 -m conntrack --ctstate NEW,ESTABLISHED -j ACCEPT
 sudo iptables -A INPUT -p tcp --dport 5280 -m conntrack --ctstate NEW,ESTABLISHED -j ACCEPT

 # STUN is over udp
 sudo iptables -A INPUT -p udp --dport 3478 -m conntrack --ctstate NEW,ESTABLISHED -j ACCEPT

The port 7777 is used for a proxy for peer-to-peer (bytestream) file transfer. If peer-to-peer file sharing is intended for use, an additional rule should be set allowing incoming connections:

sudo iptables -A INPUT -p tcp --dport 7777 -m conntrack --ctstate NEW,ESTABLISHED -j ACCEPT

To see what firewall rules are in effect issue this:

 iptables -L --line-numbers

It makes sense to save the iptables rules so they are automatically get in effect after reboot

 iptables-save > /etc/iptables/rules.v4

2. Configure your XMPP server

Secure connection certificate. Get a free Let's Encrypt TLS certificate. I assume you have got a free domain myownchat.ptchat.net from https://freedns.afraid.org.

Note that ejabberd can manage (issue and update) TLS certificates on its own, but this needs some configuration as described in the acme configuration option: https://docs.ejabberd.im/admin/configuration/basic/#acme. An advantage of the standalone certificate management system (as here) is that it is slightly less tricky and can easily be used with a web server on the same machine. Why not also configure a web server for a small static web site here? Ejabberd is very lightweight and will happily coexist with many other servers running on the same machine.

 sudo certbot --standalone certonly -d myownchat.ptchat.net

This command will ask a few questions and issue a TLS certificate. This process is done over http so http port 80 must allow incoming connections. If this is not so, use the following command:

 sudo iptables -A INPUT -p tcp --dport 80 -m conntrack --ctstate NEW,ESTABLISHED -j ACCEPT

Do not forget to save iptables rules with the iptables-save as above.

The certificate files are located in /etc/letsencrypt/live/myownchat.ptchat.net/fullchain.pem directory.

For the sake of security, the certificate directories have by default no access to anyone except the admin (root) user. But this precludes the XMPP server ejabberd to access the certificate. This can be easily fixed with the following commands

First, add ejabberd to the root group

sudo adduser ejabberd root

Second, allow access to the certificate directories to the group

sudo chmod g+rx /etc/letsencrypt/live/myownchat.ptchat.net
sudo chmod g+rx /etc/letsencrypt/live
sudo chmod g+rx /etc/letsencrypt/

Configure ejabberd. Once the preparations are done, it is time to configure the ejabberd messaging server. Edit the configuration file (assuming the mcedit text editor is used)

sudo mcedit /etc/ejabberd/ejabberd.yml

This is a long configuration file that may look scary. But in fact only a few changes are required to make the server running with the default options. But note that the indents are important, try to keep them as in the original file.

Any line starting with # is considered a comment, this can be easily used to disable specific options by "commenting them out."

First, set up the host name that is used for the server, it is the same as the domain:

 hosts:
  - myownchat.ptchat.net

Second, configure the location of the TLS certificates that are used by the server:

 certfiles:
  - "/etc/letsencrypt/live/myownchat.ptchat.net/fullchain.pem"
  - "/etc/letsencrypt/live/myownchat.ptchat.net/privkey.pem"

Configure the admin users who can manage the XMPP server:

 acl:
   admin:
      user:
       - ""
       - "myname": "myownchat.ptchat.net"

Then, add configuration for http-file-upload module that will allow file sharing (sending files):

 mod_http_upload:
    put_url: https://@HOST@:5443/upload
    custom_headers:
      "Access-Control-Allow-Origin": "https://@HOST@"
      "Access-Control-Allow-Methods": "GET,HEAD,PUT,OPTIONS"
      "Access-Control-Allow-Headers": "Content-Type"

It is convenient to keep the latest messages on the server, it is done with the "mam" module:

 mod_mam:
   assume_mam_usage: true
   default: always

Ejabberd supports several other communication protocols in addition to XMPP. For example, it also works with MQTT that is typically used for IoT devices. If this functionality is not used, just comment out the MQTT module to disable it.

 # mod_mqtt: {}

The STUN and TURN protocol is mainly used for voice calls and needs the actual IP address of the server (replace with your server IP addfress)

 -
   port: 3478
   ip: "::"
   transport: udp
   module: ejabberd_stun
   use_turn: true
   ## The server's public IPv4 address:
   turn_ipv4_address: "1.2.3.4"

An important issue is wether to allow anonymous registrations of new users. I strongly recommend not allowing this for security reasons. For a small private server, you will normally add users manually and set them initial passwords. Every user can then change password within the client program. So, you need to disable the mod_register by commenting it out:

 # mod_register:
 #   ## Only accept registration requests from the "trusted"
 #   ## network (see access_rules section above).
 #   ## Think twice before enabling registration from any
 #   ## address. See the Jabber SPAM Manifesto for details:
 #   ## https://github.com/ge0rg/jabber-spam-fighting-manifesto
 #   ip_access: trusted_network

Start server! And that's all minimal configuration. Now it's time to start the server:

sudo systemctl start ejabberd

If there are any errors and the server fails to start, Linux logs can be inspected with this command:

sudo journalctl -xe

or logs for only ejabberd:

sudo journalctl -xe --unit ejabberd

Additional stuff. The above is enough to make the XMPP server running for text. If voice is required, you need to configure the DNS as described here: https://www.process-one.net/blog/how-to-set-up-ejabberd-video-voice-calling/. DNS is normally configured using the control panel of the domain registrar.

The TLS certificate that is managed by certbot is updated each 90 days. This is an automatic process, but the ejabberd server must know when certificate is changed. This can be done using the deploy hook. Just create the hook file reloadxmpp.sh (the file name can be anything):

 sudo mcedit /etc/letsencrypt/renewal-hooks/deploy/reloadxmpp.sh

and add the following commands:

 #!/bin/sh
 ejabberdctl reload_config

This file must be executable, so issue this command:

 sudo chmod ugo+x /etc/letsencrypt/renewal-hooks/deploy/reloadxmpp.sh

The last note on the server is that it should be regularly updated for bug fixes and security updates. This is done automatically by installing unattended-upgrades above. Yet, it is a good practice to log in regularly over the ssh, check logs and update the system:

sudo apt update -y && sudo apt-get upgrade -y

3. Configure the XMPP users and client application

Register new users. First, you need to register the XMPP users. The quickest method is to use the command line on the server, the command ejabberdctl has advanced functions.

A secure random password can be generated withy pwgen, e.g. the following generates passwords with 18 symbols:

pwgen 18

It normally generates an array of possible passwords to choose from.

Now, to register the user myname, It is the admin user configured in the main configuration file /etc/ejabberd/ejabberd.yml above.

#                         user   domain               password
sudo ejabberdctl register myname myownchat.ptchat.net pee8chogh9Heel6hei

Other users can be configured similarly. Note that the full user name for XMPP has the same format se email: myname@myownchat.ptchat.net. This is due to the federated nature of both systems: you need to know both the user and the server with whom to communicate.

For this example let's register two additional users:

sudo ejabberdctl register john.dow myownchat.ptchat.net ohyeeLeefo9yief4gu
sudo ejabberdctl register anna.karenina myownchat.ptchat.net hejo7phiy2iFeW9She

Use! The final step is configure the client program on the user's device. The biggest difficulty at this step is the plenty of choice. For any major platform, one can choose any of the many available XMPP client programs. Some email programs, e.g. Thunderbird also support XMPP (although only a limited subset of features). Check out the https://xmpp.org. The configuration for the client is simple:

  • Server: your server, in the example above it is myownchat.ptchat.net

  • User name: your user name. In the example we used above, it can be myname

Note that the option to create new account must NOT be enabled as long as the account has already been created on the sever and the in-band registration (mod_register, see above) is disabled for security.

Pidgin configuration Thunderbird configuration Conversations configuration

Some programs accept the full user name without specifying user and domain separately. Then the user is just myname@myownchat.ptchat.net. If you plan to use the peer-to-peer (bytestream) file transfer (but this is not mandatory), you should also find where the file transfer proxy is configured and set it with the proxy subdomain, for our example it should be proxy.myownchat.ptchat.net. And that is all for basic client configuration.

I recommend the Blabber XMPP application for devices running Android. Yaxim is the best option for minimalists, it is notoriously miniature (only a few megabytes) and works great even on the oldest and weakest devices. Miranda NG is a powerful XMPP client program for Windows. There are also a few web-based clients: https://conversejs.org/ and https://web.xabber.com/ that you can try right away without installing anything.

The final step is to fill the contact list (called roster) with the addresses of the people (or maybe devices, because XMPP can be easily configured for bots accepting commands). Just remember that the address is full name as in email: user@server.domain. One useful option is so called Shared roster groups: then you can configure a group of contacts without the need to add them manually.

Happy chatting!

Further

There are many advanced options and possibilities in ejabberd. Just check the documentation at the official web site: https://www.ejabberd.im/ and documentation https://docs.ejabberd.im/.

There are also a few useful tutorials, e.g. https://www.process-one.net/blog/how-to-move-the-office-to-real-time-im-on-ejabberd/

Oct 23, 2020

Using Subversion to manage Office files

Because Subversion works best (and can track) plain text files, it is not well adapted for versioning normal Microsoft Office or LibreOffice/OpenOffice documents. However, both are actually zipped XML files. Therefore, it is possible both directly (binary) and using flat XML text (full version control/merge support).

Microsoft Office

For Microsoft Office, there are extensions for Subversion: Msofficesvnf, OfficeSVN and MagnetSVN.

Also, TortoiseSVN can use native Microsoft Word "compare versions" tool to check for differences between versions. Check out the Diff-Scripts in the TortoiseSVN installation directory. Note that these scripts are js and can be blocked by corporate or university security policy: ask the IT!

Subversion keywords

Subversion keywords (properties) can be managed in Microsoft Word files using SvnProperties4MSOffice.

For more information see https://gotomation.info/2019/01/svn-version-control-office-documents/.

If special software for adapting Office files is not used, it is recommended to use Microsoft uncompressed XML formats for all outputs. While they take more disk space (because it is unzipped), these are plain text XML, so Subversion treats them very efficiently. Also, svn keywords/tags can be used within the text without any additional tools.

LibreOffice or OpenOffice

For LibreOffice, the easiest way is to use .fodt format for saving the document (instead of .odt or .docx), FODT is a flat XML format. A drawback is that it is unzipped and takes much more disk space. But Subversion does not store all versions of the whole file, it saves effectively differences between the versions. Therefore, there is little or no overhead within the version control system of working with fodt files.

Quite importantly, it is then trivial to add keywords to the fodt file on the svn system. Then, it is easy to include normal svn keywords/tags] such as $Revision 1234$ into whenever needed into the fodt file and it will autoupdate on every commit without any additional tools. But note that the whole tag $Revision 1234$ must have the same formatting (i.e. no bold/italic/other font within and including the $ $ delimiters).

For more information see https://wiki.documentfoundation.org/Libreoffice_and_subversion and https://wiki.documentfoundation.org/Svn:keywords.

Conflicts

To avoid conflicts when several people are working concurrently with svn-tracked files, use svn locks. This is because the files are like binary and cannot be easily merged, unlike normal plain text code. In fact, they could be merged, but do not always expect merge to work as expected because the text file includes complex tags and these may be broken at merge.

It is also difficult to resolve conflicts visually. A useful trick is to set this property on the file: svn propset svn:needs-lock "true" file_name.fodt. Then, any svn update will result this file becoming read-only. To allow editing, file lock must be enabled. This ensures that only one user can edit the file at a time.

How differences between versions can be checked?

Because the Libreoffice files are not just text, checking differences is not trivial. Normal diff tool will result in lots of messy XML differences.

But there is a Linux bash script that helps comparing the files through converting FODT to PDF and then running diffpdf utility:

There is also a Windows/DOS batch script that does this trick:

The script requires diffpdf program that is found in most Linux distributions. A Windows version is open source but id not normally distributed in the binary ".exe" form

How to use diffodt script

  • Compare working copy with the latest revision from svn: diffodt paper.fodt

  • Compare the working document with r9925: diffodt 9925 paper.fodt

  • Compare two specific versions of the document: diffodt 9925 9987 paper.fodt

Integrating Subversion into LibreOffice User Interface

Lo_SVN is a LibreOffice extension that adds a basic Subversion functionality into the LibreOffice interface. Then, basic svn commands are available from the LibreOffice menu.

Lo_SVN

posted at 09:47  ·   ·  Subversion  svn  wiki  Q&A

Nov 20, 2019

How to make an array and initialize it with a sequence of values in Fortran?

How do you make an array and initialize it with a sequence of values? For example, I want a list from 0.25 to 1.5 that is separated with 0.25. In other words I want something similar to seq(0.25,5,0.5) in R.

Equally spaced real array with fixed increment in Fortran

Producing an equally spaced array from V1 to VN with increments ΔV

1. Each of the values in the above vector can be calculated as:

2. The total number of values N in the array ending with a fixed known VN is equal to

3. It is not possible to use a simple piece of code like this to produce real type array in Fortran:

Array = [V1:VN:Incr]

4. Such a construction cannot be used in modern Fortran, even though old versions could accept a similar construction based on implied loop with real type index counter:

real :: r ! Index must be integer in loops!
print *, (r, r=V1,VN,Incr)

5. In modern Fortran standard do loops can only have integer indexing variable. Real indexing in do loops is one of the very few features that had been deleted from the language because it can create lots of problems in float point computations due to finite precision in computer hardware.

The old code might work with modern compilers but it may require special legacy compiler options. The printing-only code as above may still work but would issue a compiler warning.

6. Initialising such equally spaced real type arrays in Fortran implied loops must use the formulas defined in 1. and 2.

# Produce exactly N_VALS values starting from INIT with increments INCR
Array = [( INIT + INCR * (i-1), i=1,N_VALS )]

Where the number of array elements N_VALS is calculated as:

N_VALS = floor( (END - INIT) / INCR + 1 )

N_VALS = ceiling( (END - INIT) / INCR + 1 )

The floor and ceiling functions convert real value to integer as the lower or upper nearest integer; they can give different values when division cannot be done without the remainder

# All values starting from INIT with increments INCR and up to the limit END
Array = [( INIT + INCR * (i-1), i=1,floor((END-INIT)/INCR+1) )]

7. This code does not seem to be a very simple and elegant solution. Ideally, the code should be packaged into a function returning the desired grid array. But such function could not be used in declarations of array parameters. In the later case the one-liner code should be used as above.

Integer arrays

By the way, it is quite easy to produce an integer array, e.g. here is an initialisation for array from 1 to 100 (|1,2,3,...,100|). This can be useful for indexing arrays.

integer, parameter, dimension(*) :: IDX_ARRAY = (/(i,i=1,100)/)

Examples:

A. Produce an array of 10 values starting from 1.0 with increments 0.1

Array = [( 1.0 + (i-1) * 0.1, i=1,10 )]

Result:

1.00000000 1.10000002 1.20000005 1.29999995 1.39999998
1.50000000 1.60000002 1.70000005 1.79999995 1.90000010

Declaration of a parameter array:

real, parameter, dimension(*) :: Array = [( 1.0 + (i-1) * 0.1, i=1,10 )]

However, note that not all compilers may support assumed array size dimension(*) in such array declaration statement, this requires newer Fortran standard (fortunately, recent versions of Intel and GNU Fortran do support assumed size arrays). In such a case declaration must explicitly set the number of array elements:

real, parameter, dimension(10) :: Array = [( 1.0 + (i-1) * 0.1, i=1,10 )]

B. Produce an array of starting from 1.0 to 2.0 with increments 0.145; note that lower value (floor) for the array size is used:

Array = [( 1.0 + 0.145 * (i-1), i=1, floor((2.0-1.0)/0.145 + 1) )]

Result:

1.00000000 1.14499998 1.28999996 1.43499994 1.57999992
1.72499990 1.87000000

C. The same as (B) but the upper value (ceiling) for the array size is used:

Array = [( 1.0 + 0.145 * (i-1), i=1, ceiling((2.0-1.0)/0.145 + 1) )]

Result:

1.00000000 1.14499998 1.28999996 1.43499994 1.57999992
1.72499990 1.87000000 2.01499987

D. In the case B., declarations of parameter arrays can be done like this:

real, parameter, dimension(*) :: Array =                                  &
                 [( 1.0 + 0.145 * (i-1), i=1, floor((2.0-1.0)/0.145 + 1) )]

or, if the compiler does not support assumed size arrays (*), with explicitly calculated array size:

real, parameter, dimension(floor((2.0-1.0)/0.145 + 1)) :: Array =         &
                 [( 1.0 + 0.145 * (i-1), i=1, floor((2.0-1.0)/0.145 + 1) )]

Test program

! This program illustrates how to produce equally spaced real vectors with
! fixed increment in Fortran.
!
! 1. Produce exactly N_VALS values starting from INIT with increments INCR
! Array = [( INIT + INCR * (i-1), i=1,N_VALS )]
!
! 2. All values starting from INIT with increments INCR and up to the limit END
! Array = [( INIT + INCR * (i-1), i=1,floor((END-INIT)/INCR+1) )]
!-------------------------------------------------------------------------------
program spaced_array

    ! Integer counter for implied loops defining vectors.
    integer :: i

    ! Example A. Produce an array of 10 values
    ! starting from 1.0 with increments 0.1
    real, parameter, dimension(*) :: Array1 = [( 1.0 + (i-1) * 0.1, i=1,10 )]


    ! Example B. Produce an array of starting from 1.0 to 2.0
    ! with increments 0.145.
    ! Note that lower value (floor) for the array size is used.
    real, parameter, dimension(*) :: Array2 = &
    [( 1.0 + 0.145 * (i-1), i=1, floor((2.0-1.0)/0.145 + 1) )]

    ! Example C. The same as (B) but the upper value (ceiling) for the
    ! array size is used.
    real, parameter, dimension(*) :: Array3 = &
    [( 1.0 + 0.145 * (i-1), i=1, ceiling((2.0-1.0)/0.145 + 1) )]

    ! Print the sizes of the arrays that were declared above.
    print *, "Array sizes (Array1, Array2, Array3)", &
    size(Array1), size(Array2), size(Array3)

    ! Print the parameter arrays that were declared above.
    print *, "Array1", Array1
    print *, "Array2", Array2
    print *, "Array3", Array3

end program spaced_array

PDF Card

A PDF version of this document is available here: https://budaev.info/images/spaced-array.pdf.

Nov 20, 2019

Jun 13, 2018

Gaussian random numbers in Fortran

The HEDTOOLS tools library has a module for working with random numbers BASE_RANDOM​. There is, in particular, a set of procedures for generating Gaussian random values: ​RNORM and RNORM_ARRAY. These are based on the Kinderman & Monahan, augmented with quadratic bounding curves method (Leva, 1992: algorithm 712, Trans. Math. Software, 18, 4, 434-435​).

I have made a quick comparison of the quality of the Gaussian random numbers generated by the simple Box-Muller method (Box & Muller, 1958​)

Classical (ancient) Fortran code:

normrand_number = dsqrt(-2.*dlog(drand(0)))*dcos(2.*pi*drand(0))

that has been used in TEG codes so far...

and the algorithm 712 as implemented in HEDTOOLS using this test program (see attachment).

Fortran code for the test program:

program test_bm
  use csv_io
  use base_random, rand_x => rand     ! Alias rand() as rand_x() for ifort.
  !use IFPORT, only : rand_x => rand  ! This is the Intel Fortran tweak.

  integer, parameter :: prec = 8, arrsize=100000
  character(len=255), parameter :: filename1="file_01.csv", filename2="file_02.csv"
  real(kind=prec), dimension(arrsize) :: norand1, norand2
  real :: timer_start, timer_end
  !-------------------------------------------------------------------------------
  ! Generating Box-Muller random numbers
  call cpu_time(timer_start)  ! START
  do i=1, arrsize
    norand1(i) = sqrt(-2.*log(rand_x(0)))*cos(2.*pi*rand_x(0))
  end do
  call cpu_time(timer_end)    ! END
  print *, "Box-Muller took: ", timer_end - timer_start
  ! Write random normal data to CSV
  call CSV_MATRIX_WRITE(norand1, filename1)
  !-------------------------------------------------------------------------------

  !-------------------------------------------------------------------------------
  ! Generating based on algorithm 712
  call cpu_time(timer_start)  ! START
  call RNORM_ARRAY(norand2)
  call cpu_time(timer_end)    ! END
  print *, "Alg. 712 took: ", timer_end - timer_start
  ! Write random normal data to CSV
  call CSV_MATRIX_WRITE(norand2, filename2)
  !-------------------------------------------------------------------------------

end program test_bm

Comparison of Box-Muller and A712

The alg. 712 looks slightly faster than the simple Box-Muller transform.

alg. 712 is much better, as the Box-Muller significantly deviates from the normal distribution, alg. 712 does not (using the Anderson-Darling test from the nortest R package).

  # Gaussian random numbers by Box-Muller deviate from the Normal distribution:
  > ad.test(data_bm$X1)
      Anderson-Darling normality test
  data:  data_bm$X1
  A = 581.7, p-value < 2.2e-16
  # Gaussian random numbers by Kinderman & Monahan's A712 do not deviate from the Normal distribution:
  > ad.test(data_a712$X1)
      Anderson-Darling normality test
  data:  data_a712$X1
  A = 0.46975, p-value = 0.2474

So, the alg. 712 procedure implemented in HEDTOOLS should be used instead of the Box-Muller method.

References

  • Box, G. E. P., & Muller, M. E. (1958). A note on the generation of random normal deviates. The Annals of Mathematical Statistics, 29(2), 610–611. ​http://doi.org/10.1214/aoms/1177706645

  • Leva, J. L. (1992). Algorithm 712; a normal random number generator. ACM Transactions on Mathematical Software, 18(4), 454–455. ​http://doi.org/10.1145/138351.138367