Sergey Budaev

Oct 23, 2020

Using Subversion to manage Office files

Because Subversion works best (and can track) plain text files, it is not well adapted for versioning normal Microsoft Office or LibreOffice/OpenOffice documents. However, both are actually zipped XML files. Therefore, it is possible both directly (binary) and using flat XML text (full version control/merge support).

Microsoft Office

For Microsoft Office, there are extensions for Subversion: Msofficesvnf, OfficeSVN and MagnetSVN.

Also, TortoiseSVN can use native Microsoft Word "compare versions" tool to check for differences between versions. Check out the Diff-Scripts in the TortoiseSVN installation directory. Note that these scripts are js and can be blocked by corporate or university security policy: ask the IT!

Subversion keywords

Subversion keywords (properties) can be managed in Microsoft Word files using SvnProperties4MSOffice.

For more information see https://gotomation.info/2019/01/svn-version-control-office-documents/.

If special software for adapting Office files is not used, it is recommended to use Microsoft uncompressed XML formats for all outputs. While they take more disk space (because it is unzipped), these are plain text XML, so Subversion treats them very efficiently. Also, svn keywords/tags can be used within the text without any additional tools.

LibreOffice or OpenOffice

For LibreOffice, the easiest way is to use .fodt format for saving the document (instead of .odt or .docx), FODT is a flat XML format. A drawback is that it is unzipped and takes much more disk space. But Subversion does not store all versions of the whole file, it saves effectively differences between the versions. Therefore, there is little or no overhead within the version control system of working with fodt files.

Quite importantly, it is then trivial to add keywords to the fodt file on the svn system. Then, it is easy to include normal svn keywords/tags] such as $Revision 1234$ into whenever needed into the fodt file and it will autoupdate on every commit without any additional tools. But note that the whole tag $Revision 1234$ must have the same formatting (i.e. no bold/italic/other font within and including the $ $ delimiters).

For more information see https://wiki.documentfoundation.org/Libreoffice_and_subversion and https://wiki.documentfoundation.org/Svn:keywords.

Conflicts

To avoid conflicts when several people are working concurrently with svn-tracked files, use svn locks. This is because the files are like binary and cannot be easily merged, unlike normal plain text code. In fact, they could be merged, but do not always expect merge to work as expected because the text file includes complex tags and these may be broken at merge.

It is also difficult to resolve conflicts visually. A useful trick is to set this property on the file: svn propset svn:needs-lock "true" file_name.fodt. Then, any svn update will result this file becoming read-only. To allow editing, file lock must be enabled. This ensures that only one user can edit the file at a time.

How differences between versions can be checked?

Because the Libreoffice files are not just text, checking differences is not trivial. Normal diff tool will result in lots of messy XML differences.

But there is a Linux bash script that helps comparing the files through converting FODT to PDF and then running diffpdf utility:

There is also a Windows/DOS batch script that does this trick:

The script requires diffpdf program that is found in most Linux distributions. A Windows version is open source but id not normally distributed in the binary ".exe" form

How to use diffodt script

  • Compare working copy with the latest revision from svn: diffodt paper.fodt

  • Compare the working document with r9925: diffodt 9925 paper.fodt

  • Compare two specific versions of the document: diffodt 9925 9987 paper.fodt

Integrating Subversion into LibreOffice User Interface

Lo_SVN is a LibreOffice extension that adds a basic Subversion functionality into the LibreOffice interface. Then, basic svn commands are available from the LibreOffice menu.

Lo_SVN

posted at 09:47  ·   ·  Subversion  svn  wiki  Q&A

Apr 03, 2020

Is ​Zoom safe to use? Is the company marketing and other information correct and can be trusted?

Zoom privacy and security problems

Zoom has demonstrated significant negligence with respect to cybersecurity. Additionally, the company has shown aggressive marketing campaigns and was caught at providing false information to its end users.

  • Zoom aggressively forces the user to download and install native application rather than use web browser for videoconferencing even though videoconferences will work in the web browser. This is a little suspicious. Browser-based conferences are more convenient for an occasional user and is safer due to browser sandboxing of network applications.

  • Serious security deficiency on the Apple Mac platform allowing any unauthorized remote attacker to activate web camera, connect to a conference and execute denial-of-service attack. Zoom tried to ignore and deliberately hide information about the very serious security vulnerability and was slow to fix it. ​ See here for more details, ​ and here (technical information is ​ here and ​ here). Zoom management response seem to point to quite irresponsible corporate culture.

  • More recently it appeared that Zoom was sending users' data to Facebook servers without the user's consent. This is now fixed. See ​ Vice paper ​ and this follow-up.

  • Zoom was caught at providing false and misleading information that the videoconference has "end-to-end" encryption while this was not so. Check out this. The explanation for this provided by Zoom is unsatisfactory.

  • Zoom had a serious security vulnerability that could lead to user password leak in Microsoft Windows. ​ See here for details.

  • Zoom has a strange privacy policy that, even though states that "privacy is very important to us," requires quite large collection of private user's information. There is little explanation about to why this information is collected. Unlike many other similar companies, Zoom does not release transparency report(s). See here: ​https://zoom.us/privacy

  • Electronic Privacy Information Centre has filed complaint to FCC

    • alleging that the videoconferencing company Zoom has committed unfair and deceptive practices in violation of the FTC Act. According to EPIC, Zoom intentionally designed its web conferencing service to bypass browser security settings and remotely enable a user's web camera without the knowledge or consent of the user.

  • See more details here

  • There is a growing concern on the privacy deficiency in Zoom, for more details see ​this and ​ this. Also see The Guardian.

  • Recently SpaceX has banned Zoom because of privacy concerns, see here for details.

  • Zoom has close links with China. Even though the intellectual property, management and marketing are based in the USA, many if not most developers and engineers are bsed in China (see ​Form S-1 registration statement). This can potentially lead to serious privacy and cybersecurity issues, given the Chinese regime tightening of Internet regulation (censorship, privacy etc.). One example is ​MLPS 2.0 legislation, 2019 mandating China residents and any foreign companies unrestricted access to user data. (In China, Zoom has a network of agents acting under different names but using the same platform. )

Updates: More on Zoom problems

How to increase privacy and security of using Zoom on Linux

Sandboxing. On the Linux platform, one solution is always to run Zoom videoconferencing software only in a limited sandbox. Then, Zoom client would not have access to user's files and other processes running on the system.

  • Update: This recipe works for Zoom v. 3.5.361645.0301, but not for some later versions, e.g. 3.5.374815.0324, see update below on this.

Disable any unauthorized update/upgrade of Zoom client. Do not install Zoom software via the standard reopository. Use static tar.gz archive instead. Select Other Linux OS for installation. Uncompress the static distribution in a safe directory. Disadvantage of this is that update is only manual, check out Zoom web site for new releases and read changelog. But advantage is that zoom cannot silently install any unauthorized update or software on the system.

It also makes sense to register at Zoom with the institutional email but separate password, so Zoom does not use the main institutional login (SSO login). This might help against credentials leak in case of Zoom software vulnerability. Using the institutional email to register would ensure Zoom is registered as "licensed."

Install firejail sandboxing. ​https://firejail.wordpress.com/:

sudo apt install firejail.

  • Firejail is a SUID program that reduces the risk of security breaches by restricting the running environment of untrusted applications using Linux namespaces and seccomp-bpf. ... Firejail can sandbox any type of processes: servers, graphical applications, and even user login sessions. The software includes security profiles for a large number of Linux programs: Mozilla Firefox, Chromium, VLC, Transmission etc. To start the sandbox, prefix your command with “firejail.”

Make a configuration file for Zoom in .config/firejail/. Here is the configuration file named as the main Zoom run executable: ZoomLauncher.profile (given the running executable is ZoomLauncher):

# Note: to delete all firejail profiles for all local trusted apps
#  run sudo firecfg --clean
# ----------------------------------------------------------------
# Duplication of zoom configs in noblacklist and whitelist
# sections fixes login credentials no save problem:
noblacklist ${HOME}/.config/zoomus.conf
noblacklist ${HOME}/.zoom
include /etc/firejail/disable-common.inc
include /etc/firejail/disable-devel.inc
include /etc/firejail/disable-programs.inc
include /etc/firejail/disable-passwdmgr.inc
whitelist ${HOME}/bin/zoom
whitelist ${HOME}/.config/zoomus.conf
whitelist ${HOME}/.zoom
whitelist ${HOME}/.cache/zoom
whitelist ${HOME}/downloads
include /etc/firejail/whitelist-common.inc
caps.drop all
netfilter
nodvd
nonewprivs
noroot
notv
protocol unix,inet,inet6
seccomp
private-tmp
# Needed for latest versions of Zoom and perhaps certain other Qt/QML apps
env QML_DISABLE_DISK_CACHE=1

Now Zoom client can be started from the firejail sandbox:

firejail /path_to_safe_install_location/bin/zoom/ZoomLauncher

To make it possible to use standard graphical menus, one need to make a zoom.desktop startup file in the user's directory .local/share/applications. The Exec entry of the file must include the firejail-based startup:

[Desktop Entry]
Name=Zoom Desktop [Jailed]
GenericName=Zoom videoconferencing
Comment=Zoom Desktop Client jailed
Exec=firejail /path_to_safe_install_location/bin/zoom/ZoomLauncher %f
Icon=zoom.png
Terminal=false
Type=Application
Categories=Network;Internet;Education;Qt;
X-SuSE-translate=false

Firejail caveats

Firejail can start serving all user's applications in its jail, which is often too restrictive (e.g. settings are not saved).

  • To force reconfiguring all application to run in firejail do (do not do this if you are unsure) this:

    sudo firecfg

  • To disable configuring all local applications to run in jail, do this:

    sudo firecfg --clean

  • Do this (sudo firecfg --clean) if you have problems starting applications after installing firejail.

  • To check if an application is by default starting in a jail, run it from the terminal. If terminal shows several lines like Reading profile /etc/firejail/disable-common.inc then the application runs in a jail.

A newer version of Zoom client (3.5.374815.0324) refused to run in a jailed environment and hanged.

A workaround for running recent Zoom in jail:

add the below line env QML_DISABLE_DISK_CACHE=1

to the firejail config file.

  • QML_DISABLE_DISK_CACHE Disables the disk cache and forces re-compilation from source for all QML and JavaScript files. (from QML Documentation)

How to increase privacy and security of using Zoom on Microsoft Windows

Here is a link on sandbox in Windows 10: How to use Windows sandbox.

I have not tested how this works.

Android sandbox

For Android, one solution is to use the open source ​Shelter application, then mobile Zoom can run in a secure container.

I have been running several programs that I do not like to give access to my data within Shelter. It works fine for me.

Advantages:

  • Contacts (address book) are not leaked to Zoom if a separate address book is used within shelter

  • All apps can be frozen to avoid them run all the time at the background, this reduces the chances of data leaks as well as battery drain. Freezing can be done automatically, after timeout.

Links

posted at 11:10  ·   ·  privacy  security

Nov 20, 2019

How to make an array and initialize it with a sequence of values in Fortran?

How do you make an array and initialize it with a sequence of values? For example, I want a list from 0.25 to 1.5 that is separated with 0.25. In other words I want something similar to seq(0.25,5,0.5) in R.

Equally spaced real array with fixed increment in Fortran

Producing an equally spaced array from V1 to VN with increments ΔV

1. Each of the values in the above vector can be calculated as:

2. The total number of values N in the array ending with a fixed known VN is equal to

3. It is not possible to use a simple piece of code like this to produce real type array in Fortran:

Array = [V1:VN:Incr]

4. Such a construction cannot be used in modern Fortran, even though old versions could accept a similar construction based on implied loop with real type index counter:

real :: r ! Index must be integer in loops!
print *, (r, r=V1,VN,Incr)

5. In modern Fortran standard do loops can only have integer indexing variable. Real indexing in do loops is one of the very few features that had been deleted from the language because it can create lots of problems in float point computations due to finite precision in computer hardware.

The old code might work with modern compilers but it may require special legacy compiler options. The printing-only code as above may still work but would issue a compiler warning.

6. Initialising such equally spaced real type arrays in Fortran implied loops must use the formulas defined in 1. and 2.

# Produce exactly N_VALS values starting from INIT with increments INCR
Array = [( INIT + INCR * (i-1), i=1,N_VALS )]

Where the number of array elements N_VALS is calculated as:

N_VALS = floor( (END - INIT) / INCR + 1 )

N_VALS = ceiling( (END - INIT) / INCR + 1 )

The floor and ceiling functions convert real value to integer as the lower or upper nearest integer; they can give different values when division cannot be done without the remainder

# All values starting from INIT with increments INCR and up to the limit END
Array = [( INIT + INCR * (i-1), i=1,floor((END-INIT)/INCR+1) )]

7. This code does not seem to be a very simple and elegant solution. Ideally, the code should be packaged into a function returning the desired grid array. But such function could not be used in declarations of array parameters. In the later case the one-liner code should be used as above.

Integer arrays

By the way, it is quite easy to produce an integer array, e.g. here is an initialisation for array from 1 to 100 (|1,2,3,...,100|). This can be useful for indexing arrays.

integer, parameter, dimension(*) :: IDX_ARRAY = (/(i,i=1,100)/)

Examples:

A. Produce an array of 10 values starting from 1.0 with increments 0.1

Array = [( 1.0 + (i-1) * 0.1, i=1,10 )]

Result:

1.00000000 1.10000002 1.20000005 1.29999995 1.39999998
1.50000000 1.60000002 1.70000005 1.79999995 1.90000010

Declaration of a parameter array:

real, parameter, dimension(*) :: Array = [( 1.0 + (i-1) * 0.1, i=1,10 )]

However, note that not all compilers may support assumed array size dimension(*) in such array declaration statement, this requires newer Fortran standard (fortunately, recent versions of Intel and GNU Fortran do support assumed size arrays). In such a case declaration must explicitly set the number of array elements:

real, parameter, dimension(10) :: Array = [( 1.0 + (i-1) * 0.1, i=1,10 )]

B. Produce an array of starting from 1.0 to 2.0 with increments 0.145; note that lower value (floor) for the array size is used:

Array = [( 1.0 + 0.145 * (i-1), i=1, floor((2.0-1.0)/0.145 + 1) )]

Result:

1.00000000 1.14499998 1.28999996 1.43499994 1.57999992
1.72499990 1.87000000

C. The same as (B) but the upper value (ceiling) for the array size is used:

Array = [( 1.0 + 0.145 * (i-1), i=1, ceiling((2.0-1.0)/0.145 + 1) )]

Result:

1.00000000 1.14499998 1.28999996 1.43499994 1.57999992
1.72499990 1.87000000 2.01499987

D. In the case B., declarations of parameter arrays can be done like this:

real, parameter, dimension(*) :: Array =                                  &
                 [( 1.0 + 0.145 * (i-1), i=1, floor((2.0-1.0)/0.145 + 1) )]

or, if the compiler does not support assumed size arrays (*), with explicitly calculated array size:

real, parameter, dimension(floor((2.0-1.0)/0.145 + 1)) :: Array =         &
                 [( 1.0 + 0.145 * (i-1), i=1, floor((2.0-1.0)/0.145 + 1) )]

Test program

! This program illustrates how to produce equally spaced real vectors with
! fixed increment in Fortran.
!
! 1. Produce exactly N_VALS values starting from INIT with increments INCR
! Array = [( INIT + INCR * (i-1), i=1,N_VALS )]
!
! 2. All values starting from INIT with increments INCR and up to the limit END
! Array = [( INIT + INCR * (i-1), i=1,floor((END-INIT)/INCR+1) )]
!-------------------------------------------------------------------------------
program spaced_array

    ! Integer counter for implied loops defining vectors.
    integer :: i

    ! Example A. Produce an array of 10 values
    ! starting from 1.0 with increments 0.1
    real, parameter, dimension(*) :: Array1 = [( 1.0 + (i-1) * 0.1, i=1,10 )]


    ! Example B. Produce an array of starting from 1.0 to 2.0
    ! with increments 0.145.
    ! Note that lower value (floor) for the array size is used.
    real, parameter, dimension(*) :: Array2 = &
    [( 1.0 + 0.145 * (i-1), i=1, floor((2.0-1.0)/0.145 + 1) )]

    ! Example C. The same as (B) but the upper value (ceiling) for the
    ! array size is used.
    real, parameter, dimension(*) :: Array3 = &
    [( 1.0 + 0.145 * (i-1), i=1, ceiling((2.0-1.0)/0.145 + 1) )]

    ! Print the sizes of the arrays that were declared above.
    print *, "Array sizes (Array1, Array2, Array3)", &
    size(Array1), size(Array2), size(Array3)

    ! Print the parameter arrays that were declared above.
    print *, "Array1", Array1
    print *, "Array2", Array2
    print *, "Array3", Array3

end program spaced_array

PDF Card

A PDF version of this document is available here: https://budaev.info/images/spaced-array.pdf.

posted at 10:19  ·   ·  Fortran  wiki  Q&A

Nov 20, 2019

Jun 13, 2018

Gaussian random numbers in Fortran

The HEDTOOLS tools library has a module for working with random numbers BASE_RANDOM​. There is, in particular, a set of procedures for generating Gaussian random values: ​RNORM and RNORM_ARRAY. These are based on the Kinderman & Monahan, augmented with quadratic bounding curves method (Leva, 1992: algorithm 712, Trans. Math. Software, 18, 4, 434-435​).

I have made a quick comparison of the quality of the Gaussian random numbers generated by the simple Box-Muller method (Box & Muller, 1958​)

Classical (ancient) Fortran code:

normrand_number = dsqrt(-2.*dlog(drand(0)))*dcos(2.*pi*drand(0))

that has been used in TEG codes so far...

and the algorithm 712 as implemented in HEDTOOLS using this test program (see attachment).

Fortran code for the test program:

program test_bm
  use csv_io
  use base_random, rand_x => rand     ! Alias rand() as rand_x() for ifort.
  !use IFPORT, only : rand_x => rand  ! This is the Intel Fortran tweak.

  integer, parameter :: prec = 8, arrsize=100000
  character(len=255), parameter :: filename1="file_01.csv", filename2="file_02.csv"
  real(kind=prec), dimension(arrsize) :: norand1, norand2
  real :: timer_start, timer_end
  !-------------------------------------------------------------------------------
  ! Generating Box-Muller random numbers
  call cpu_time(timer_start)  ! START
  do i=1, arrsize
    norand1(i) = sqrt(-2.*log(rand_x(0)))*cos(2.*pi*rand_x(0))
  end do
  call cpu_time(timer_end)    ! END
  print *, "Box-Muller took: ", timer_end - timer_start
  ! Write random normal data to CSV
  call CSV_MATRIX_WRITE(norand1, filename1)
  !-------------------------------------------------------------------------------

  !-------------------------------------------------------------------------------
  ! Generating based on algorithm 712
  call cpu_time(timer_start)  ! START
  call RNORM_ARRAY(norand2)
  call cpu_time(timer_end)    ! END
  print *, "Alg. 712 took: ", timer_end - timer_start
  ! Write random normal data to CSV
  call CSV_MATRIX_WRITE(norand2, filename2)
  !-------------------------------------------------------------------------------

end program test_bm

Comparison of Box-Muller and A712

The alg. 712 looks slightly faster than the simple Box-Muller transform.

alg. 712 is much better, as the Box-Muller significantly deviates from the normal distribution, alg. 712 does not (using the Anderson-Darling test from the nortest R package).

  # Gaussian random numbers by Box-Muller deviate from the Normal distribution:
  > ad.test(data_bm$X1)
      Anderson-Darling normality test
  data:  data_bm$X1
  A = 581.7, p-value < 2.2e-16
  # Gaussian random numbers by Kinderman & Monahan's A712 do not deviate from the Normal distribution:
  > ad.test(data_a712$X1)
      Anderson-Darling normality test
  data:  data_a712$X1
  A = 0.46975, p-value = 0.2474

So, the alg. 712 procedure implemented in HEDTOOLS should be used instead of the Box-Muller method.

References

  • Box, G. E. P., & Muller, M. E. (1958). A note on the generation of random normal deviates. The Annals of Mathematical Statistics, 29(2), 610–611. ​http://doi.org/10.1214/aoms/1177706645

  • Leva, J. L. (1992). Algorithm 712; a normal random number generator. ACM Transactions on Mathematical Software, 18(4), 454–455. ​http://doi.org/10.1145/138351.138367

posted at 13:22  ·   ·  Fortran  wiki  Q&A

May 06, 2018

About me

I am a researcher at the Theoretical Ecology Group, the University of Bergen, Norway. My current research focuses on animal and human behaviour in the adaptive and evolutionary perspective. How cognition, behaviour and personality have evolved through adaptation and natural selection? In my work I try to integrate both proximate and ultimate causation and use both experimental and modelling approaches.

Cognition and behaviour

The current work concerns developing a large scale simulation model that implements a general decision-making architecture in evolutionary agents. Each agent is programmed as a whole virtual organism including the genome, rudimentary physiology, the hormonal system, a cognitive architecture and behavioural repertoire. They "live" in a stochastic spatially explicit virtual 3-D environment with physical gradients, predators and prey. The primary aim of the whole modelling machinery is to understand the evolution of decision making, personality, emotion and behavioural plasticity within a realistic ecological framework.

I believe that understanding and modelling complex adaptive behaviour requires both extraneous factors and stimuli as well as endogeneous architectural mechanisms (genetic, hormonal, cognitive etc.) that produce the behaviour. Explicit proximate representation of the motivation and emotion systems, prediction-oriented cognition provides a better approach to understand the behaviour, adaptation and evolution of the whole organism. Ultimately, such an approach can help us understand the evolutionary emergence of consciousness and complex cognition.

For more details, links to source codes etc. see The AHA Model web page.

Animal personality

Although I am interested in any species, most of my work so far has been conducted on fish. Using series of tests we have shown that individual fish of several species have consistent personality traits that translate to a variety of different adaptive contexts. Individual fish with different personalities, such as shy and bold, may behave quite differently in their natural environment, e.g. prefer different social strategies (school or not to school) and different local habitats. Shy and bold fish choose their mates based on personality, personality also significantly affects their parental care tactics. Personality in fish can be linked to the operant learning performance. For example, shy fish may be more susceptible to the development of the conditioned fear, providing a link between emotion and personality in such "lower" vertebrates. It is also possible to trace the development and the appearance of consistent personality traits during the ontogeny. Certain environmental effects, like exposure to light, acting early in the ontogeny could significantly affect fish personality via the involvement of specific brain structures, such as the photosensitive habenula. Personality in fish and other species could be linked with lateral asymmetries via the involvement of the morphologically asymmetrical habenula. I am also interested in the adaptive and evolutionary mechanisms that bring about patterns of consistent personality and alternative strategies. We have shown that gender differences in personality follow from sex-related adaptive strategies in humans. In another study we have shown how a trade-off between parental food provisioning and the fry's own individual experience of searching for cryptic food creates a range of parental strategies in a cichlid fish. This reflects my specific interest in the evolution of mate choice and parental care in fish, and their potential role in sympatric speciation. Currently, we are developing models linking emotion and decision making to understand the proximate and ultimate factors governing the evolution of consistent personality.

Ecology and conservation

I am also interested in complex biological interactions at various levels, e.g. competitive interactions between multiple cladoceran species and their predators, and relationships between various associates and the host within a symbiotic community. The former is closely linked with conservation and species invasion. We have developed a model that allows to predict the population dynamics and the invasion success among freshwater cladocerans in various conditions. I took part in several conservation projects, ranging from coral reef and freshwater conservation in Vietnam to fish monitoring and protection in subarctic Siberian rivers and optimising sturgeon hatcheries. We have developed a series of quick low-technology tests for rapid assessment of the coral reef health. Additionally, we have developed hydroacoustic methods for the assessment of the fish populations in very shallow water bodies, such as large Siberian floodplains.

Links

posted at 21:30  ·   ·  research  behaviour  ecology  evolution
← Previous Page 2 of 2