by ·0 Comments

Videos recorded using the OnlineTvRecorder sometimes have a prefix consisting of a five-digit number, for instance:

71072_Vertrauter_Feind_13.06.19_20-15_kabel1_140_TVOON_DE.mpg.HQ.cut.mp4

Of course, these files fail to sort into the list of unprefixed videos and we cannot search a movie by its initial letters.In this situation, the Bash helps us very nicely with its build-in string manipulation functions. The following snippet takes the filename and removes any sequence of digits followed by an underscore (as regex: [0-9]*_) from the beginning of the filename:

filename=71072_Vertrauter_Feind_13.06.19_20-15_kabel1_140_TVOON_DE.mpg.HQ.cut.mp4
new_filename=${filename#[0-9]*_}
mv $filename $new_filename

We can pack this into a loop to rename a batch of downloaded files:

for f in *.cut.mp4
do
    new_filename=${f#[0-9]*_}
    [ "$f" != "$new_filename" ] && mv $f $new_filename
done

The guard expression avoids warnings by mv if the old and new filename equal.

 References

  • [1] Bash string operations

 

by ·0 Comments

Infinitest is a really great plugin for Eclipse. It executes all JUnit tests it can find whenever something changes. After I downloaded DKPro core, a multi-module Maven project, may CPU was running for quite a long time, until all unit tests had been executed once – even though I just wanted to look at the code…

To keep Infinitest from executing these tests over and over again on the next occasion, I put an infinitest.filters file in every single of the 72(!) modules as follows:

# Prepare file with regex that matches all test cases
echo ".*" > /tmp/infinitest.filters

# Go to the directory of the aggregator project de.tudarmstadt.ukp.dkpro.core-asl and then
find . -name "de*" -maxdepth 1 -type d -exec cp /tmp/infinitest.filters {} \;

Surely, there is also a one-liner that solves the problem, but one works very reliable for me :-)

by ·0 Comments

When we encounter strange, unexplainable problems with text files, hidden characters may be reason. This article describes several possibilities to tackle line-ending and whitespace problems.

Correcting mixed line endings

If a file has mixed line endings, the standard tool flip may help you:

echo -e "unix\nmicro\r\n" > test.txt
file test.txt
#result: test.txt: ASCII text, with CRLF, LF line terminators

A check with file reveals that the file test.txt has mixed line endings. Flip unifies the line endings to Unix (-u) or Windows (-m) standard:

flip -u test.txt
file test.txt # result: test.txt: ASCII text

flip -m test.txt
file test.txt # result: test.txt: ASCII text, with CRLF line terminators

Examining files

vim can show whitespace characters, if you enable the option list. In command mode, execute the following to show whitespaces like tabs or line endings. Unfortunately, the editor does not differentiate between different types of line endings.

:set list

Use :set nolist to return to normal view. With :set ff the program identifies the line ending standard.

If you need to get a detailed picture of the whitespace characters in your document, the octal file viewer od may be helpful, it displays the file as octal values and (interpreted) ASCII characters:

echo -e "item1\titem2\titem3\r\nline2 (unix)\n" > test.txt
od -c test.txt

The results looks as follows:

000000  69  74  65  6d  31  09  69  74  65  6d  32  09  69  74  65  6d
         i   t   e   m   1  \t   i   t   e   m   2  \t   i   t   e   m
000010  33  0d  0a  6c  69  6e  65  32  20  28  75  6e  69  78  29  0a
         3  \r  \n   l   i   n   e   2       (   u   n   i   x   )  \n
000020  0a
        \n
000021

Using cat -v text.txt, you can see bogus (non-Unix) line endings  being marked with a special symbol: ^M

item1   item2   item3^M
line2 (unix)

by ·0 Comments

The following Bash snippet outputs the users of a Unix system sorted by their user id:

cat /etc/passwd | sed 's/:/ /g' | sort -k 3 -n | awk '{print $3 " " $1}'
  • cat outputs the content of /etc/passwd to standard output
  • sed replaces all colons (default field separator) with a whitespace
  • sort orders the lines by the third, whitespace separted field which is the user id
  • awk prints out only the user name and the correspondent user id, separated by a whitespace

by ·0 Comments

Show one’s own/other’s groups

groups
groups LOGIN

Show all groups on the system (with group id):

cat /etc/group | less

Create new group

groupadd GROUP

Add user to group

usermod -G GROUP -a USER

Assign a specific home directory and shell to a user

usermod -s /bin/bash USER
usermod -d /tmp USER

If the user shall only be allowed to do SCP/SFTP transfers, then an appropriate login shell is scponly.

 

 

 

by ·0 Comments

When downloading large files with scp, it is really annoying, if the download process stalls at 90+%, because it seems that you have wasted your time and scp leaves you with a broken file. Having encountered this problem, I was lucky enough to find this post which explains a remedy: Use rsync in order to complete the interrupted download.

Assume you downloaded a file with scp like this:

scp user@hostname:/path/to/file /local/path/to/downloaded/file

then you can continue the download as follows (note the –partial option):

rsync --partial --progress --rsh=ssh user@hostname:/path/to/file /local/path/to/downloaded/file

Rsync finds out that a large portion of the file is already there and continues where scp once has stopped.

Links

  • [1] Article with solution

by ·0 Comments

Miscellaneous

!! Re-execute previous command.
!27 Execute command 27 in the history (look it up with: history | tail)
Ctrl + R Reverse search in history (type Ctrl + R to search incrementally)
Ctrl + L Clear the terminal. The same as typing clear
Ctrl + M Enter/return. Useful if inside a long command.
Ctrl + S/Q Freeze terminal/Restore frozen terminal

Move and View Commands

Ctrl + A Go to begin of line (mnemonic: “Anfang” (DE))
Ctrl + E Go to end of line (mnemonic: “end”)
Ctrl + F or Arrow Right Forward one character.
Ctrl + B or Arrow Left Backward one character.
Alt + F or Ctrl + Arrow Right Forward one word.
Alt + B or Ctrl + Arrow Left Backward one word.
Shift + Page Up/Page Down Scroll view up/down one page

Edit Commands

Ctrl + K Delete from cursor to end of line
Ctrl + U Delete from cursor to begin of line
Ctrl + W Delete word left of cursor
Alt + D Delete word right of cursor
Ctrl + D Delete character under cursor or exit bash if line is empty.

by ·0 Comments

Maven-managed Eclipse projects generate appropriate project settings (classpath, build path, etc.) from the pom.xml. The following files need to be ignored in a Maven Eclipse project:

.settings
.project
.classpath
target/

As a shorthand, the following Bash command adds these lines to your .gitignore. It does not matter if your .gitignore is located in the project directory or in any parent of it.

echo “.settings
.project
.classpath
target/” >> .gitignore

Links

  • [1] gitignore man page

by ·0 Comments

Some image processing programs expect the file extension to be in lower case. Many digital cameras, however, tend to use an uppercase extension (JPG in case of the Panasonic TZ-31).

The following Bash one-liner renames all JPG files in the current folder to have the lowercase jpg extension afterwards. The ${…} expression is called parameter expansion, which is quite a powerful mechanism:

for file in *.JPG; do  mv $file "${file%.*}.jpg"; done

Links

  • [1] Manpage of the Bash (see Section on Parameter Expansion)
  • [2] Parameter Expansion explained on bash-hackers.org